Priorities

I need to prioritize things a bit more, otherwise there will be too many started, but unfinished projects.

Top priority should have now the Android Coloring App for Kids. I want to have it published within 1-2 months with at least 50 coloring templates ready.

Sharing top priority is version 1 of the Opensource Games Library including making advertisement about it, which I want to have done within one month.

There is the Hale project, a Java RPG, which I want to have in a state that there is a Windows installer and it's also running on Linux and the most obvious current glitches are removed. Would be nice to have this done within the next 3-4 months.

And there is the FreeRails project which needs design, further rewrite of the model core and a new GUI which would be nice to have in 5-6 months.

I'm tempted to do a bit here and a bit there, especially when I'm frustrated with one thing, but that doesn't really finish anything, so I should stop doing that and instead focus on a single project at a time.

FreeRails: Are we on the right track?

Track management is among the most often performed actions within any railroad simulation. It's also very close to the heart of the programming and design of any railroad simulation. Lately, I was interested in getting a better overview about how the track management in FreeRails work as well as possibly streamlining it and documenting it properly.

The map consists of an rectangular grid consisting of square-sized tiles. Each tile (except those at the border) have exactly eight neighbors which can be uniquely identified by compass points (north, north-east, ...) given the current center tile or by grid positions (row, column).

The total track on the map consists of many track pieces, where each piece connects two neighboring tiles (diagonal connections have ~1.41 (square-root of 2) times the length of horizontal or vertical connections).


Path finding of the trains works on a graph where the tiles are nodes and the track pieces connecting neighboring tiles are the edges.

Building and removing track works by adding and removing track pieces. In particular, the planning of a longer piece of newly built track is done by path finding again.

The track itself is visualized by rendering each tile according to its track configuration. The track configuration is an 8 bit value indicating if there is a connecting to one of the 8 neighbors from the current tile. See the attached image for some examples.

Track configuration encoded as 8 bit value

The track configurations are not independent from each other. For every connection on a tile towards a neighboring tile, this neighboring tile must also have a connection to this tile. This invariant must be obeyed by not allowing to change track configurations directly, but only by allowing adding or removing of track pieces at a time.

Track pieces have no direction, any train can go on them both ways. However, a train can change direction at every tile at most by 90 degree, effectively inducing some kind of directionality.

Track pieces can be single tracked or double tracked. There can only be one running train on each track piece (and trains have a certain extent, also measured in track pieces). However, stopped trains do not count as obstacles.

The track configuration of a tile is sufficient to draw it uniquely on the tile. Not all possible 8 bit values are valid.

Bridges and stations are a special case. Stations have a orientation and only allow track parallel to their orientation. Bridges span a water tile (other track cannot be put on water) and consist of two track pieces resulting in a parallel configuration.

That should be all for now. There is of course more (like trains). FreeRails currently follows the concept outlined above quite well. For some reason the track configuration is a 9 bit value (center part is also encoded and used to indicate no track at all) where I think that 8 bits would be sufficient and it's possible to change track configurations directly, whereas it's better to only change track pieces and change the track configuration only at a single place in the model.

Will be continued...

Milestone 1.0 for the opensourcegames list created

47 issues still to solve and then everything is ready for going public (really advertising it as something of a usable, real thing).

Check the progress at: https://github.com/Trilarion/opensourcegames/milestone/1

FreeRails: Integrated CVS, SVN and Git history from 2000-2017

Typically, there are lots of warnings of changing the history of Git repositories after they were already pushed to the public, but I think that this is one of the typical exceptions from this rule.

After starting with FreeRails I learned a lot about converting CVS and SVN repositories to Git and about using advanced Git history editing commands like git replace or git filter-branch. Therefore I could really include the history of the FreeRails 1&2&3 and Railz 1&2 projects in a single Git repository.

On the other hand, although there are three forks on Github, there has not been done any work on them yet. The chances that somebody did do something and gets into trouble now is rather very slim. However, in this case, I'd be happy to assist. Nothing is lost. It's just a matter of chaining sufficiently many git commands.

Civil: Integrated CVS history from 2000-2004

The project has not progressed much lately but at least I managed to convert the original CVS code repository containing all the development work of 2000-2004 into the Git format and then I could persuade the existing Git repository to prepend these old code changes. Like everything related to Git it wasn't straightforward (conflicts where none should be) but worked in the end.

That brings the project to a whopping 1600 commits since the year 2000. A read-only copy of the CVS converted to Git has also been put on Gitlab.

FreeRails: Next programming steps

I finished quite some substantial changes under the hood, but I'm far from finished. It left me a bit exhausted mentally. Getting your head to solve complex puzzles in your free time is rewarding but also somewhat tiring. That's why I'm currently looking for some easier tasks as well as some planning tasks before returning to the inner workings.

Possible tasks include

• Go through all locations in the code marked TODO and work on the easier ones.
• Make some hard-coded properties of the game model like the economic climates (recession, moderate, boom) or the game rules (can connect to other railroads) serializable and then load their default state from a file when a scenario is loaded
• Go through the code (especially the moves, the move generators and the handler) and identify code that does game logic and move it to the model.
• Add to the design document details of the planned client-server interaction.
• Describe the game model in the design document.

 So, that's what I will do now before starting some major rebuilding like

• Moving Activities from the world to the Trains where they belong
• Replacing CompositeMoves with single step comprehensive Moves and removing undo on the moves (we don't undo anymore)
• Create a Player specific World view, so that the player doesn't always have to specify his Player object (alternatively, store your Player object with your credentials in a context, that then is stored next to the world)

which are all (among others) prerequisites for creating a new and modern looking client.

FreeRails: Still under the hood

Progress was a bit slower in the last couple of months. Time for an update. Work is done mainly under the hood working on the model and the model updates in order to make it simpler and more resilient. The aim is to fully understand the game model, make it error free and easy to maintain and extend. A side effect is unified loading/saving of scenarios and savegames.

Only after this essential work has been finished, one can think about adding a new chassis (client interface). As a reminder, the latest dev progress is shown here.

Dynamic table is online

A dynamic table showing all entries of the open source game database, is online. One can sort the columns as well as limit the number of rows by using a search field.

I also converted a lot of svn/cvs repositories to git lately and put them on Github/Gitlab.com.

Dynamic table showing open source game entries

400 entries in the OS games collections

There are now 400 entries in the collection of open source games on Github.

Also I started creating a dynamic HTML table using Vanilla-Datatables as well as checking the source repositories and converting some sources (cvs, svn, sources only as snapshots in downloads) to git.

All git source repositories can also be cloned and updated automatically. Currently 180 git based repositories requiring 40 GB disc space can be archived with a single click.

Civil: Slow at the moment

The progress of the rescue attempt is a bit slow lately, five months passed without any update to the code base. Reasons might include:

• I'm rather busy with other (more interesting) projects
• The project is in a dynamically typed language (Python) instead of a statically typed language (like Java). This makes refactoring much more difficult. Also dependencies between different section of the code are more difficult to understand. This makes the next task: understand the code base more difficult than really necessary. I'm curious how large and complex Python projects get maintained actually.

However, I think it's fine if it goes slower. The current state is on Github for all to see and continue.

Plan for the open source games repository

The plan for the open source game repository is as follows:

• Finish adding entries for all the open source projects I could find.
• Determine license and source code status for all of them.
• Create a dynamic table web site to show them all and filter them.
• For those not on Github, Launchpad, Bitbucket, .. import the code into a public (reliable) repository.
• Keep and update local copies of these repositories automatically.
• Build these projects and improve the build/packaging where possible and desired.
• Try to rescue abandoned projects.
• Increase number of supported platforms (like a port to Android) where suitable.
• Offer downloads of fresh builds where not already existing.

Of course, in essence this repository will remain a database/catalogue of open source games, nothing more. However, the additional activities rely on the database. It's kind of an infrastructure/service around development of open source games.

So many aspects

Last time I wrote about the roles (different kind of jobs) that are required for developing an open source game. Let's focus now on one such job (Programmer) and see how many aspects this single job contains for developing for example open source strategy games.

• GUI programming (almost all games have some sort of graphical user interface)
• 2D/3D graphics programming (to display the game world, map, ...)
• Serialization/Deserialization (to store and retrieve game states (aka savegames))
• Network programming (for multiplayer capabilities)
• AI (for computer opponents)
• Data modelling (to store and advance a game model)
• Algorithms (like path finding)

This means that there are many different topics one must know about to make it all coming true. Of course, the most efficient way is to use (and know) as many stock solutions to these problems as possible.

300 entries in the OS games collections

I continued collecting information about open source games and there are now 300 games included in the collection of open source games on Github.


In the mean time I also tried to improve/rescue projects that were inactive for a longer time: FreeRails, Civil and Hale.

Try to improve Hale

Recently started tinkering a bit with Hale, a turn based RPG with deep tactical combat and character customization. It's in a very good shape and might only need a few improvements and maybe also a bit of advertisement.

Continuation of the development of Hale takes place on Github.

So many roles...

One thing I like in particular about being an open source game developer is the variety of jobs that need to be done. A non-exhaustive list of available jobs would be..

• Game designer: inventing game concepts, rules, properties
• Software designer: design algorithms, interfaces, entities, relations
• User interface designer: designing the screens and buttons
• Programmer: implement it all
• Software quality tester: debugging, testing
• Artwork creator: drawing, composing, creative writing
• Project manager: publish releases, coordinate activities
• Technical writer: writing documentation, design specifications, player manual
• Play tester: play intermediate versions
• Web designer: maintain an online presence
• Community manager: keep in contact with fans and coworkers
• Journalist: write about the project
• Legal expert: manage license of various content
• Translator: internationalize the content

I do not need to stay with one job. For example, in case programming gets boring, I can do something else like writing about it, which I'm doing now.

Of course I have favorite roles and I would be happy if more volunteers would contribute to the project doing one job or the other. But so far it's more or less a one man show with about a dozen different jobs done by a single guy in his free time!

FreeRails: Taking notes...

Usually, I like scribbling down things with a pen on paper a lot. It gives me a lot of creative freedom.

But piles of paper are difficult to keep organized (or from vanishing). So for this project, I started writing down notes with an office application. Basically just long bullet lists of things that some objects do or how they are connected or how actions are sequenced, mixed with some conclusions of what to change.

Now there are already 16 pages of notes and the file is still growing. However, while I continue to make changes, most of the information in these notes gets outdated again, the overall value is limited. It's really just an extension of my mental capacities thing but still an essential tool for my work.

FreeRails: Back to the drawing board

This is a bit technical post.

Now is the time to go back to the drawing board, write down the existing design and even change it. One such issue is the overall design of the client/server framework. Looking at what is there and what is regarded as modern design I came up with this proposal.

Schematic of the proposed Client/Server design in FreeRails

The client is basically a Model-View-Presenter scheme with the View being very passive and easy to change if for example and Android or Web port is attempted. The presentation logic is in the presenter which updates the view and reacts on user input. The presenter/controller is a big black box here and needs to be detailed later. The client also contains a full copy of the model/game world, making him a thick client. He acts upon changes in the model and proposes changes to it. However, the model just reroutes these proposed changes to the server.

The server checks the proposed change for applicability and either accepts it or rejects it. It then applies the change to itself and sends it to all clients so they can apply it to their copy of the model and the user will see them. It follows that this way all the client models will be in sync with the server model, although they may lag behind. To make sure they don't miss anything, the broadcasted, accepted changes will be enumerated or similar.

One can see nicely, how the information flows in a circle, from the view (user input) through presenter, local client model, server model, local client model again, presenter again and finally a change is displayed.

I think this requires the lowest amount of coding effort while still delivering robust performance and good separation of concern and modularity.

P.S. For drawing the design diagrams I use draw.io which is an excellent tool and works for me like a charm.

FreeRails: Replacement of world differences object

In the last week I could refactor the current code in such a way that generic differences of world objects and actions that update world objects with these differences aren't needed anymore. I replaced the uses of these powerful but also very complex objects by a series of more primitive updates on the world. This also allowed me to remove code that tracked changes between nested lists (needed to represent differences of the world object).

I could reduce the size of the code base considerably without changing the functionality except for a small bit on the client side when displaying user messages which can be repaired later.

This step should make it easier to understand and advancing the code. Immediate next step will be some software design of the server-client relation and of the user interface as an initial step of rebuilding the client.

FreeRails: Next step, rebuilding the client

The refactoring of the code is coming to an end. I made some pages of notes on how the code works. The next step seems to be clear: the rebuilding of the client using JavaFX and in the course of this simplifying the server and client interaction if possible while retaining all the features.

This should result in a simpler, easier to modify code base and also more modern look at the same time. However, it will also take some time and meanwhile it will look from the outside as if not much progress is happening. This will surely limit the interest until this task is finished.

On the other side, someone else posted about the project on Reddit and I posted about it on the FreeGameDev and Java Gaming forums. This brought considerable interest and visitors on the project page. It's a good start.

FreeRails: Understanding the code

I begin to understand the code. That took quite some time staring at the screen and doing some refactoring. To quantify, roughly two hours per night for three nights a week for two months. Indeed, that long.

With a better documentation the start might have been faster, but I guess that ultimately there is just almost always a huge cost upfront when learning about a new code base. It just takes time.

To give an analogy: In the beginning the code was just like a bunch of single trees to me. Now it's like group of trees interacting and a vague notion of the whole forest, aka the complete picture.

There is hope of getting the complete picture rather sooner than later and there is hope that once you have the complete picture you can do anything you want to do with it including any simplification or continuation of the development as well as documenting the complete picture so that others have it a bit easier.

One thing I consider is branching from the code and, for the first time, really cut some functionality of the original code (the world difference update moves). The disadvantage would be that functionality would get lost and the advantage that quite some code would not be necessary anymore, so I would need to understand less and also I don't like the idea of moves which basically update the whole world. Once you have them you basically don't need any other moves anymore.  Of course the missing functionality would have to be included again at a later time. Hmm, it's a difficult decision. I will postpone it a bit to give me more time thinking about it.

FreeRails: Current screenshot

0.4.1 - Main screen

Here is another screen shot of the current version. It doesn't look much different from the last screenshot. Maybe it's just to prove that all the recent changes in the code (mostly refactoring and deleting unused code) didn't break anything important.

Coloring templates for the Coloring app

The working on the coloring app progresses well. I realized that I had the main part (the coloring) already implemented last time. Now it's just a few lines of code here and there, mostly implementing the gallery functionality where you can watch your results. However, surprisingly, the biggest challenge currently is getting enough high quality content. I can only take appropriately licensed (Public Domain or Creative Commons or equivalent) material and it needs to consist of about 20-50 separate areas (not every picture actually is suitable for coloring).

My best source so far is openclipart.org.

I will get search until I have around 50 nice coloring templates, then polish the app and test it on all available devices and then I will jump into the water and publish on googleplay. That may still take about a month, I guess.

FreeRails: Current development goals

I'm intending to bring the project to a state where it is relatively simple to implement all the intended features for a "version 1.0" release. I hope others will join and will bring it to the full feature release then. With limited resources on my side, bringing the game to this intermediate point may take some time.

The milestones I want to reach in any case (may take some time) are:

• More complete description of the game design in the documentation
• Clear separation of model, view, controller, server, client, serialization code
• Reduction of dependencies and complexity of the code
• Improved graphics and graphical user interface using JavaFX
• Runnable, bug-free version with an installer
• Some kind of roadmap towards version 1.0

If this can be reached within the year 2018, I will be happy. Also from then on, I can continue managing the project but probably not actively develop it any further due to time constraints. However, I would also consider it rescued then.

FreeRails: How to untangle code?

There are three basically different approaches

• Study the code, find the crucial parts that are connected but should not, find a solution to untangling them, then do the programming
• Study the code and untangle code wherever you find it, until all but the crucial part remains, then tackle it
• Study the code and rewrite the code (in a simpler form) in another set of sources which then could replace the original code at some point

The last method is very risky and may easily fail. The first one is less risky and overall probably taking the shortest amount of time. But somehow my head is not big enough to grasp the full complexity of FreeRails yet. So I go for the slow but relative safe way of taking the second option currently, slowly reading through the code base, frequently running the tests and running the game, optimizing locally, untangling the code one piece at a time. It's quite an effort, but it's also the only good way I know. It will take some time.

Civil: Next steps

There seems to be a major issue with the game. That is that the server does not seem to advance the game world except for the time. Units do not move, probably do not fire at each other too. There was an overlay layer for the action phase in the interface but it is commented out. Also the description of version 0.83 (the last official release) reads as if the game was real time then, so one would expect a continuous update. However, maybe that was not fully implemented in 0.83?

The question is: How can the game be fixed in this regard?

The second issue is that although the editor kind of works now with PyQt5 there is still a lot of work to do with it. I would like to use PyQt5 also for the main game in order to reduce dependencies, use less code (PyQt5 brings many features already).

Finally I would like use as much as possible from the old code, at least the whole game model and the graphics and the layers.

The crucial next steps should therefore be:

• Understand the inner workings of the code base
• Identify and separate into model, view, controller, server
• Find out why the game model is not updating as it should
• Replace the view and controller with PyQt5
• Add a more or less working editor
• Release
• Make a plan for the future development

Android coloring app for kids

Two years ago I started a coloring app for kids on Android. It's completely open source and loosely based on another open source coloring app. It's basically not a very challenging project and serves to also get to know the Android environment better.

The code is on Github.

However, after some work I stopped in an intermediate state. Recently, I rediscovered it after it got even some interest of others on Github and continue working on it now.

Statistics of open source games

The list is by no means complete but I thought I would share some statistics about the ~200 investigated open source games projects.

State

Approximately 50% are mature, 25% in development and not yet mature and about 20% abandoned.

This may be skewed though because games in development are definitely harder to spot than mature projects.

Most used Programming languages

• C++ (40%)
• C (24%)
• Java (8%)
• Python (8%)
• Lua (4%)
• Javascript (2%)

A solid knowledge in C/C++ or Java or Python will be very useful for aspiring open source contributors.

Copyright

Most often the code is licensed by

• GPL-2.0 (37%)
• GPL-3.0 (19%)
• MIT (13%)
• Custom (7%)

 which means that open source projects in general prefer to keep modifications open source too.

Civil: Refactoring, refactoring, refactoring

While upgrading the code base from Python 2, PyGame 1.5 and PyQt3/4 (state of the art in 2004) to Python 3, PyGame 1.9 and PyQt5 you need to approximately change only 20-30% of the code base and often enough it can be done by some refactoring.

However, with Python being a dynamically typed language, refactoring is not so simple. Also the Civil project misses unit tests. What worked in the end was:

• Heavy usage of regular expressions to modify code that follows a certain naming structure to, for example, exchange arguments. I learned quite a lot there too.
• Starting the program again and again, doing the same stuff over and over and fixing runtime errors one by one, so next time it runs a bit further.
• Using a source versioning system and saving intermediate steps often (with Git I often amend commits to be able to revert uncommitted changes if needed) so one can go back if nothing else helps.

By far the most typical runtime error I encountered was

AttributeError: 'module' x has no attribute 'y'

which mostly indicates that the variable has a different type of what people expected, which can have many different causes. Still, fixing things can be very satisfying. :)
 

FreeRails: How to reduce complexity of the code?

I looked at the code base of FreeRails now for quite some time. It's not simple to understand. There is quite a lot of complexity involved.

On the one hand that is expected. Managing railways isn't simple. Incorporating the logic of trains running on tracks with a delivery schedule, cargo producers, financial transactions, extending the track network in a changing world, ... that is even a difficult problem in the real world. :)

Mapping this to a client/server framework where the server even has to defend itself against illegal moves (no underwater tracks so far) of the client or offer some kind of undo capability adds its share to the problem.

Add some common practices like missing documentation or using overly complex (cyclic) dependencies and the code soon mutates into a huge spaghetti monster. (That is by no means an accuse of anyone, it's just what happens naturally. Also reading code is often harder than writing it.)

How to untangle the spaghetti, reduce the complexity and facilitate further development? That is the big question!

I'm not aware of an easy way to do that. It rather takes much time and the first steps are usually the hardest while later most things more or less fall in place. But the project is open source and on Github, so everyone can take part.

Railway Empire looks so nice

Most of the times commercial games look much better than open source games. A large group of skilled professionals working full time for many months with highly refined tools and frameworks - that cannot help but must outshine all attempts of a few partly skilled volunteers in their free time.

And so it is here too. There are indeed some Railway management games out there and since at least 10 years in quite good 3D graphics.

But the soon to be released Railway Empire (to be released on 26th January 2018) by Kalypso Media/Gaming Minds Studio just stuns me by the beautiful graphics.

Need a proof? Look at that gorgeous screenshot of a train in a landscape.

Makes me wonder how meaningful the revival of FreeRails is anymore? On the one hand it's fun to go through the code and improve it and surely the look can be improved dramatically from the look back then even with modest means, but on the other hand professional, commercial products have improved so dramatically from the early 2000s to now. Maybe there was a demand and a reason to clone RailRoad Tycoon in 2000-2005, but I'm not sure there is any reason now anymore.

Civil: Screenshots of 0.84

The main screen

This is the main screen as it looked in 2004 but in 2018.

FreeRails: Refactoring, refactoring, refactoring

The complexity of the current code base of FreeRails is quite high. There are many classes and methods which are highly (sometimes cyclically) dependent on each other and often enough comments are missing. Additionally the game was thought to be a complex multi-player game with a server-client structure, with pathfinding for the trains, and complicated updates of the game world state (especially the trains).

That might be a part of the reason why development on FreeRails more or less stalled after 2005. It's very quite difficult to read this code and possibly extend or modify it.

On the other hand, the original makers used tests to continuously check their code and they wrote down their design ideas in a document. Also Java as a strongly typed language running on a standardized virtual machine, offers many tools to entangle such difficult to understand code bases without loosing the ability to execute the game.

That's why I could refactor the code base now for a week without loosing anything. Using static code analysis (code inspections in IntelliJ) and removing unused code, applying automatic fixes and manual improvements, replacing custom solutions by Java standards (which did not all exist in 2005 like Lambda functions in Java 8) I could reduce the number of code lines from 39k lines of code in version 0.4.1 to 33k lines of code now which means roughly 15% less code to understand. And as I said without loosing anything in functionality, the game looks and runs as before.

Civil: Minor release 0.84

After getting the game to run, I decided to release the current state of the code as version 0.84. Downloads are hosted on PyPI and there is a Windows installer on bintray.

I used the right to increase the version of the GPL-2.0 open source license and published the code under the GPL-3.0 license. However, the content and look is probably very similar to version 0.83 from 2004. The editor is not yet working.

There are two ways to get Civil 0.84:

1. Use the Windows installer

Windows Installer (18 MB) comes bundled with its own Python runtime.

2. Use PyPI

Install with 'pip install civil'.

Civil: Getting the game to run

Getting a game that saw last development many years ago to run is a hard task. It used Python 2, I wanted to use Python 3. It used PyGame 1.5.3, now the actual version is 1.9.3. And because there was no ' binary distribution' I cannot be sure if ever it run. And there are parts written in C which I would like to avoid. Python is still there and even more popular now, but being a weakly typed language, makes it harder to refactor or detect unused code.

In the end I managed (and could restructure the project too) by

• Repeatedly (many, many times) starting the game, letting errors happen, looking at the stack trace, setting breakpoints, inspecting variables, fixing code.
• Replacing old  code with fixed code in multiple location simultaneously using regular expressions and project wide find&replace in PyCharm
• Using PyCharms refactoring capabilities
• Using the 2to3 script to update Python 2 to Python 3 code

A mixture of all these methods finally made Civil run for me on a modern system with the limitation, that units do not move on the battle field. I'm not sure, if the original developers actually fixed that.

Civil seems to be free of obvious bugs.

FreeRails: Minor release 0.4.1

After an initial round of cleanup of the code base, and building the original FreeRails 0.4.0 code with Java 8, I released this slightly improved version as 0.4.1. Downloads will be hosted on bintray.

Download 0.4.1

I used the right to increase the version of the GPL-2.0 open source license and published the code under the GPL-3.0 license. However, the content and look is almost exactly the same as version 0.4.0 from 2008.