Day eleven of Titancraft. Before every commit the game runs 12 automated test suites: a server and bot players walk to the ruins, kill the boss, build a house, shoot a tree, row a boat together. A full run took 13 minutes. The game is about to grow several times over, and the tests with it, so today we sped up the tests themselves: a full run now fits in 3.5 minutes.
Where the time went
Each suite starts its own world: a server and one or two bot clients, no windows. The bots follow a script. After a set number of seconds every game instance prints a summary — who stands where, what is built, what is in the backpack — and the suite compares the summaries.
The timers had generous margins. The logs show the suites spent almost all their time waiting: two bots kill the boss at second 71, and the suite waited until second 204. The last arrow is pulled out of the tree at second 23 of 64.
On top of that, suites ran strictly one after another, and so did the cases inside them. Five movement scenarios ran one by one, two fights one by one, even though each had its own world and port.
Three steps
At the same time. Every suite has its own port and temporary worlds, and bots without a character name save nothing, so suites have nothing to fight over. We started running four at once, and a run dropped to 4.5 minutes. Then we hit the longest suite: the run to the boss.
Early summary. The game got a --stop-file switch. The suite watches the logs, and once everything it needs has happened — the boss is dead, the arrows are pulled, the tree is down, the player has entered the world — it creates the file. Every game instance prints its summary and quits right away. The old timer stays as a safety limit. The run to the boss now takes 90 seconds, arrows 33.
Inside suites, too. Five movement scenarios take 12 seconds instead of 58, two fights with legendary weapons 33 instead of 75.
Where we had to slow down
On the first try everything ran at once, more than ten game instances, and a run took 135 seconds. But the checks that compare client and server started to fail. On a busy machine the client falls behind: the boat passenger drifts off their seat, and movement prediction drifts from the server beyond tolerance.
So tests/run.sh now counts game instances, not suites: one suite holds a single instance, another holds six. At most ten run at a time, and the lag and boat suites run last, one at a time. That costs 76 seconds, but we no longer have to guess whether a failure is in the game or in the load.
The short suite
For checking edits there is now tests/run.sh quick. It compiles the project first and stops right away if a script has an error. Then it runs five fast suites at once: networking, latency, movement, world and fish.
It is short enough to run after every notable change, with the full run before a commit.
What's left in the game
- The run to the boss takes 90 seconds, 71 of them spent walking and fighting. It is now the longest part of the run.
- Every suite runs even when only one system changed. Next step: run only the affected ones.
- The boat passenger falls behind their seat when the machine is busy. They should be driven by the boat, not by their own movement.
- Under heavy overload the network drifts apart. That matters on weaker computers; it's next to investigate.
- The check that a pack notices the archer doesn't always pass and is waiting to be investigated.