devnotes

Testing Times

Introduction

The most important feature in software is stability and Lead Pursuit has worked extremely hard to maintain the highest standards. One cause of instability in games is bad data. It can lead to the infamous Crash To Desktop (CTD) error, where the game completely shuts down, dumping the player to the Windows desktop. But bad data can also throw up huge in-game realism wobblies if those issues are not tracked and corrected.

When we talk about data, we are referring to the tens of thousands of variables which help define the characteristics of everything in Falcon 4.0: Allied Force -- anything from the range of an AIM-120 missile to the texture set used by a particular aircraft. In these development notes, Lead Programmer Julian Onions discusses the business of maintaining data integrity.

Designer's Testing Notes

Have you ever wondered what it takes to make a good database of aircraft, weapons, systems and the relationships between them? The data in this simulation is so deeply intertwined it defies some of the best brains trying to keep track of what is happening.

Consider, for instance, that one of the items of data we need to know, is "at what distance does an aircraft become a threat to you?" An F-16 is probably good for 20nm, as it may well have Aim-120s of some sort. An A-10 is probably only dangerous within a couple of miles. This is an example of one of the variables in the database, that the AI will use in its reaction. It doesn't mean that the AI knows the enemy is packing AIM-120s, but if it detects an F-16, then its a good idea to stay outside of 20 miles or so, unless it wants a fight.

To work out in general terms what the effective range of an aircraft is, it means iterating through all its weapon stations, isolating those that are air-to-air capable, examining each possible munition that could be placed there, and then seeing what its optimal range is, at a range of altitudes. This is the sort of operation that computers are good at, and humans bad at. However computers are notoriously bad at saying, "Wait a minute, that's crazy, an AN-2 should not be considered hostile at 50nm?!?".

Humans or Computers ?

So having a human in the loop is extremely useful. How do you square these two things and use the power of the computer, with the common sense of a well read human? In this project we decided to accomplish this by writing a test suite. The suite can be run manually or automatically. It has intimate knowledge of the overall database and can correlate and range check lots of things.

In the above example, the test suite goes through the records, computing the effective range of the aircraft and then compares it to the value in the database. This involves traversing multiple files in different formats to come up with a result. If the result is not the same as the expected value, an error is flagged up, and one of the database people will investigate. My colleague, Ed Kiefer, had the uneviable but vital task of correcting database entries running into the thousands -- the results are major improvements.

Each time the test suite is run, the database is subjected to well over 300,000 tests -- which even "blew up" the test suite at one point because of the enormous number of tests to run. The analysis ranges from the simplistic case of checking a value is within a well-defined range, to complicated interrelations between tables.

This has been very painful at times and getting the database into shape has taken many, many hours. Thousands of errors have been found. Some serious, some minor. The efforts have been worth that pain though. It used to be the case we would get application crashes when a certain action happened in the game. The reason was down to a corrupt list. Including this check in the tests turned up another 10 or more objects with the same issue, that were used less frequently, but just waiting to cause problems for the player.

In summary, it's been tricky work, but the substantial changes to the database have undoubtedly strengthened stability and helped ensure that the simulation performs to expectation.