Inside the Box serves as a forum for individuals involved in the production of Gearbox Software content to share personal motives, methods, process and results. Gearbox Software projects are created by a diverse range of individuals spanning a spectrum of different backgrounds, interests, objectives and world views. The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of Gearbox Software or any of its individual members outside of the author.
I’ve been a Jack-of-all-trades programmer at Gearbox for over 14 years where I’ve worked on Opposing Force, the Brothers in Arms games, Borderlands and many more. Gearbox is mostly known for first person shooters, but we’ve worked on a skateboarding game and we were involved in bringing Samba De Amigo, a game about maracas, to the Wii. Despite that I have to admit that I never quite expected to be involved in working on a real-time strategy game again.
I was a bit surprised when, earlier this year, I was told by a manager at Gearbox we were looking at bidding on the Homeworld franchise. He told me that we would be getting a listing of the files that THQ had gathered from the franchise. He also mentioned that Brian Martel, Gearbox’s Chief Creative Officer, had downloaded the public domain Homeworld source code. I was asked if I would be interested in looking into this to see what we would be getting if we succeeded in acquiring Homeworld from THQ.
This (and other images in the article below) are among the really awesome things we found while digging around old source materials for the Homeworld series.
Homeworld was released around the time we finished Opposing Force and I remember having played it a bit back then. Being somewhat of a packrat I still had my boxed copy of the game on a shelf in a closet at home. A month or so ago I dusted it off and it is now sitting on my desk at work. I had also worked for a while on shipping a real-time strategy game, Dominion: Storm Over Gift 3, at the company I worked for before coming to work at Gearbox. Getting involved with working on the Homeworlds would allow me to step back, for a little while, to what I did before working on first person shooters.
The source code for Homeworld that Brian had downloaded was released by original developer Relic Entertainment at the end of September in 2003, not long before they shipped Homeworld 2. Homeworld was released at the end of 1999 which means, at this point, the source code was over 13 years old. Compiling the code with the tools I had available was going to be a challenge.
Homeworld shipped using, it appears, Microsoft Visual Studio 97 as the main C/C++ compiler. I decided to go ahead and try to build the game with the most current Microsoft compiler to see what I would get. This did result in a number of warnings and errors due to the changes that compilers have gone through in the 13 or so years that have passed since this code was written.
I was not that far into looking at the Homeworld code when we got the listing of the files for both Homeworld and Homeworld 2 that THQ would provide to a winning bidder in the auction.
It’s interesting to evaluate what you think code is doing based entirely off directory and file names. A lot could be gathered from what I could see in that list, but there was still a bit of guesswork as well. The main things I was looking at were if the source code seemed complete, and if there was any use of 3rd party libraries we should be aware of. I was mainly going to focus on what would be provided to us for Homeworld 2 since I had already been looking at the source for the first Homeworld.
The renderer was straight OpenGL and I could not see any signs of the Direct3D wrapper that had been provided for Homeworld. The sound appeared to primarily go through DirectSound calls, though I could also see an “mss6” directory that I imagined could be Miles Sound System. There was a “NetworkDP8” project which I assumed to be DirectPlay 8 (a networking support API in DirectX that was no longer updated by Microsoft with the release of DirectX 9). It seemed like DirectShow was used to play videos, but there were Divx DLL’s that indicated that was the codec used. It was definitely written in C++ and had Visual Studio project files so loading it up to compile would likely be easier than it had been for Homeworld. It seemed like getting this to work in a modern development environment would not be that much of a challenge.
I made notes about what I had found, constructed a list of all 3rd party software and libraries I could see and noted anything that seemed to have a GPL license versus a retail license. This evaluation and documentation was sent along to the Gearbox partners at which point the waiting game started. Gearbox Senior Producer Brian Burleson has written an “Inside The Box” article that talks about what was happening before and during the auction process. I had moved on to helping out on other projects at Gearbox while occasionally firing off an e-mail to the partners asking where things were in the auction process. About three weeks passed between my sending my update to the partners and Gearbox CEO Randy Pitchford coming down to tell me that we had acquired the Homeworld franchise. Another month or so passed before all the legal machinations were finalized and we got the files from THQ.
I was eager to see the contents of the files I had only been able to see as a list in an Excel document. Given that the original game was in straight C code I imagined things would be quite different after the transition to C++. Given the issues I encountered compiling 13 year old code with a modern compiler I was interested in seeing how much would compile right out of the box. I fired up Visual Studio 2012 and got ready to open one of the projects in the Homeworld 2 source directory.
At this point it’s worth noting that building Homeworld 2 involves building numerous compiled libraries and DLL’s. One of the advantages to this is the fact that a lot of the tools for the game build and link in some of these libraries. This gave me a way in which I could start small and see what it was going to take to compile parts of the game. I started with the Homeworld 2 background editor. My first attempts to compile some of the code pointed out to me just how much the Microsoft compiler has evolved in the last 10 years.
The first challenge was that a number of warnings appeared related to Microsoft changing some standard functions to conform to ISO standards. While these issues were warnings that didn’t prevent the code from compiling a lot of them were scrolling by which made it difficult to pick through the output from the compiler. In some cases I disabled the warning and in others I revised the code that generated the warning to conform to new standards.
The next challenge was dealing with a few basic issues with the format of code that essentially slid by in the past but that modern compilers will not let slide by. Writing code is interesting in the fact that you’re dealing with a “language” that can be very unforgiving when you get it wrong. Sometimes something is obviously wrong, like ordering something in a foreign language that translates into “I would like a plate of toasters”. Other times you mix up the words for chicken with beef marrow and the results are not incorrect, but were still not what you expected.
When writing code a programmer relies on the compiler to catch errors much the same way a writer relies on spelling and grammar checking in word processors (or a human editor reviewing the story). If the code compiles, we move on and might not notice the error until running the code produces obviously incorrect results. When dealing with massive amounts of code, and the occasional typo, the chance exists that some issues will go unnoticed.
For anyone familiar with code, here are a few simple examples of things that would compile before but that would not compile now:
With the more obvious errors out of the way I got to the most time consuming issues I had to resolve. These issues were related to how C++ has changed over the years. In the earlier days of C++ becoming more wide spread, standardization of the available code and libraries for it was a bit sketchy at times. Microsoft’s early implementation of some standard libraries and conventions was a bit different than that used by other compilers. Searching on the web became my best friend when figuring out what changes were made to some of the standard libraries that were causing the code to fail to compile. The majority of the issues were related to how C++ handles lists of things. Most games keep track of numerous lists of things so there were quite a few issues related to the use of lists that I had to track down.
The time came when the above issues were resolved, the compile succeeded, and it was time to try running the game. I was a little amazed that the first time I ran Homeworld 2 the front end menu came up with working mouse cursor and sound. I could navigate through the menus and everything was working fine. It was gratifying to see all those changes culminate in something I could see and hear. With all the changes I had made I expected the game to eventually crash, which it did when I tried to start a campaign or skirmish game.
That first crash, and the few that followed, were mostly related to two things. One goes back to what I mentioned about the changes that have been made when managing lists in C++. Some of those changes caused the way Homeworld 2 sometimes goes through lists of things to not work in the same way. The other change is that more checks have been added to compiled code to identify potential problems while the game is running. There were not that many areas where this was causing a problem so after a few more changes to the code I was up and running with the 1st mission of the game.
I have to admit that, even though the game is 10 years old, it is still compelling. Now that I could play through the first mission I just kept playing through the next few missions. Everything seemed to work now and I was having fun playing through the fruits of Relic Entertainment’s past, and my more recent, labor. We didn’t have a build system set up for Homeworld at that point so, now that I had something running, I went old school and put what we used to call a “frankenbuild” on the network. I knew there were a few people here that were interested in playing Homeworld 2, and we had some artists and a designer looking at the game, so I wanted there to be an easily accessible copy for them to use.
Mere minutes had passed after sending an e-mail to the company telling everyone that a copy of the game was on the network when I got a few e-mails from people that had encountered an issue when trying to run the game. We hadn’t really started using Visual Studio 2012 for our other shipping products yet so most people at the studio did not have the runtime libraries for that version of Visual Studio. I was amazed, though, that the feedback had come so quickly.
After feeling some amount of success from this accomplishment I decided to move on to compiling Homeworld. I now had some experience with the issues I might encounter migrating older code to the latest compiler. It was easy to imagine that the solutions to many of the compile errors I had seen in my prior attempt to build the code would be easier to take care of.
I started by creating a Visual Studio 2012 project for Homeworld. Builds for Homeworld were set up to be run from the command line, so I needed to create a project file from scratch. Luckily Homeworld is built as a single executable so it was very straight forward to drop all the source files into a project and to pull the compile options from their build script. Once I could compile it I did a build and worked on fixing the compile issues. Most of the issues were related to the warnings I mentioned above where Microsoft had made changes to conform to current standards.
The most challenging issue I ran across is one where I have to put a shout out in thanks to the mod community that had been working with this code after it was released. Campaign missions in Homeworld start as script files that are preprocessed through the Microsoft compiler (a step in compiling that replaces keywords in the file fed into it with replacement text defined in a common file that all mission scripts reference). The file created by this process is then fed into a utility (kas2c.exe) that converts the script into C source that is compiled in as part of the Homeworld executable. I was a little unclear on how this step should work, but after a few internet searches I tracked down forum posts that gave me the information I needed.
The results of running Homeworld for the first time were much the same as they were when running Homeworld 2. The UI and mouse came up, I could navigate through menus and sound was working. I started up the campaign and got a crash. The reasons for the crashes I encountered when running Homeworld were quite different than they were for Homeworld 2.
When investigating the first crash I encountered the results showed crazy values for some data in the background that were way off what you would expect. These are always fun to investigate since it’s sometimes quite difficult to track down what initially caused data to get corrupted. Luckily I thought about the “old days” a bit and remembered one thing.
When compiling code there is something called “Data Structure Alignment”. In modern versions of the Microsoft compiler the alignment defaults to 8 bytes. In the past (when memory was a little tighter) the alignment would default to 4 bytes. An example of this would be eggs and an egg carton. You send two people out to two chicken coops with an “Egg Carton Alignment” of 12. One finds 10 eggs so he only has to use one egg carton and the extra two spots are just unused. The other person finds 16 eggs so they have to use two, and 8 spots are unused. If you set the “Egg Carton Alignment” to 6, the number of cartons is different as are the number of unused spots for the 2nd guy who now has 3 egg cartons with 2 empty spots in the last one. So the smaller alignment means less wasted space, but it can make things a little slower which is why 8 bytes is generally used these days.
The next crash occurred when I would destroy a target drone in the 1st mission. Homeworld has a nice system set up for the way it calls code for the effects. They had to do a few creative things to make the system work properly in older versions of Windows. The modern error checking that happens at runtime I mentioned before had some issues with a couple of these workarounds. The solution to this one was to disable that particular error since the Homeworld code worked even though it was doing something the error checking thought was a problem.
After fixing these issues Homeworld ran like a charm. I’ve played through a number of missions in both Homeworld and Homeworld 2 now and have only encountered one random crash (for which I believe I’ve found the fix). In addition to so many other things working I was also amazed to see that multiplayer still worked well, despite DirectPlay being deprecated by Microsoft.
The amount of time it took me and the number of changes I had to make to get Homeworld running was substantially shorter than it was to get Homeworld 2 running. This mostly speaks to the fact that C was pretty well set as far as standards go in 1999. C++, on the other hand, was still evolving rapidly. That and a substantially larger codebase made the process of porting Homeworld 2 a bit more involved.
It was interesting to step back in time while getting these two games up and running. I got to see a bit of history in the code and I got to reminisce a bit about some of the stages in that evolution. My first job after graduating college was doing object oriented development and C programming for Windows 2.0. After this most of the work I did was with C++ in Windows and Unix with a little bit of Objective C sprinkled in there for one or two projects.
When I started at my first game development job I went back to doing straight C programming since that’s what was used to create the engine we were working with. This trend continued with my 2nd game development job since the Quake and Quake 2 engines were coded in C. At Gearbox I’ve worked with the Half-Life engine, where the Quake code was wrapped with C++, and with the Unreal 2 and 3 engines where there are generally custom versions of standard functions and constructs.
The Homeworld 2 code base is actually the first production code base I’ve ever worked with that uses the C++ Standard Library containers to this extent. It is also my first exposure to Lua since it uses Lua script to support a very data driven architecture. There were basically a fair amount of things Homeworld 2 uses in its code that, over 18 years, I had been insulated from on a day to day basis while working with the production engines I had been working with. It has been fun working on all of this and re-exploring the past 14 years of evolution that C, C++ and Windows development have gone through.
It has also been interesting to see another approach to data driven development. At Gearbox we had always been migrating in that direction, but when a small group of us were working on Borderlands many years ago that methodology really moved forward here. Seeing the approach taken with using Lua script in creating the Homeworld 2 gameplay rules, constructs, and UI has been cool to see.
Now, however, it’s time to move on from exploring the past and to instead move on to the present and future. Some things have changed since the Homeworld games were released so now I’ve been moving forward on updating some of the infrastructure in both games. When we release both games back to the public we want to support what’s available in the present. I look forward to talking about that work…..later.
