StarCraft AI Competition

The 6th annual AIIDE StarCraft AI Competition was held in October/November 2015, hosted by the University of Alberta Department of Computing Science, and organized by David Churchill and Michael Buro. This year the competition ran for two full weeks - from October 21st to November 4th, the state of the art in StarCraft AI bots competed against each other 24 hours a day in the Computing Science Dapartment at the U of A.

The tournament format for 2015 remains unchanged from previous years, with bots playing one on one in the full game of Starcraft: BroodWar using BWAPI. Bots play against each other in round-robin format with each bot playing against each other bot on each of the 10 maps. After a round robin has been played on one map, another round robin begins on the next map, and this process is repeated until the tournament time limit is reached. At the end of the tournament the results are trimmed to the end of the last completed round robin (for fairness) and bots are ranked based on their overall win percentage. No playoffs are played.

Bots are given access to read and write directories throughout the tournament in order to implement AI techniques such as learning about opponents for strategy selection. Bots can continuously write to a specific 'write' folder, and read from a specific 'read' folder. The contents of the 'write' folder are copied to the 'read' folder after each full round robin has been played - this ensures that no bot has any advantage over another bot due to individual game scheduling. The result of this process is that bots have access to information from every previously completed round. Full details of the file IO system can be seen on the rules page.

For full details on tournament format and rules, check the official competition rules section.

In order to play as many games as possible between the bots, custom Starcraft AI tournament managing software was written to allow the automatic scheduling and running of games on a local network. With many bots incorporating AI techniques which learn about opponents over time, it was very important to play as many games as possible to ensure that bots have enough time to learn as possible, and to get statistically significant results. Starcraft AI matches have a notoriously high amount of results variance over just a few games per pairing, often taking thousands of games for results to converge.

Since 2011, competitions at the UofA were held in an undergraduate computer lab utilizing 20 Windows XP machines. Since the computer lab was actively used by students, this meant that games could only be played between the end of one school term and the beginning of the next (usually at the end of August). This year the competition was run on Virtual Machines so that games could be run for much longer - thanks to a significant effort by Nicolas Barriga (another PhD student in RTS AI at the UofA). In total, 4 Linux servers each hosted 3 Windows XP virtual machines for a total of 12 virtual machines for the tournament to run on. This, combined with the fact that AIIDE was scheduled later than normal (in November) meant that we could let the competition run for a full two weeks - more than double the running time from last year. An additional advantage to virtual machines was that the competition could be monitored and controlled remotely with remote desktop software. By using KRDC through an ssh tunnel, all 12 machines were able to be simultaneously controlled and the tournament could be stopped and restarted in just a matter of minutes from home. In previous years, debugging or restarting the tournament involved driving to campus, physically walking around a computer lab and restarting the software manually.

This is the question we always get asked when we tell people we are doing a BroodWar AI competition. This competition relies completely on BWAPI as a programming interface to BroodWar. BWAPI was created by reverse engineering BroodWar and relies on reading and writing to the program memory space of BroodWar in order to read data and issue commands to the game. Since any program that does this can essentially be seen as a map hack or cheat engine, Blizzard has told us that they don't want us to do anything similar for StarCraft 2. In fact, most of the StarCraft 2 EULA specifically deals with not modifying the program in any way. We are happy that Blizzard have allowed us to continue holding tournaments using BWAPI, and they have also helped out by providing prizes to the AIIDE tournament, however until their policy changes we will not be able to do the same for StarCraft 2.

There are other RTS game engines available for competitions as well. One such engine is ORTS - a free software RTS game engine which ran several competitions until 2010 when BWAPI was released and the first AIIDE Starcraft AI Competition was held. Another engine is microRTS, a Java RTS engine which plays a simplified grid-based RTS game and is designed specifically for testing AI techniques.

For an excellent up to date overview on the state of the art in StarCraft AI techniques and bot artchitecture descriptions, I highly recommend reading the following publications:

This year, 25 teams registered to compete in the competition, however 3 bots were withdrawn after problems arose during the testing phase of the tournament. One registered bot was unable to submit the source code for the proper version of the bot (a requirement for the open source AIIDE competition) and two other bots crashed a majority of their games, and decided to withdraw rather than compete this year. With 22 entries, this was the largest competition ever for AIIDE - a huge success!

The following table lists the entrants for this year's competition. On the right-hand side of the table you will see the 'Download' column which has 3 links: 'bot' (download the bot's submitted source code), 'rep' (download the bot's replays from this tournament), and 'faq' (survey responses from each bot author giving details about each bot and describing the techniques they used). You can click on a bot name to view details about the bot and its results in past competitions - it will be displayed in place of the table below, try it out! This functionality is provided by our new Starcraft AI Competition Data Archive! You can click on one of the following links to see details about past AIIDE competitions. Note:

This year we also had the most even race distribution of any year at AIIDE. In previous years, Zerg had been poorly represented, but this year we had 5 new Zerg submissions! We also had the first ever Random race submission this year with UAlbertaBot. By choosing Random, you get randomly assigned one of the three races by Starcraft after the game has started. This means that it is harder to program a bot to play Random since you need to have strategies for all three races, however it does provide an advantage since your opponent does not know what race you are until they actually scout you in the game.

All of the 2015 detailed results can be seen here: 2015 AIIDE StarCraft Competition - Final Results.

This year the tournament ran for two full weeks on 12 machines, resulting in 20,788 total games played - more than twice the previous record of 10,251 the year before. In the graph below you can see the number of games played in every major Starcraft AI competition so far. The Computational Intelligence in Games (CIG) competition uses the same tournament managing software as AIIDE, but does not run for quite as long. And while the Student Starcraft AI tournament (SSCAI) usually has more entrants than AIIDE it uses a different tournament structure - like playing 2 or 3 rounds of round robin before having a playoff for the top few bots. This makes the results of this year's AIIDE competition the most accurate representation of bot skill so far.

The following table shows the final results for this year's tournament. This year's competition was extremely close with 1st/2nd place and 3rd/4th place being statistical ties with less than a 1% difference in win percentage between them. The three new Zerg entrants did extremely well - with tscmoo coming 1st, ZZZKBot coming 2nd, and Overkill coming 3rd. UAlbertaBot came in fourth place while playing Random which is also a great accomplishment. If you check out the detailed results you will also see that UAlbertaBot was the only bot to finish with a > 50% win rate against every other bot - even though it did not have the overall highest winning percentage. You can click on the column headers in the table below to sort the rows based on the results.

The AvgTime column indicates the average game length of a bot in minutes:seconds. You can see from these results that ZZZKBot's zergling rush strategy resulted in far quicker games than any other bot, with its average game less than half the time of the tournament average.

The Hour column shows how many games a bot took to the hour time limit, for which the game was then decided by in-game score. Games typically only go to the full hour when one bot fails to find the last remaining buildings of a vanquished opponent. This year 3.5% of all games went to the hour time limit, but the majority of these hour long games are played by the worst performing bots. This is probably because of logic bugs which the these bots that make them unable to find and/or finish off their enemies.

The Crash column indicates how many times the bot crashed during the competition. A crash in this sense is a programming error that caused the bot's program to terminate, resulting in a windows error shutting down StarCraft. The tournament manager software monitors the Starcraft.exe process and can detect when it has shut down, recording it as a crash.

The Timeout column indicates how many times a bot 'timed out', resulting in a game loss. The exact conditions for a time out are specified on our rules page, but the main two reasons are when a bot takes more than one minute on a single frame (usually caused by infinite loops), or when a bot has more than 320 frames which take more than 55ms to compute. Since this is a real-time strategy game, we want to enforce strict rules on bots taking too long to decide on what to do.

The following graph shows each bot's win percentage over time throughout the course of the competition. You can zoom in on the graph by dragging an area with the left mouse button, or you can view a larger version in a new window by clicking here. Only those bots with a > 50% win rate are shown by default, but you can show or hide any of the results by clicking a bot's name in the graph legend to the right.

This graph shows just how close the competition was, with 1st/2nd place and 3rd/4th place alternating positions several times about 2/3 of the way through the tournament. In fact, if the tournament had ended with the same number of games played as in 2014, the results for 1st/2nd and 3rd/4th would have been reversed. We can also clearly see the effects of bots that learned throughout the competition with the most exceptional example being Aiur who finished in 5th place. During the course of the competition Aiur went from 63% wins to 73% wins between rounds 20 and 90 - a huge gain of 10%. Overkill's learning also helped it gain the few percentage points necessary to overtake UAlbertaBot (which did not do any learning this year) during the last half of the competition. A graph showing each bot's wins per round can be seen by clicking here.

Each year we want to answer the question "How do the best bots perform vs. the best human players?", so we hold a human vs. machine match between the best ranking bots in the competition and an expert human player. This year our human vs. machine match's human player was Djem5, a Russian Broodwar player who is widely regarded as one of the best non-Korean Protoss players in the world. Djem5 has been playing Starcraft since 2003 and has recently won several top foreign Broodwar tournaments such as the Team Liquid Legacy Cup. This year 3 matches were played: A best-of-5 match vss tscmoo, and two best-of-3 matches vs. ZZZKBot and UAlbertaBot. It is important to note that these human vs. machine matches are a little bit unfair for the bots themselves, as they were designed to play in a tournament consisting of thousands of games, and as such the bots may be trying out risky strategies or tactics that they would not have chosen if designed to play in a shorter best-of-n type matchup where they may have implemented safer strategies.

Each match was commentated by Paul Paradies and can be watched on Youtube, or you can download the Starcraft replay files and watch in-game. Click the Show Spoiler button for a quick analysis of what happened in each game.

Game 1: Watch on Youtube
In game one of the series, Djem5 did a standard Zealot rush vs. Tscmoo who did a Hatchery first Hydralisk build. Tscmoo tried to defend the rush by placing static defenses (Sunken Colonies) near its expansion, however the human player realized that it was able to run by those defenses into the main base where it was able to kill the bot's workers, crippling its economy and eventually winning the game. This game is a good example of the type of calculation that humans are quite good at - recognizing the global impact of actions rather than concentrating on local battles. Most of the bots in the competition would have seen the sunken colonies in the expansion and decided that they were the first target that they should attack, unaware of the long-term benefits of running by to the main worker line. These sorts of long-term decision making are what make RTS games such a difficult domain for traditional AI methods.

Game 2: Watch on Youtube
In the next game, both the bot and the human went for quick expansions in order to play a more economic game. Several minutes into the game, the bot again had only built defenses in its expansion and had not yet built any mobile defensive units. I feel that this was because of my mistake in running the man vs. machine match - I had forgotten to correctly copy the learned data from the first game so that the bot could know that this strategy wasn't working vs. the human, so it tried it again a second time. In the bot vs. bot tournament, if Tscoo had found this strategy to be losing it would have switched to another one in the next game. In the end, Djem5's upgrade speed zealots were enough to take out Tscmoo's defenses and win the game.

Game 3: Watch on Youtube
The third game of the series was much more interesting as both players were for a more economic build order and Djem5 did not attack so early. Tscmoo was able to get 3 bases up and running with a large amount of Hydralisks for defense. Half way through the game, Djem5 send an attack force of Zealots to the first expansion of Tscmoo and saw an army of Hydralisks and Sunken Colonies defending, so he instead decided to take his army and attack the third base of Tscmoo instead. As he attacked the expansion, Tscmoo's Hydralisks came to defend but were not able to stop the Zealots in time and the expansion was lost. The next few minutes saw some battles between the Zealots and the Hydralisks in the middle of the map. Tscmoo does a clever combat simulation before it decides to engage in battle in an attempt to lose as few units as possible - however in this situation it ended up being its downfall. By predicting that the Hydraslisks could not defeat the Zealot army, the Hydralisks ran away from the Zealots in different directions, scattering the Zerg army across the map. By splitting his forces, Tscmoo was much weaker than had they all been in the same place. Djem5 again killed the third base of Tscmoo, built up an army of Zealots, Dragoons, and High Templars and was able to use Psionic Storm to break through the bot's defenses. This game showed just how difficult the AI problems in RTS games can be - the combat simulator which had proven quite useful in most situations throughout the tournament found itself actually being harmful in a certain situation. Deciding which AI techniques to use in a given situation is quite a hard problem.

Game 1: Watch on Youtube
ZZZKBot implements a 4-pool Zergling rush strategy every game. This means that it foregoes all economic activities and builds aggressive Zergling units as fast as possible to try and overwhelm its opponent in the first few minutes of the game. ZZZKBot did very well in the competition since this sort of early rush tactic usually requires defending with worker units - a tactic that not many bots have yet implemented. In the first game of this series Djem5 scouted ZZZKBot's base with an early worker unit and was able to see the rush coming and easily defended it with Photon Cannons at its expansion. ZZZKBot unlike Tscmoo did not have any combat simulation and just sent wave after wave of zerglings into the cannons to die. After the initial defense, Djem5 simply sent a few units to the bot's base to clean up the remaining units. Human players are very good at stopping this sort of 'cheesey' tactic and so this game was almost over before it started.

Game 2: Watch on Youtube
The exact same thing happened in game 2 of the series as this is the only real tactic that ZZZKBot employs. Now that everyone is aware of ZZZKBot's strategy I doubt that it will do so well in next year's competition.

Game 1: Watch on Youtube
In the first game of the series UAlbertaBot randomly received Protoss and both UAlbertaBot and Djem5 start out quite similarly with early Zealots. UAlbertaBot employs a strategy where it is always attacking - using the results of a combat simulation to determine when it should continue pressing the attack or to retreat. Since Djem5 was able to keep up with the Zealot production of UAlbertaBot the combat simulator told it to keep retreating, which resulted in the back and forth movement near the Djem5's base. Djem5 then implemented the perfect counter strategy against UAlbertaBot which was to form a wall of Zealots behind which he placed several ranged Dragoon units. UAlbertaBot attempted to target and attack the Dragoons of Djem5 however the wall of Zealots prevented them from passing, which makes the internal path-finding engine of Starcraft go a bit crazy. Due to this combination of bad path-finding and combat simulation retreating, UAlbertaBot lost a few early Zealots near Djem5's base and he was able to counter attack and win the game. This game shows how important problems like multi-agent pathfinding are, and how annoying it is that Starcraft's in-game pathfinding engine does not include dynamic units in its calculations, resulting in the strange behaviour near Djem5's base.

Game 2: Watch on Youtube
In the next game UAlbertaBot got the Terran race, and attempted to rush Djem5 with Marines. Djem5 formed a range-upgraded Dragoon defense and was easily able to stop the rush from UAlbertaBot. UAlbertaBot's combat simulator contained a logic error which did not factor in the range upgrade of the Dragoon units and therfore moved its Marines too close to the Dragoons on several occasions, taking some free Marine losses. This enabled Djem5 to counter attack and clean up UAlbertaBot's base shortly after. This loss was simply a matter of UAlbertaBot choosing a poor build order (Marine rush) which doesn't work very well against Protoss - and especially not well against expert human Protoss players.

It is obvious from the results of the human vs. machine match that Starcraft AI bots still have a long ways to go before beating the best expert human players. However, despite Djem5 making the bots look silly this year, several bots such as Tscmoo krasi0 have done additional testing vs. human players on ICCup in which they were able to defeat D-ranked, and even some C-ranked players. After the human players have played one or two games against the bots they are then easily able to detect and exploit small mistakes that the bots make in order to easily win the majority of games. So while bots are able to win some games against decent amateur human players, they do not do well in best-of-n game scenarios where the humans get to learn their behaviours.

Overall I believe the skill level of the bots has gone up quick quickly in the past two years. The 1st, 2nd, and 3rd place finishers in the 2014 competition placed 7th, 6th, and 10th place respectively in the 2015 competition. With the skill level bots continuing to rise, and by keeping the competition open source and fixing the blunders that the bots made this year I believe that in a year or two we will have several bots which will be able to easily defeat amateur human players. We are still a long way from beating the professionals though! For additional discussion feel free to read the concluding remarks from the 2013 report which I believe remain true today.

Thanks to everyone for competing, and I hope you enjoyed the 2015 AIIDE Starcraft AI Competition! gg