Results of the IJCAI/CIG 2017 Level Generation Competition

Our second AIBIRDS Level Generation competition was held jointly as part of CIG 2017 in New York, August 22-25, 2017 and IJCAI 2017 in Melbourne, Australia August 19-25, 2017. Participants had to submit a level generation program that creates game levels under several constraints. During the competition, we generated levels for each participant and randomly selected 10 of them. This was done by a member of the organisation committee who is not involved with any of the generators. The generators were evaluated by a panel of judges according to fun and entertainment, creativity and difficulty. Fun and entertainment determined the overall ranking, while creativity and difficulty were sub-categories we ranked. Overall we had 5 participants this year:

  • Funny Quotes ft. Dominoes (Yuxuan Jiang, Ryota Ishii, Tomohiro Harada, Ruck Thawonmas from Ritsumeikan University Japan), the defending Champion from 2017
  • MCTS ft. Blocks (Yuxuan Jiang, Tomohiro Harada, Ruck Thawonmas from Ritsumeikan University Japan)
  • Scrap Maps (Ryota Ishii, Tomohiro Harada, Ruck Thawonmas from Ritsumeikan University, Japan)
  • Mathew Stephenson's Generator (Matthew Stephenson, Australian National University, Australia)
  • Lucas Ferreira's Generator (Lucas Ferreira, University of California, Santa Cruz, United States)

This means we had 50 levels to evaluate. We had 7 teams of independent judges in Melbourne and 4 teams of independent judges in New York. Each team evaluated all 50 levels and reported the results back to us in terms of scores between 0 and 10 for the five different generators. The identity of the generators and which levels belonged to which generator was kept a secret until after the judging scores were all aggregated and the ranking was finalised. We had to make sure this process was absolutely fair as two of the five generators were from members of the organisation committee.  

We did two different rankings and both led to exactly the same results. The first ranking was that we added up all scores given by the judging teams and ranked participants according to highest score. For the second ranking we only considered the ranking of the individual judging teams. For example if the "Fun ranking" of judging team 1 ranked the generators as A first and E fifth, then A received 1 point and E 5 points. If two generators were equally second, they both received 2 points and the next ranked generator received 4 points. For the second ranking, the generator with the lowest score wins. 

 

Sum over all 11 judging teams (highest score wins)    
                    Fun         Creativity          Difficulty
Matthew Stephenson's Generator 80.5 72.9 59.6
Funny Quotes ft. Dominoes 60.2 60.4 78.4
MCTS ft. Blocks   52.2 44.9 69.1
Lucas Ferreira's Generator 44.5 37.7 46.7
Scrap Maps Generator    28 28 21
         
         
Ranking of all 11 judging teams (lowest score wins)  
                    Fun         Creativity          Difficulty
Matthew Stephenson's Generator 12 15 29
Funny Quotes ft. Dominoes 24 24 17
MCTS ft. Blocks   34 35 23
Lucas Ferreira's Generator 40 38 39
Scrap Maps Generator    48 44 54

 

Matthew Stephenson's Generator won the Fun and the Creativity ranking, with Funny Quotes, the winner from 2016 second and MCTS third. The second and third teams are both from Ritsumeikan University in Japan. As can be seen, 10 of the 11 judging teams liked Matthew's generator best, one judging team who liked the Scrap Maps Generator best liked Matthew's generator second best. Therefore, the result is clear: Matthew Stephenson's Generator is the Winner of the AIBIRDS 2017 Level Generation Competition!!! Congratulations Matthew, well done!

Some judges explicitly mentioned how much they enjoyed playing Matthew's levels and that they could have continued playing. Immediately after the winners were announced at CIG, Matthew had his regular CIG paper presentation about the winning generator, a nice coincidence giving everyone the chance to hear first hand how the generator works. The second prize winner, Funny Quotes presented their CIG 2017 paper the day after. 

The difficulty ranking was won by Funny Quotes, last years overall winner, with MCTS second and Matthew's generator third. What is remarkable is that last year, the levels generated by Funny Quotes were very easy to solve. A lot of effort must have gone into making the levels hard this year, and it clearly succeeded. Interestingly, this year as well as last year, the winner of the fun and entertainment ranking is not the agent with the most difficult levels, but a generator whose levels are easier to solve. Players obviously prefer more creative levels over more difficult levels. 

One major benefit of having level generators available, is that it is now possible to create large numbers of training and test levels for developing improved AI agents for the main AIBIRDS competition. Particularly agents based on Deep Reinforcement Learning, a technique that has taken much of AI by storm and is very successful for many other games, will benefit from a large number of available levels. We hope all participants will make their generators available, so people can use them to improve their AI agents. 

This was our second level generation competition. We thank all participants for submitting sophisticated generators and all judging teams for contributing to the evaluation. We hope to continue our competition next year and encourage all interested teams to participate in this exciting challenge.