Results of the CIG 2018 Level Generation Competition
Our third AIBIRDS Level Generation competition was held in Maastricht on August 15th as part of CIG 2018. Participants in the competition had to submit a level generation program that creates game levels for Angry Birds that satisfy several constraints, such as the number of pigs or blocks types. During the competition, 50 levels were generated for each participant, of which six were then randomly selected. These selected levels were then evaluated by a panel of judges based on the categories of fun, creativity and difficulty. The fun category rankings determined the overall winner, while creativity and difficulty were sub-categories.
This year we had 6 participants:
- Funny Quotes ft. Dominoes (Yuxuan Jiang, Ryota Ishii, Tomohiro Harada, Ruck Thawonmas from Ritsumeikan University, Japan), ranked 2nd in 2017 and the winner from 2016.
- Iratus Aves (aka. MSGv2.0) (Matthew Stephenson from Australian National University, Australia), winner from 2017.
- Rube Goldberg (Yang Changeun, Shizhe Wang, Pornchok Sirisubkulchi, Ruck Thawonmas, Tomohiro Harada from Ritsumeikan University, Japan)
- Tanager (Lucas Ferreira from University of California, Santa Cruz, United States)
- Pixel Image (Jingdi Xu, Pornchok Sirisupkulchai, Yuuki Okido, Tomohiro Harada, Ruck Thawonmas from Ritsumeikan University, Japan)
- MelodyGen (Ceren Kocaogullar from Koc University, Istanbul, Turkey)
A team of eight independent judges evaluated all selected levels and reported the results back to us in terms of scores between 0 and 10 for each generator. The identity of the generators and which levels belonged to which generator was kept a secret until after the judging scores were all aggregated and the ranking was finalised.
Two different rankings were performed using these scores, and both led to exactly the same results. The first ranking was that we added up all scores given by the judging teams and ranked participants according to highest score (maximum 80 points). For the second ranking we only considered the ranking of the individual judging teams. For example if the "Fun ranking" of judging team 1 ranked the generators as A first and E fifth, then A received 1 point and E 5 points. If two generators were equally second, they both received 2 points and the next ranked generator received 4 points. For the second ranking, the generator with the lowest score wins (minimum 8 points).
Fun (sum of scores/sum of rank):
1. Iratus Aves 58/9
2. MelodyGen 46/21
3. Funny Quotes 44/23
Difficulty (sum of scores/sum of rank):
1. Iratus Aves 55/15
2. Funny Quotes 48/16
3. Pixel Image 41/23
Creativity (sum of scores/sum of rank):
1. Iratus Aves 55/11
2. MelodyGen 46/19
3. Funny Quotes 44/21
Iratus Aves by Matthew Stephenson won the Fun, Creativity and difficulty rankings, making it the clear winner of the AIBIRDS 2018 Level Generation Competition! MelodyGen came in second place overall, closely followed by Funny Quotes ft. Dominoes in third place and Tanager in fourth place. All four of these generators were developed by separate teams at different universities.
The source code for the winning generator Iratus Aves is available at: https://github.com/stepmat/IratusAves
Several of the generators this year focussed on creating visually appealing levels, rather than just on gameplay. Notable examples include the Funny Quotes and Pixel Image level generators. These levels seemed to focus less on the gameplay aspect of the generated levels, and we may consider including an additional “aesthetic” category in future competitions to reflect this.
It is also interesting that many judges found the levels generated by MelodyGen to be fun and creative, despite varying difficulty of the levels. These levels were created based on pieces of music, perhaps providing an interesting dynamic to the level.
Other generators such as Tanager and Rube Goldberg appeared to suffer from several bugs or generation errors, hence the low scores they received. We hope that this issues will be fixed in the future, and encourage all participants to resubmit again next year!
One major benefit of having level generators available, is that it is now possible to create large numbers of training and test levels for developing improved AI agents for the main AIBIRDS competition. Particularly agents based on Deep Reinforcement Learning, a technique that has taken much of AI by storm and is very successful for many other games, will benefit from a large number of available levels. We hope all participants will make their generators available, so people can use them to improve their AI agents.
This was our third level generation competition. We thank all participants for submitting sophisticated generators and all judges for contributing to their evaluation. We hope to continue our competition next year and encourage all interested teams to participate in this exciting challenge.
You can find the results of the 2017 competition here, and the results of the 2016 competition here