2013 Benchmarks
In order to allow participants to easily compare the performance of their agents with participants of the 2013 competition, we benchmarked the twenty agents that participated in the 2013 Angry Birds AI competition. We ran them on the first 21 levels of the freely available Poached Eggs levels. These are the default levels when using the Chrome version of Angry Birds. Each agent had a time limit of 3 minutes per level on average, that is, a total time of 63 minutes for the 21 levels. Agents could spend the total time in any way they wanted. We used the same settings of the agents as in the competition. The following table gives an overview of the performance of the twenty agents:
Rank | Team | Total score | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Level 6 | Level 7 | Level 8 | Level 9 | Level 10 | Level 11 | Level 12 | Level 13 | Level 14 | Level 15 | Level 16 | Level 17 | Level 18 | Level 19 | Level 20 | Level 21 |
1 | AngryHex | 974670 | 30390 | 54160 | 41890 | 28000 | 64440 | 24990 | 36900 | 24860 | 49570 | 50570 | 53510 | 57750 | 42010 | 58190 | 46550 | 63430 | 46820 | 50020 | 38460 | 46970 | 65190 |
2 | WISC | 963160 | 29030 | 52180 | 41950 | 27850 | 65810 | 24640 | 25520 | 55810 | 33680 | 39170 | 50000 | 54980 | 32450 | 65060 | 49200 | 63430 | 54750 | 44740 | 37370 | 47980 | 67560 |
3 | Angry Concepts | 954030 | 29410 | 52250 | 41320 | 28160 | 64160 | 15660 | 24630 | 47170 | 48670 | 48290 | 56130 | 58600 | 50360 | 58050 | 43340 | 55000 | 45990 | 44030 | 37880 | 43730 | 61200 |
4 | Beau Rivage | 952390 | 29760 | 43160 | 40500 | 28680 | 61000 | 33540 | 45780 | 49150 | 33110 | 42200 | 57420 | 55600 | 35190 | 62770 | 47270 | 55890 | 49880 | 38820 | 29780 | 37020 | 75870 |
5 | HungryBirds | 951440 | 31210 | 53860 | 42040 | 20980 | 65700 | 28430 | 40580 | 27020 | 50710 | 53070 | 56260 | 55410 | 30130 | 59780 | 41040 | 55310 | 43530 | 46770 | 32690 | 56050 | 60870 |
6 | Luabab | 894840 | 29600 | 52180 | 40760 | 28030 | 66100 | 16860 | 32180 | 48330 | 41840 | 54110 | 45580 | 53040 | 41110 | 65640 | 28910 | 65670 | 39550 | 37680 | 16820 | 36970 | 53880 |
7 | Dan | 893370 | 29210 | 42760 | 41610 | 27990 | 63840 | 25700 | 45990 | 23390 | 45930 | 49570 | 38570 | 54990 | 32270 | 57550 | 47280 | 63000 | 42770 | 48290 | 22040 | 36910 | 53710 |
8 | Black Forest Cuckoos | 874650 | 29880 | 52390 | 41600 | 28980 | 65380 | 0 | 32380 | 56310 | 23050 | 52360 | 54360 | 57750 | 29770 | 57560 | 55300 | 59940 | 52920 | 41410 | 36940 | 46370 | 0 |
9 | IHSEV | 861830 | 29360 | 52780 | 41240 | 36810 | 51860 | 33790 | 30130 | 57570 | 24310 | 29860 | 59070 | 40960 | 42840 | 65640 | 30700 | 51730 | 47940 | 47320 | 30940 | 0 | 56980 |
10 | Naïve Agent | 858730 | 29510 | 52230 | 40620 | 20680 | 55160 | 16070 | 21590 | 25730 | 35490 | 32600 | 46760 | 54070 | 49470 | 50590 | 46430 | 55210 | 48140 | 49430 | 37920 | 36790 | 54240 |
11 | Sniper | 785520 | 29760 | 42720 | 40500 | 18620 | 65850 | 35490 | 28780 | 48170 | 26960 | 35540 | 51150 | 49190 | 26180 | 65640 | 0 | 56530 | 37980 | 40460 | 30390 | 55610 | 0 |
12 | Wanderer | 774190 | 30840 | 60400 | 41900 | 36770 | 63550 | 26910 | 22070 | 57780 | 51480 | 68740 | 30620 | 42320 | 20050 | 58200 | 25520 | 38960 | 37560 | 35220 | 25300 | 0 | 0 |
13 | s-birds | 754190 | 30080 | 26240 | 42240 | 19050 | 38160 | 36180 | 49120 | 38340 | 41670 | 54940 | 57950 | 47460 | 22550 | 63560 | 42990 | 65750 | 38790 | 39120 | 0 | 0 | 0 |
14 | Lambdaers | 705010 | 30970 | 52390 | 42070 | 19580 | 63750 | 26400 | 45880 | 57600 | 49370 | 61820 | 44120 | 38390 | 29350 | 54340 | 44040 | 0 | 44940 | 0 | 0 | 0 | 0 |
15 | A.Wang | 680640 | 22400 | 43320 | 31910 | 28080 | 48440 | 24770 | 23960 | 27580 | 42590 | 57610 | 47410 | 46980 | 27490 | 65640 | 36160 | 66550 | 39750 | 0 | 0 | 0 | 0 |
16 | Akbaba | 641080 | 28640 | 35540 | 41910 | 19120 | 47470 | 0 | 23880 | 45380 | 36960 | 56000 | 48460 | 52310 | 0 | 59420 | 0 | 47230 | 44260 | 54500 | 0 | 0 | 0 |
17 | ObjectS | 569000 | 29670 | 43200 | 40620 | 11720 | 65850 | 15570 | 21310 | 37170 | 22610 | 51250 | 57930 | 45200 | 26570 | 65640 | 34690 | 0 | 0 | 0 | 0 | 0 | 0 |
18 | AngryPKU | 476100 | 18420 | 33280 | 24070 | 10120 | 35650 | 26450 | 0 | 27840 | 24660 | 30160 | 31740 | 45870 | 0 | 45640 | 32790 | 55890 | 0 | 0 | 33520 | 0 | 0 |
19 | JulyPlayer | 352340 | 29360 | 52520 | 35360 | 20690 | 46300 | 17400 | 21810 | 45370 | 48670 | 34860 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
20 | FEI2 | 211870 | 28360 | 34160 | 30470 | 27650 | 64710 | 26520 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
High scores | 1134920 | 31210 | 60400 | 42240 | 36770 | 65850 | 36180 | 49120 | 57780 | 51480 | 68740 | 59070 | 58600 | 50360 | 65640 | 55300 | 66550 | 54750 | 54500 | 38460 | 56050 | 75870 | |
2012 High scores | 1080390 | 33340 | 60880 | 42880 | 38160 | 67630 | 32550 | 31240 | 48270 | 51570 | 58530 | 60420 | 63440 | 36940 | 65790 | 48200 | 66180 | 55150 | 57320 | 38720 | 51440 | 71740 | |
3 stars score | 1042000 | 32000 | 60000 | 41000 | 28000 | 64000 | 35000 | 45000 | 50000 | 50000 | 55000 | 54000 | 45000 | 47000 | 70000 | 41000 | 64000 | 53000 | 48000 | 35000 | 50000 | 75000 |
It is interesting to see that almost all participating agents hold at least one high score for the 21 levels. But it is clear that solving more levels is better than having high scores. The score required in order to get a three star rating is listed as well and is taken from the angrybirdsnest website. In 2 of the 21 levels, none of the agents received three stars, in level 1 and also in level 14. While the number of agents that obtained three stars could be an indication for the hardness of a level, this is clearly not the case for level 1.
The four semi-finalists of the competition are also the best performing agents in our benchmark. But surprisingly, the two agents that lost the semi-finals are ahead of the two agents that won the semi-finals. Black Forest Cuckoos pushed the Naive agent out of the top 8 agents.
We also included the high scores obtained in the 2012 benchmarks. However they cannot be compared with the 2013 scores as the method we used was different. In 2012, agents had 30 minutes for each level and the first solution obtained was recorded. None of the 2012 agents finished in under 63 minutes.