2013 Benchmarks

  • Print

In order to allow participants to easily compare the performance of their agents with participants of the 2013 competition, we benchmarked the twenty agents that participated in the 2013 Angry Birds AI competition. We ran them on the first 21 levels of the freely available Poached Eggs levels. These are the default levels when using the Chrome version of Angry Birds. Each agent had a time limit of 3 minutes per level on average, that is, a total time of 63 minutes for the 21 levels. Agents could spend the total time in any way they wanted. We used the same settings of the agents as in the competition. The following table gives an overview of the performance of the twenty agents:

Rank Team Total score Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level 8 Level 9 Level 10 Level 11 Level 12 Level 13 Level 14 Level 15 Level 16 Level 17 Level 18 Level 19 Level 20 Level 21
1 AngryHex 974670 30390 54160 41890 28000 64440 24990 36900 24860 49570 50570 53510 57750 42010 58190 46550 63430 46820 50020 38460 46970 65190
2 WISC 963160 29030 52180 41950 27850 65810 24640 25520 55810 33680 39170 50000 54980 32450 65060 49200 63430 54750 44740 37370 47980 67560
3 Angry Concepts 954030 29410 52250 41320 28160 64160 15660 24630 47170 48670 48290 56130 58600 50360 58050 43340 55000 45990 44030 37880 43730 61200
4 Beau Rivage 952390 29760 43160 40500 28680 61000 33540 45780 49150 33110 42200 57420 55600 35190 62770 47270 55890 49880 38820 29780 37020 75870
5 HungryBirds 951440 31210 53860 42040 20980 65700 28430 40580 27020 50710 53070 56260 55410 30130 59780 41040 55310 43530 46770 32690 56050 60870
6 Luabab 894840 29600 52180 40760 28030 66100 16860 32180 48330 41840 54110 45580 53040 41110 65640 28910 65670 39550 37680 16820 36970 53880
7 Dan 893370 29210 42760 41610 27990 63840 25700 45990 23390 45930 49570 38570 54990 32270 57550 47280 63000 42770 48290 22040 36910 53710
8 Black Forest Cuckoos 874650 29880 52390 41600 28980 65380 0 32380 56310 23050 52360 54360 57750 29770 57560 55300 59940 52920 41410 36940 46370 0
9 IHSEV 861830 29360 52780 41240 36810 51860 33790 30130 57570 24310 29860 59070 40960 42840 65640 30700 51730 47940 47320 30940 0 56980
10 Naïve Agent 858730 29510 52230 40620 20680 55160 16070 21590 25730 35490 32600 46760 54070 49470 50590 46430 55210 48140 49430 37920 36790 54240
11 Sniper 785520 29760 42720 40500 18620 65850 35490 28780 48170 26960 35540 51150 49190 26180 65640 0 56530 37980 40460 30390 55610 0
12 Wanderer 774190 30840 60400 41900 36770 63550 26910 22070 57780 51480 68740 30620 42320 20050 58200 25520 38960 37560 35220 25300 0 0
13 s-birds 754190 30080 26240 42240 19050 38160 36180 49120 38340 41670 54940 57950 47460 22550 63560 42990 65750 38790 39120 0 0 0
14 Lambdaers 705010 30970 52390 42070 19580 63750 26400 45880 57600 49370 61820 44120 38390 29350 54340 44040 0 44940 0 0 0 0
15 A.Wang 680640 22400 43320 31910 28080 48440 24770 23960 27580 42590 57610 47410 46980 27490 65640 36160 66550 39750 0 0 0 0
16 Akbaba 641080 28640 35540 41910 19120 47470 0 23880 45380 36960 56000 48460 52310 0 59420 0 47230 44260 54500 0 0 0
17 ObjectS 569000 29670 43200 40620 11720 65850 15570 21310 37170 22610 51250 57930 45200 26570 65640 34690 0 0 0 0 0 0
18 AngryPKU 476100 18420 33280 24070 10120 35650 26450 0 27840 24660 30160 31740 45870 0 45640 32790 55890 0 0 33520 0 0
19 JulyPlayer 352340 29360 52520 35360 20690 46300 17400 21810 45370 48670 34860 0 0 0 0 0 0 0 0 0 0 0
20 FEI2 211870 28360 34160 30470 27650 64710 26520 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  High scores 1134920 31210 60400 42240 36770 65850 36180 49120 57780 51480 68740 59070 58600 50360 65640 55300 66550 54750 54500 38460 56050 75870
  2012 High scores 1080390 33340 60880 42880 38160 67630 32550 31240 48270 51570 58530 60420 63440 36940 65790 48200 66180 55150 57320 38720 51440 71740
  3 stars score 1042000 32000 60000 41000 28000 64000 35000 45000 50000 50000 55000 54000 45000 47000 70000 41000 64000 53000 48000 35000 50000 75000

It is interesting to see that almost all participating agents hold at least one high score for the 21 levels. But it is clear that solving more levels is better than having high scores. The score required in order to get a three star rating is listed as well and is taken from the angrybirdsnest website. In 2 of the 21 levels, none of the agents received three stars, in level 1 and also in level 14. While the number of agents that obtained three stars could be an indication for the hardness of a level, this is clearly not the case for level 1. 

The four semi-finalists of the competition are also the best performing agents in our benchmark. But surprisingly, the two agents that lost the semi-finals are ahead of the two agents that won the semi-finals. Black Forest Cuckoos pushed the Naive agent out of the top 8 agents. 

We also included the high scores obtained in the 2012 benchmarks. However they cannot be compared with the 2013 scores as the method we used was different. In 2012, agents had 30 minutes for each level and the first solution obtained was recorded. None of the 2012 agents finished in under 63 minutes.