2017 Benchmarks

  • Print

In order to allow participants to easily compare the performance of their agents with participants of the previous competitions, we benchmarked all agents that participated in the Angry Birds AI competition. We ran them on the first 21 levels of the freely available Poached Eggs levels. These are the default levels when using the Chrome version of Angry Birds. Each agent had a time limit of 3 minutes per level on average, that is, a total time of 63 minutes for the 21 levels. Agents could spend the total time in any way they wanted. We used the same settings of the agents as in the competition. The following table gives an overview of the performance of all 57 agents that participated so far:

Rank Team Total Score Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level 8 Level 9 Level 10 Level 11 Level 12 Level 13 Level 14 Level 15 Level 16 Level 17 Level 18 Level 19 Level 20 Level 21
1 Datalab (2016) 1018230 29890 52180 41910 27960 58210 25120 45820 44830 51080 52570 56420 54750 41510 64940 38740 63840 50620 46610 38300 53170 79760
2 PlanA+ (2014) 1002380 30480 62370 40620 29000 69440 36970 32020 47320 26440 56830 47240 58210 34010 65640 54910 57530 51190 52120 39440 45980 64620
3 Datalab (2014) 981120 31620 52000 41890 19790 70320 15700 45720 43190 50420 56790 50650 53420 32010 55640 46450 57380 48570 45730 35470 54680 73680
4 HeartyTian (2016) 979050 28590 52120 41910 20630 67110 16960 45820 37810 49320 52660 56420 53220 49180 58200 31470 61450 51010 44990 37550 53360 69270
5 AngryHex (2013) 974670 30390 54160 41890 28000 64440 24990 36900 24860 49570 50570 53510 57750 42010 58190 46550 63430 46820 50020 38460 46970 65190
6 Datalab (2015) 973300 31430 52120 41910 27970 63660 34480 46290 54830 41490 49690 35080 51150 48130 58200 46530 58930 44870 44520 37380 42640 62000
7 WISC 963160 29030 52180 41950 27850 65810 24640 25520 55810 33680 39170 50000 54980 32450 65060 49200 63430 54750 44740 37370 47980 67560
8 AngryHex (2014) 960320 32660 52580 41910 19690 68090 25180 24680 43330 42740 54890 53570 54860 41200 57150 41100 61470 50260 48050 36780 39010 71120
9 Angry Concepts 954030 29410 52250 41320 28160 64160 15660 24630 47170 48670 48290 56130 58600 50360 58050 43340 55000 45990 44030 37880 43730 61200
10 Beau Rivage 952390 29760 43160 40500 28680 61000 33540 45780 49150 33110 42200 57420 55600 35190 62770 47270 55890 49880 38820 29780 37020 75870
11 HungryBirds 951440 31210 53860 42040 20980 65700 28430 40580 27020 50710 53070 56260 55410 30130 59780 41040 55310 43530 46770 32690 56050 60870
12 UFC 947680 31940 61680 43480 36810 66880 36080 45880 59120 49790 42630 59280 58190 48090 65640 47090 62230 45410 56860 30600 0 0
13 Datalab (2017) 947240 28650 52130 41910 27850 63410 33300 40610 38530 44900 51000 39030 50390 40420 58420 48390 51910 51330 50690 27840 37440 69090
14 Bambird (2016) 942790 30680 51420 42530 28600 65630 25260 23560 33220 50310 40570 45210 55940 36860 57740 50740 56420 49940 49740 32890 48250 67280
15 Seabirds 942720 29540 53840 33490 29160 65620 17660 30590 58170 34970 56390 58250 53500 49050 55640 40600 61880 42290 40530 23110 39210 69230
16 Eagles Wing (2017) 941840 32710 52400 41910 28210 41160 33300 45710 44150 24230 55840 39850 46970 20960 60850 46110 62280 52090 44000 39000 49940 80170
17 AngryHex(2015) 940630 32660 52580 41910 21590 62940 33900 30110 46030 22960 63860 53570 54860 32220 53520 47120 55710 50260 42070 37040 44730 60990
18 AngryBER 935330 28510 43420 41910 19390 62880 35610 31980 45050 48670 51980 51550 54720 42510 45640 47090 50000 47740 44110 36450 35990 70130
19 RedBacks 933120 32220 53860 30480 19380 70350 33470 45740 29180 33320 59190 45250 56910 42300 45640 41540 66570 39370 48530 30460 40760 68600
20 PlanA+(2015) 929060 30940 51450 40620 28100 48740 36690 22680 0 40970 62060 56610 57850 21260 65640 54910 56440 47540 52520 37550 35090 81400
21 PlanA+(2017) 922480 30340 51570 32490 27770 66400 26120 22630 27920 50090 40100 38250 61960 34450 65640 47310 58310 48030 50840 36050 38630 67580
22 IHSEV (2017) 915540 30010 61450 42380 28470 64300 33530 32390 38960 40710 35100 41690 51060 23440 59710 42560 62460 47080 45640 20050 45110 69440
23 AngryHex (2016) 906160 31520 52320 41870 29080 68410 35360 0 47860 46850 62670 44080 56280 49820 73960 46760 55640 51250 40840 28480 43110 0
24 Luabab 894840 29600 52180 40760 28030 66100 16860 32180 48330 41840 54110 45580 53040 41110 65640 28910 65670 39550 37680 16820 36970 53880
25 Dan 893370 29210 42760 41610 27990 63840 25700 45990 23390 45930 49570 38570 54990 32270 57550 47280 63000 42770 48290 22040 36910 53710
26 IHSEV (2014) 891590 29520 53070 41440 21860 56470 33470 29920 36630 40620 32300 31570 61070 41330 59250 34830 54160 47830 42840 40100 40470 62840
27 IHSEV(2015) 889090 28810 61390 41820 28160 63880 33820 45740 25770 33230 64270 55220 46830 48050 60100 42320 63760 46870 43380 20730 34940 0
28 Imapct Vactor 886990 29710 60480 42040 19700 65440 33870 36880 25720 34770 46870 57490 55230 49620 0 45860 58870 47070 48210 25040 37650 66470
29 Tori 885450 32130 52700 41870 19410 66910 15780 21800 47820 33870 32880 40750 62290 33560 69970 45410 65760 48830 39360 0 46670 67680
30 Black Forest Cuckoos 874650 29880 52390 41600 28980 65380 0 32380 56310 23050 52360 54360 57750 29770 57560 55300 59940 52920 41410 36940 46370 0
31 AngryHex (2017) 865470 32800 52450 41860 28150 66460 34950 0 28720 40790 70350 52270 56360 42600 57460 39170 58130 49700 48970 21240 43040 0
32 IHSEV (2013) 861830 29360 52780 41240 36810 51860 33790 30130 57570 24310 29860 59070 40960 42840 65640 30700 51730 47940 47320 30940 0 56980
33 Naïve Agent (2013) 858730 29510 52230 40620 20680 55160 16070 21590 25730 35490 32600 46760 54070 49470 50590 46430 55210 48140 49430 37920 36790 54240
34 Naïve Agent (2016) 855370 29800 52610 40260 19000 54660 33830 23440 57840 25900 40750 57220 57240 29740 58340 49050 60050 47970 41590 38760 37320 0
35 Naïve Agent (2015) 838590 29760 34230 40850 13440 57640 24590 24590 43360 50710 59970 52950 55720 23070 57940 48710 63780 39280 46370 35850 35780 0
36 IHSEV (2016) 812060 28830 61390 42210 11290 67510 33820 20640 34300 40850 49980 39690 59360 34870 57860 51400 62980 41350 39240 34490 0 0
37 Sniper 785520 29760 42720 40500 18620 65850 35490 28780 48170 26960 35540 51150 49190 26180 65640 0 56530 37980 40460 30390 55610 0
38 Bambird (2017) 774730 8870 52070 42280 28980 65990 25140 18360 47440 32420 47000 56140 53550 0 57220 50920 59530 47290 48430 33100 0 0
39 Wanderer 774190 30840 60400 41900 36770 63550 26910 22070 57780 51480 68740 30620 42320 20050 58200 25520 38960 37560 35220 25300 0 0
40 Condor 765870 30170 34380 41920 27950 56380 35320 22500 44800 39320 37730 37910 58440 25870 74170 32780 55590 41750 39950 28940 0 0
41 AngryBNU 763720 29760 62370 41870 11720 68340 25240 45890 33020 50550 34320 46870 47810 40120 0 36440 59670 48790 46780 34160 0 0
42 Naïve Agent (2014) 756680 29760 43250 40180 10590 62490 14980 22150 35840 36050 52570 39310 48660 30000 45640 44190 52300 39530 39590 29460 40140 0
43 s-birds 754190 30080 26240 42240 19050 38160 36180 49120 38340 41670 54940 57950 47460 22550 63560 42990 65750 38790 39120 0 0 0
44 SBirds (2017) 732080 28290 52290 42380 28800 65820 26860 0 29780 24890 67820 57890 58700 32210 0 42430 53430 41180 0 21440 0 57870
45 S-birds Avengers 706110 28520 51770 35380 27950 60920 26000 0 48010 49050 0 48320 45150 22130 53760 41240 0 45000 45750 38200 38960 0
46 Lambdaers 705010 30970 52390 42070 19580 63750 26400 45880 57600 49370 61820 44120 38390 29350 54340 44040 0 44940 0 0 0 0
47 Adil 683380 29950 51960 42820 0 69790 36150 31340 54890 48970 0 0 60670 0 71430 32910 61770 51480 0 39250 0 0
48 A.Wang 680640 22400 43320 31910 28080 48440 24770 23960 27580 42590 57610 47410 46980 27490 65640 36160 66550 39750 0 0 0 0
49 ValeFina007 661870 29930 43450 40370 28750 59160 25110 22840 44170 32760 0 48960 57250 20070 0 31330 59830 39480 39260 39150 0 0
50 Akbaba 641080 28640 35540 41910 19120 47470 0 23880 45380 36960 56000 48460 52310 0 59420 0 47230 44260 54500 0 0 0
51 AngryDragons 620390 28120 33440 30430 28820 65650 0 0 29340 25590 0 39580 51340 30840 55640 33920 53800 45660 38390 29830 0 0
52 sBirds-returns 595530 28290 33890 41910 10230 56220 17860 22420 47820 51390 0 30560 56790 28830 0 0 46260 46910 0 31280 44870 0
53 ObjectS 569000 29670 43200 40620 11720 65850 15570 21310 37170 22610 51250 57930 45200 26570 65640 34690 0 0 0 0 0 0
54 AngryPKU 476100 18420 33280 24070 10120 35650 26450 0 27840 24660 30160 31740 45870 0 45640 32790 55890 0 0 33520 0 0
55 sBirds (2016) 461450 29760 43330 40850 19540 55070 15980 0 0 41180 47610 52040 47690 29950 0 38450 0 0 0 0 0 0
56 JulyPlayer 352340 29360 52520 35360 20690 46300 17400 21810 45370 48670 34860 0 0 0 0 0 0 0 0 0 0 0
57 FEI2 211870 28360 34160 30470 27650 64710 26520 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  3 stars 1042000 32000 60000 41000 28000 64000 35000 45000 50000 50000 55000 54000 45000 47000 70000 41000 64000 53000 48000 35000 50000 75000
  2013 High Scores 1134960 31210 60400 42240 36810 65850 36180 49120 57780 51480 68740 59070 58600 50360 65640 55300 66550 54750 54500 38460 56050 75870
  2014 High Scores 1103820 32660 62370 42040 29000 70350 36970 45740 48010 50420 59190 57490 61070 49620 65640 54910 66570 51190 52120 40100 54680 73680
  2015 High Scores 1133350 32660 61680 43480 36810 66910 36690 46290 59120 51390 63860 59280 62290 48130 69970 54910 65760 51640 56860 37550 46670 81400
  2016 High Scores 1168020 32660 62370 43480 36810 70350 36970 49120 59120 51480 68740 59280 62290 50360 73960 55300 66570 54750 56860 40100 56050 81400
  2017 High Scores 1103530 32800 62370 42380 28980 68340 35650 45890 47440 50550 70350 57890 58700 42600 74170 50920 62460 52090 50690 39150 49940 80170
  Total High Scores 1169980 32800 62370 43480 36810 70350 36970 49120 59120 51480 70350 59280 62290 50360 74170 55300 66570 54750 56860 40100 56050 81400

It is interesting to see that 15 agents hold at least one high score for the 21 levels, but none of the previous winners hold a high score. It is clear that solving more levels is better than having high scores and that doing well in the benchmarks does not necessarily translate to success in the competition, where all levels are unknown. The current AIBIRDS Champion Eagles Wing is not doing particularly well and is only ranked #16, but did solve all 21 levels. The score required in order to get a three star rating is listed as well and is taken from the angrybirdsnest website. In all 21 levels at least one agent received three stars. 

Datalab, the clear winner of the 2014 and 2015 competitions is still the best agent in the benchmarks. Their 2016 version still holds the top spot in the benchmarks (still without any individual high score) and together with the 2014 version of PlanA+ forms the prestigious club of agents with a score of over 1,000,000. Surprisingly, none of the 2017 agents is in the top 10. But we have 3 new high scores: AngryHex in Level 1 and 10, and Condor in Level 14. 

Since 2014 the provided vision system detects the real shape of objects and also detects ground and hills. Without detecting those, some of the Poached Eggs levels above level 21 cannot be solved. Because this is now possible, we also benchmarked the 2014+ agents on the second set of 21 Poached Eggs levels. We give the result of the 37 agents in the following table: 

Rank Team Total Score Level 2-1 Level 2-2 Level 2-3 Level 2-4 Level 2-5 Level 2-6 Level 2-7 Level 2-8 Level 2-9 Level 2-10 Level 2-11 Level 2-12 Level 2-13 Level 2-14 Level 2-15 Level 2-16 Level 2-17 Level 2-18 Level 2-19 Level 2-20 Level 2-21
1 Datalab (2015) 1066420 43780 51090 101100 46300 79660 38060 40760 50670 30370 39940 87380 40800 65940 47090 61330 52000 32240 55590 40890 0 61430
2 Datalab (2017) 1060610 46830 52840 102860 30990 83050 50400 43180 47460 22300 36440 85040 35170 75540 40120 43300 58670 26250 43320 40040 32790 64020
3 Datalab (2016) 1011420 42450 52940 99070 52260 81270 45850 37630 42920 40440 35740 0 42650 75270 50800 61330 58160 25160 55080 41870 0 70530
4 Eagles Wing (2017) 896630 55020 49980 93030 45270 72120 53310 43290 44510 30250 34890 71840 41340 79220 40490 0 64240 29200 48630 0 0 0
5 DataLab (2014) 876090 64050 53130 108510 52620 78810 46040 45690 45090 23390 40620 0 31570 0 0 31120 56140 29440 55800 45580 0 68490
6 HeartyTian (2016) 856440 0 50890 111010 55060 67480 69140 40590 50630 24870 43960 0 43950 65380 38260 48890 0 32060 0 0 50440 63830
7 PlanA+ (2014) 703650 0 96180 99490 0 73460 51280 53450 49200 20640 43600 53470 46720 70930 45230 0 0 0 0 0 0 0
8 AngryBER 682530 48710 0 100570 53240 74940 57350 44700 0 21910 40720 90540 0 77000 44620 28230 0 0 0 0 0 0
9 AngryHex (2017) 668690 45980 55810 110130 55320 77190 64100 0 50520 20730 0 76290 44290 68330 0 0 0 0 0 0 0 0
10 Seabirds 665680 60200 0 98120 0 74480 44200 0 42640 31200 25830 0 42960 68060 37170 50670 0 0 52170 0 37980 0
11 PlanA+ (2017) 653720 47160 52270 95410 0 66260 37260 43520 0 0 32830 88370 52010 78100 0 0 0 0 0 0 0 60530
12 AngryBNU 618820 47530 56960 0 0 63580 37850 44180 0 45080 30200 80680 39430 71020 53020 0 0 0 0 49290 0 0
13 SBirds (2016) 609830 52560 0 88260 0 81430 39510 43680 51200 18220 44720 83600 0 68470 38180 0 0 0 0 0 0 0
14 Naïve Agent (2016) 584290 49170 0 90200 43940 0 41380 61830 51170 45970 41000 86980 0 72650 0 0 0 0 0 0 0 0
15 IHSEV (2016) 553440 51040 47600 93390 0 0 42760 34470 52510 31430 41560 0 43170 66690 0 0 0 0 48820 0 0 0
16 AngryHex (2014) 541690 47530 0 98940 48580 76530 0 0 47170 25060 36600 87580 0 73700 0 0 0 0 0 0 0 0
17 IHSEV (2017) 513740 0 45120 101170 54660 67320 46740 0 47770 17790 0 0 34460 65220 0 33490 0 0 0 0 0 0
18 Naïve Agent (2014) 503580 59480 0 91300 0 0 58270 51040 50300 46240 42330 0 32650 71970 0 0 0 0 0 0 0 0
19 Tori 489590 0 0 92100 0 0 49060 51990 0 44440 40650 0 36800 80010 37020 0 0 0 57520 0 0 0
20 AngryHex (2016) 488470 58270 44370 102880 48940 66670 57510 0 44820 24880 40130 0 0 0 0 0 0 0 0 0 0 0
21 AngryDragons 465740 59990 0 0 0 69900 35820 49210 57300 30470 23800 0 41140 65530 32580 0 0 0 0 0 0 0
22 IHSEV (2015) 452590 62660 0 89220   79730 43340 59690 45990 0 0 0 0 71960 0 0 0 0 0 0 0 0
23 IHSEV (2014) 440110 63350 45790 96480 0 0 56750 40880 50490 18680 26490 0 0 0 41200 0 0 0 0 0 0 0
24 RMIT RedBacks 405000 0 0 0 0 64980 0 49460 47510 23680 34470 0 0 67620 37970 0 0 0 41540 37770 0 0
25 Impact Vactor 327340 46110 0 0 49190 0 37240 48020 0 0 0 0 35880 67230 43670 0 0 0 0 0 0 0
26 Bambird (2016) 294440 0 0 0 0 73860 49940 37080 0 25080 0 0 0 75010 33470 0 0 0 0 0 0 0
27 ValeFina007 292060 56000 49670 0 0 63580 61980 60830 0 0 0 0 0 0 0 0 0 0 0 0 0 0
28 Naïve Agent (2015) 276070 62080 0 99020 0 69450 45520 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
29 Bambird (2017) 242150 0 44620 90070 0 0 50430 0 57030 0 0 0 0 0 0 0 0 0 0 0 0 0
30 SBirds (2017) 223710 48250 0 87240 0 0 0 43350 44870 0 0 0 0 0 0 0 0 0 0 0 0 0
31 Condor 190860 54810 41840 94210 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
32 S-birds (2014) 181310 0 0 96270 56550 0 0 0 0 0 0 0 0 0 28490 0 0 0 0 0 0 0
33 UFC 150490 59690 0 90800 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
34 sBirds-returns 92670 0 0 92670 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35 PlanA+ (2015) 48380 0 0 0 48380 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
36 AngryHex (2015) 27000 27000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
37 Adil 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  3 stars 1248000 60000 60000 102000 50000 80000 82000 50000 53000 28000 40000 69000 60000 70000 50000 62000 36000 62000 60000 47000 52000 75000
  2014 High Scores 1209020 64050 96180 108510 56550 78810 58270 53450 57300 46240 43600 90540 46720 77000 45230 31120 56140 29440 55800 45580 0 68490
  2015 High Scores 1148159 62669 51090 101100 48380 79730 49060 59690 50670 44440 40650 87380 40800 80010 47080 61330 52000 32240 57520 40890 0 61430
  2016 High Scores 1332320 64050 96180 111010 56550 81430 69140 61830 57300 46240 44720 90540 46720 80010 50800 61330 58160 32240 57520 45580 50440 70530
  2017 High Scores 1229030 56000 56960 110130 55320 83050 64100 60830 57030 45080 36440 88370 52010 79220 53020 43300 64240 29200 48630 49290 32790 64020
  Total High Scores 1351240 64050 96180 111010 56550 83050 69140 61830 57300 46240 44720 90540 52010 80010 53020 61330 64240 32240 57520 49290 50440 70530

DataLab is clearly the best agent on these levels, which are much more difficult than the first 21 levels. The 2017 champion Eagles Wing is the best of the rest. 16 of the agents have individual high scores and for 14 levels at least one of the agents has a three star score.  All 21 levels are solvable, but the 2017 version of DataLab is the only agent that can solve all levels. We have 5 new high scores from 2017 agents: DataLab in Level 5, PlanA+ in Level 12,  Eagles Wing in Level 16, and AngryBNU in Level 14 and Level 19 (which is a new three star score). 

When considering all 42 levels, team DataLab Birds is well ahead of the other agents, with the 2015-2017 versions all  achieving a score of over 2 million. Eagles Wing is number 5, just behind DataLab (2014).  

Rank Team Total Score
1 Datalab (2015) 2039720
2 Datalab (2016) 2029650
3 Datalab (2017) 2007850
4 DataLab (2014) 1857210
5 Eagles Wing (2017) 1838470
6 HeartyTian (2016) 1835490
7 PlanA+ (2014) 1706030
8 AngryBER 1617860
9 Seabirds 1608400
10 PlanA+ (2017) 1576200
11 AngryHex (2017) 1534160
12 AngryHex (2014) 1502010
13 Naïve Agent (2016) 1439660
14 IHSEV (2017) 1429280
15 AngryHex (2016) 1394630
16 AngryBNU 1382540
17 Tori 1375040
18 IHSEV (2016) 1365500
19 IHSEV (2015) 1341680
20 RMIT RedBacks 1338120
21 IHSEV (2014) 1331700
22 Naïve Agent (2014) 1260260
23 Bambird (2016) 1237230
24 Impact Vactor 1214330
25 Naïve Agent (2015) 1114660
26 UFC 1098170
27 AngryDragons 1086130
28 SBirds (2016) 1071280
29 Bambird (2017) 1016880
30 PlanA+ (2015) 977440
31 AngryHex (2015) 967638
32 Condor 956730
33 SBirds (2017) 955790
37 ValeFina007 953930
34 S-birds (2014) 935500
35 sBirds-returns 688200
36 Adil 683380