2016 Benchmarks

In order to allow participants to easily compare the performance of their agents with participants of the previous competitions, we benchmarked all agents that participated in the Angry Birds AI competition. We ran them on the first 21 levels of the freely available Poached Eggs levels. These are the default levels when using the Chrome version of Angry Birds. Each agent had a time limit of 3 minutes per level on average, that is, a total time of 63 minutes for the 21 levels. Agents could spend the total time in any way they wanted. We used the same settings of the agents as in the competition. The following table gives an overview of the performance of all 47 agents that participated so far:

Rank Team Total Score Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 Level 8 Level 9 Level 10 Level 11 Level 12 Level 13 Level 14 Level 15 Level 16 Level 17 Level 18 Level 19 Level 20 Level 21
1 DataLab (2016) 1018230 29890 52180 41910 27960 58210 25120 45820 44830 51080 52570 56420 54750 41510 64940 38740 63840 50620 46610 38300 53170 79760
2 PlanA+ (2014) 1002380 30480 62370 40620 29000 69440 36970 32020 47320 26440 56830 47240 58210 34010 65640 54910 57530 51190 52120 39440 45980 64620
3 DataLab (2014) 981120 31620 52000 41890 19790 70320 15700 45720 43190 50420 56790 50650 53420 32010 55640 46450 57380 48570 45730 35470 54680 73680
4 HeartyTian 979050 28590 52120 41910 20630 67110 16960 45820 37810 49320 52660 56420 53220 49180 58200 31470 61450 51010 44990 37550 53360 69270
5 AngryHex (2013) 974670 30390 54160 41890 28000 64440 24990 36900 24860 49570 50570 53510 57750 42010 58190 46550 63430 46820 50020 38460 46970 65190
6 Datalab (2015) 973300 31430 52120 41910 27970 63660 34480 46290 54830 41490 49690 35080 51150 48130 58200 46530 58930 44870 44520 37380 42640 62000
7 WISC 963160 29030 52180 41950 27850 65810 24640 25520 55810 33680 39170 50000 54980 32450 65060 49200 63430 54750 44740 37370 47980 67560
8 AngryHex (2014) 960320 32660 52580 41910 19690 68090 25180 24680 43330 42740 54890 53570 54860 41200 57150 41100 61470 50260 48050 36780 39010 71120
9 Angry Concepts 954030 29410 52250 41320 28160 64160 15660 24630 47170 48670 48290 56130 58600 50360 58050 43340 55000 45990 44030 37880 43730 61200
10 Beau Rivage 952390 29760 43160 40500 28680 61000 33540 45780 49150 33110 42200 57420 55600 35190 62770 47270 55890 49880 38820 29780 37020 75870
11 HungryBirds 951440 31210 53860 42040 20980 65700 28430 40580 27020 50710 53070 56260 55410 30130 59780 41040 55310 43530 46770 32690 56050 60870
12 UFC 947680 31940 61680 43480 36810 66880 36080 45880 59120 49790 42630 59280 58190 48090 65640 47090 62230 45410 56860 30600 0 0
13 BamBirds 942790 30680 51420 42530 28600 65630 25260 23560 33220 50310 40570 45210 55940 36860 57740 50740 56420 49940 49740 32890 48250 67280
14 SeaBirds 942726 29540 53840 33490 29160 65620 17660 30590 58170 34970 56390 58250 53500 49050 55640 40606 61880 42290 40530 23110 39210 69230
15 AngryHex (2015) 940630 32660 52580 41910 21590 62940 33900 30110 46030 22960 63860 53570 54860 32220 53520 47120 55710 50260 42070 37040 44730 60990
16 AngryBER 935330 28510 43420 41910 19390 62880 35610 31980 45050 48670 51980 51550 54720 42510 45640 47090 50000 47740 44110 36450 35990 70130
17 RedBacks 933120 32220 53860 30480 19380 70350 33470 45740 29180 33320 59190 45250 56910 42300 45640 41540 66570 39370 48530 30460 40760 68600
18 PlanA+ (2015) 929060 30940 51450 40620 28100 48740 36690 22680 0 40970 62060 56610 57850 21260 65640 54910 56440 47540 52520 37550 35090 81400
19 AngryHex (2016) 906160 31520 52320 41870 29080 68410 35360 0 47860 46850 62670 44080 56280 49820 73960 46760 55640 51250 40840 28480 43110 0
20 Luabab 894840 29600 52180 40760 28030 66100 16860 32180 48330 41840 54110 45580 53040 41110 65640 28910 65670 39550 37680 16820 36970 53880
21 Dan 893370 29210 42760 41610 27990 63840 25700 45990 23390 45930 49570 38570 54990 32270 57550 47280 63000 42770 48290 22040 36910 53710
22 IHSEV (2014) 891590 29520 53070 41440 21860 56470 33470 29920 36630 40620 32300 31570 61070 41330 59250 34830 54160 47830 42840 40100 40470 62840
23 IHSEV (2015) 889090 28810 61390 41820 28160 63880 33820 45740 25770 33230 64270 55220 46830 48050 60100 42320 63760 46870 43380 20730 34940 0
24 Impact Vactor 886990 29710 60480 42040 19700 65440 33870 36880 25720 34770 46870 57490 55230 49620 0 45860 58870 47070 48210 25040 37650 66470
25 Tori 885450 32130 52700 41870 19410 66910 15780 21800 47820 33870 32880 40750 62290 33560 69970 45410 65760 48830 39360 0 46670 67680
26 Black Forest Cuckoos 874650 29880 52390 41600 28980 65380 0 32380 56310 23050 52360 54360 57750 29770 57560 55300 59940 52920 41410 36940 46370 0
27 IHSEV (2013) 861830 29360 52780 41240 36810 51860 33790 30130 57570 24310 29860 59070 40960 42840 65640 30700 51730 47940 47320 30940 0 56980
28 Naïve Agent (2013) 858730 29510 52230 40620 20680 55160 16070 21590 25730 35490 32600 46760 54070 49470 50590 46430 55210 48140 49430 37920 36790 54240
29 Naïve Agent (2016) 855370 29800 52610 40260 19000 54660 33830 23440 57840 25900 40750 57220 57240 29740 58340 49050 60050 47970 41590 38760 37320 0
30 Naïve Agent (2015) 838590 29760 34230 40850 13440 57640 24590 24590 43360 50710 59970 52950 55720 23070 57940 48710 63780 39280 46370 35850 35780 0
31 IHSEV (2016) 812060 28830 61390 42210 11290 67510 33820 20640 34300 40850 49980 39690 59360 34870 57860 51400 62980 41350 39240 34490 0 0
32 Sniper 785520 29760 42720 40500 18620 65850 35490 28780 48170 26960 35540 51150 49190 26180 65640 0 56530 37980 40460 30390 55610 0
33 Wanderer 774190 30840 60400 41900 36770 63550 26910 22070 57780 51480 68740 30620 42320 20050 58200 25520 38960 37560 35220 25300 0 0
34 Naïve Agent (2014) 756680 29760 43250 40180 10590 62490 14980 22150 35840 36050 52570 39310 48660 30000 45640 44190 52300 39530 39590 29460 40140 0
35 s-birds 754190 30080 26240 42240 19050 38160 36180 49120 38340 41670 54940 57950 47460 22550 63560 42990 65750 38790 39120 0 0 0
36 S-birds Avengers 706110 28520 51770 35380 27950 60920 26000 0 48010 49050 0 48320 45150 22130 53760 41240 0 45000 45750 38200 38960 0
37 Lambdaers 705010 30970 52390 42070 19580 63750 26400 45880 57600 49370 61820 44120 38390 29350 54340 44040 0 44940 0 0 0 0
38 Adil 683380 29950 51960 42820 0 69790 36150 31340 54890 48970 0 0 60670 0 71430 32910 61770 51480 0 39250 0 0
39 A.Wang 680640 22400 43320 31910 28080 48440 24770 23960 27580 42590 57610 47410 46980 27490 65640 36160 66550 39750 0 0 0 0
40 Akbaba 641080 28640 35540 41910 19120 47470 0 23880 45380 36960 56000 48460 52310 0 59420 0 47230 44260 54500 0 0 0
41 AngryDragons 620390 28120 33440 30430 28820 65650 0 0 29340 25590 0 39580 51340 30840 55640 33920 53800 45660 38390 29830 0 0
42 sBirds-returns 595530 28290 33890 41910 10230 56220 17860 22420 47820 51390 0 30560 56790 28830 0 0 46260 46910 0 31280 44870 0
43 ObjectS 569000 29670 43200 40620 11720 65850 15570 21310 37170 22610 51250 57930 45200 26570 65640 34690 0 0 0 0 0 0
44 AngryPKU 476100 18420 33280 24070 10120 35650 26450 0 27840 24660 30160 31740 45870 0 45640 32790 55890 0 0 33520 0 0
45 sBirds (2016) 461450 29760 43330 40850 19540 55070 15980 0 0 41180 47610 52040 47690 29950 0 38450 0 0 0 0 0 0
46 JulyPlayer 352340 29360 52520 35360 20690 46300 17400 21810 45370 48670 34860 0 0 0 0 0 0 0 0 0 0 0
47 FEI2 211870 28360 34160 30470 27650 64710 26520 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  3 stars 1042000 32000 60000 41000 28000 64000 35000 45000 50000 50000 55000 54000 45000 47000 70000 41000 64000 53000 48000 35000 50000 75000
  High Scores 1168020 32660 62370 43480 36810 70350 36970 49120 59120 51480 68740 59280 62290 50360 73960 55300 66570 54750 56860 40100 56050 81400

It is interesting to see that 16 agents hold at least one high score for the 21 levels, but none of the previous winners hold a high score. It is clear that solving more levels is better than having high scores and that doing well in the benchmarks does not necessarily translate to success in the competition, where all levels are unknown. The current AIBIRDS Champion Bambirds is not doing particularly well and is only ranked #13, but did solve all 21 levels. The score required in order to get a three star rating is listed as well and is taken from the angrybirdsnest website. In all 21 levels at least one agent received three stars. 

This year for the first time Datalab, the clear winner of the 2014 and 2015 competitions is also the best agent in the benchmarks. Their 2016 version got eliminated in the semi finals, but instead took the lead in the benchmarks (still not holding any individual high score) and joins PlanA+ in the prestigious club of agents with a score of over 1,000,000. 

Since 2014 the provided vision system detects the real shape of objects and also detects ground and hills. Without detecting those, some of the Poached Eggs levels above level 21 cannot be solved. Because this is now possible, we also benchmarked the 2014+ agents on the second set of 21 Poached Eggs levels. We give the result in the following table: 

Rank Team Total Score Level 2-1 Level 2-2 Level 2-3 Level 2-4 Level 2-5 Level 2-6 Level 2-7 Level 2-8 Level 2-9 Level 2-10 Level 2-11 Level 2-12 Level 2-13 Level 2-14 Level 2-15 Level 2-16 Level 2-17 Level 2-18 Level 2-19 Level 2-20 Level 2-21
1 DataLab (2015) 1066420 43780 51090 101100 46300 79660 38060 40760 50670 30370 39940 87380 40800 65940 47090 61330 52000 32240 55590 40890 0 61430
2 DataLab (2016) 1011420 42450 52940 99070 52260 81270 45850 37630 42920 40440 35740 0 42650 75270 50800 61330 58160 25160 55080 41870 0 70530
3 DataLab (2014) 876090 64050 53130 108510 52620 78810 46040 45690 45090 23390 40620 0 31570 0 0 31120 56140 29440 55800 45580 0 68490
4 HeartyTian 856440 0 50890 111010 55060 67480 69140 40590 50630 24870 43960 0 43950 65380 38260 48890 0 32060 0 0 50440 63830
5 PlanA+ (2014) 703650 0 96180 99490 0 73460 51280 53450 49200 20640 43600 53470 46720 70930 45230 0 0 0 0 0 0 0
6 AngryBER 682530 48710 0 100570 53240 74940 57350 44700 0 21910 40720 90540 0 77000 44620 28230 0 0 0 0 0 0
7 SeaBirds 665680 60200 0 98120 0 74480 44200 0 42640 31200 25830 0 42960 68060 37170 50670 0 0 52170 0 37980 0
8 sBirds 609830 52560 0 88260 0 81430 39510 43680 51200 18220 44720 83600 0 68470 38180 0 0 0 0 0 0 0
9 Naïve Agent (2016) 584290 49170 0 90200 43940 0 41380 61830 51170 45970 41000 86980 0 72650 0 0 0 0 0 0 0 0
10 IHSEV (2016) 553440 51040 47600 93390 0 0 42760 34470 52510 31430 41560 0 43170 66690 0 0 0 0 48820 0 0 0
11 AngryHex (2014) 541690 47530 0 98940 48580 76530 0 0 47170 25060 36600 87580 0 73700 0 0 0 0 0 0 0 0
12 Naïve Agent (2014) 503580 59480 0 91300 0 0 58270 51040 50300 46240 42330 0 32650 71970 0 0 0 0 0 0 0 0
13 Tori 489590 0 0 92100 0 0 49060 51990 0 44440 40650 0 36800 80010 37020 0 0 0 57520 0 0 0
14 AngryHex (2016) 488470 58270 44370 102880 48940 66670 57510 0 44820 24880 40130 0 0 0 0 0 0 0 0 0 0 0
15 AngryDragons 465740 59990 0 0 0 69900 35820 49210 57300 30470 23800 0 41140 65530 32580 0 0 0 0 0 0 0
16 IHSEV (2015) 452590 62660 0 89220   79730 43340 59690 45990 0 0 0 0 71960 0 0 0 0 0 0 0 0
17 IHSEV (2014) 440110 63350 45790 96480 0 0 56750 40880 50490 18680 26490 0 0 0 41200 0 0 0 0 0 0 0
18 RMIT RedBacks 405000 0 0 0 0 64980 0 49460 47510 23680 34470 0 0 67620 37970 0 0 0 41540 37770 0 0
19 Impact Vactor 327340 46110 0 0 49190 0 37240 48020 0 0 0 0 35880 67230 43670 0 0 0 0 0 0 0
20 BamBirds 294440 0 0 0 0 73860 49940 37080 0 25080 0 0 0 75010 33470 0 0 0 0 0 0 0
21 Naïve Agent (2015)  276070 62080 0 99020 0 69450 45520 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22 S-birds (2014) 181310 0 0 96270 56550 0 0 0 0 0 0 0 0 0 28490 0 0 0 0 0 0 0
23 UFC 150490 59690 0 90800 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
24 PlanA+ (2015) 48380 0 0 0 48380 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
25 AngryHex (2015) 27008 27008 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
26 Adil 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
27 sBirds-returns 0 0 0 92670 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  3 stars 1216000 60000 60000 102000 50000 80000 82000 50000 53000 28000 40000 69000 60000 70000 50000 62000 36000 62000 60000 47000 52000 75000
  High Scores 1332320 64050 96180 111010 56550 81430 69140 61830 57300 46240 44720 90540 46720 80010 50800 61330 58160 32240 57520 45580 50440 70530

Datalab is clearly the best agent on these levels, which are much more difficult than the first 21 levels. 12 of the agents have individual high scores and for 13 levels at least one of the agents has a three star score. From level 2-17 on, none of the agents has a three star score. Previously, Level 2-20 wasn't solved by any agent and seemed to be the most difficult level so far for AI agents. This year, two newcomers, HeartyTian and Seabirds solved that level, so now all 21 levels are solvable, but not by a single agent. 

When considering all 42 levels, team DataLab Birds is well ahead of the other agents, with the 2015 and 2016 versions achieving a score of over 2 million: 

1 DataLab (2015) 2039720
2 DataLab (2016) 2029650
3 DataLab (2014) 1857210
4 HeartyTian 1835490
5 PlanA+ (2014) 1706030
6 AngryBER 1617860
7 SeaBirds 1608406
8 AngryHex (2014) 1502010
9 Naïve Agent (2016) 1439660
10 AngryHex (2016) 1394630
11 Tori 1375040
12 IHSEV (2016) 1365500
13 IHSEV (2015) 1341680
14 RMIT RedBacks 1338120
15 IHSEV (2014) 1331700
16 Naïve Agent (2014) 1260260
17 BamBirds 1237230
18 Impact Vactor 1214330
19 Naïve Agent (2015)  1114660
20 UFC 1098170
21 AngryDragons 1086130
22 sBirds (2016) 1071280
23 PlanA+ (2015) 977440
24 AngryHex (2015) 967638
25 S-birds Avengers (2014) 887420
26 sBirds-returns 688200
27 Adil 683380