Novelty Track

This year for the first time, we introduce the AIBIRDS Novelty Track. In past AIBIRDS competitions, all game objects were known in advance and didn't change over the years. However, in the real Angry Birds game we get new objects all the time. New birds with new capabilities, new types of blocks with different behaviour, new game features, new background and so on. Humans can deal with those novelties very effectively, for AI this is much harder. Learning based systems, such as Deep Learning systems, often need a lot of training data to be able to perform well. If novelty is introduced, these systems typically need to retrain, again with lots of training data. One of the big challenges in AI is to develop systems that can deal with novelty as efficiently and as effectively as humans and adjust to it quickly. Encouraging the development of such AI systems and testing and comparing them is the purpose of our novelty track. 

There are many fundamentally different kinds of novelty, we call them novelty levels. Our aim is to introduce new novelty levels for each competition. For the first competition, we focus on three different novelty levels (according to: Ted Senator. Open World Novelty Hierarchy, in: Science of Artificial Intelligence and Learning for Open-World Novelty, BAA, 2019):

  • Novelty level 1 (Class): Previously unseen classes of objects or entities. This corresponds to new game objects with new properties, such as a new type of block that behaves differently to previous block types. These new game objects can be visually distinguished from known objects, but at first sight it is unknown how they behave.
  • Novelty level 2 (Attribute): Change in a feature of an object or entity, such as color, shape, or orientation not previously relevant to classification or action. In our competition this corresponds to modified object properties, for example, wood blocks have now twice the mass as before. Many of these novelties cannot be seen, but lead to a different game play behaviour. We will not introduce new capabilities and will not change the environment, only game objects. 
  • Novelty level 3 (Representations): Change in how entities or features are specified, corresponding to a transformation of dimensions or coordinate systems, not necessarily spatial or temporal. An example for our competition would be that screenshots will be in greyscale only rather than in colour.

We plan to run the novelty track as follows (this might change based on feedback we get from participants):

  1. For each novelty level, we introduce several novelties. For example for novelty level 1, a single new game object is one novelty, or a new block material such as clay blocks is one novelty. For novelty level 2, the specific change of one or more existing parameter values of an existing game object would be one novelty. The novelties we introduce for the competition are unknown to participants. 
  2. For every novelty we introduce, we generate game levels that each include this one particular novelty (in future competitions, game levels may include more than one novelty from different novelty levels). Game levels can include more than one of the same novel object. We randomly divide the generated game levels into a training set and a competition set. 
  3. Participants submit their novelty agents to us seven days before the competition. We take these agents and take three copies of them A0, A1, and A24. A1 will be trained for 1 hour on the training set. A24 will be trained for 24 hours on the training sets. A0 will not be trained. All this training will be done on our computers to ensure agents are comparable. Participants can train as much as they like before they submit their agents. 
  4. For every participant, all three agents, A0, A1, and A24 (after training) will enter the competition. During the competition there will be multiple elimination rounds, depending on the number of agents. In each round, agents have 30 minutes to solve 8 new levels from the competition set. Agents are ranked by their overall score, that is the sum of their personal high score for each of the 8 competition levels. The highest ranked agents proceed to the next round until a winner is determined. 
  5. There will be four different prize categories: The highest ranked agent without training, the highest ranked agent with 1 hour training, the highest ranked agent with 24 hours training, and the agent with the best combined score across all three variants. There will also be a subcategory for the best performing agent for each novelty level. 

The competition will use the Science Birds software originally developed by Lucas Ferreira. We have extended the original framework to include novelty and also offer a speed up of the gameplay by up to 50 times. Our new Science Birds framework allows you to generate and load game levels and immediately play them, that is you can easily use machine learning and deep learning approaches for developing and training your agents. We also developed a new API which not only includes screenshots, but also noisy ground truth which resembles what you could obtain from the screenshots using state of the art computer vision. Agents can use either screenshots, noisy ground truth or both. 

The current version of our software framework is available in open source together with 200 game levels and 12 novelties (5 for novelty level 1, 5 for novelty level 2, and 2 for novelty level 3). We have the same 200 game levels for each of the 12 novelties, that is a total number of 2600 levels that you can use for developing and testing your agent. We plan to release more sample novelties closer to the competition. The novelties used in the competition will be different from the sample novelties provided in advance, but of a similar type. 

We hope you find this new track appealing and hope to advance the state of the art in this important field of AI.