How we A/B test game art and icons

When it comes to free to play design, our  starts with creative freedom and then decides through quantitative feedback.

In practice what does that mean?   

Lots of experiments and A/B tests.  Not just for features, UI and economy - but also game art.  Here are some of the tactics we use:

App Icon 

An app icon is always lowest hanging fruit when it comes to increasing downloads.

We use , built into the Google Play Developer Console for this purpose.

An example with one of our 3d simulation games:

Here we tested different icon compositions trying to “beat” the prior champion.  One, two or no avatars? Elements from the theme, the gameplay, or just focus on characters? 

After a couple of days of data we found a single avatar and gameplay background performed best, outperforming the original icon with a high degree of confidence (it’s handy to be able to see the expected performance interval directly in the results)

In one of our older dressup games we found an icon with a “popstar feel” improved install performance over 10% relative to the original - more than we would have guessed.

Apart from icons, we also use Google Play Store Listing Experiments to A/B test other aspects - screenshots, description etc.  A handy tool!

Testing Game Art

We also test game art in two main phases of development:

1. First to experiment with concepts and see what works
   
2. Then with produced art to guide balancing and content distribution
   

Testing Concepts

Early in the design cycle our designers come up with art concepts and we get feedback on style / subject matter.

For example, in a farm prototype we wanted to add more crafting, and tested to see what players preferred to make.  In this case there was a strong preference for cakes over other types of food and drinks:

In another example we wanted to see what type of items players would want to sell next in a :

The feeling of spamming is always a concern with popups so typically we:

• configure surveys to be opt-in
• don’t ask players who are in early sessions
• only gather minimum responses needed to gain statistical confidence 
• configure polls to give virtual currency as thanks 

We use the and web dashboard for our in-game image and text surveys. 

Sometimes we want to ask questions of people outside of our app’s current audience.  For this we use , while more expensive, can be helpful especially when exploring a new audience type (you can setup qualifying questions, for example - do you play social casino games?)

Apart from surveys we also use with hardcore fans for more traditional feedback and discussions:

Testing Produced Art

Once we have art created, we then poll the finished items.  For example shoes in a fashion game:

Why?  Mainly as input for balancing and content distribution.  

For example an excellent first impression in NUX flow helps next day retention, so we plan for some of the highest rated items to appear up front.  

We also want to make sure premium bundles and items have the most desirable content to improve conversion etc.

Source -

Just like the level of excitement tends to cycle up and down through a movie or game, I think a similar approach works when it comes to arranging content in a mobile 3d game.  

A/B Testing Builds

Once everything is ready, we’ll often A/B test the new build itself using Google Play’s .  Using this, we can decide how much of install traffic will be on a new version or the control (existing one).  

If it’s a major art or feature change (for example a recent Christmas themed update during the holidays), we usually roll out 20% of our traffic and monitor over a week or more. This allows us to check at least the early retention and monetization metrics before we commit to roll it out 100%.

Conclusion

Being able to test art has had a number of benefits for our studio.  

First it lets us try lots of ideas - mild and wild - at the concept phase, which is quite liberating from a creative standpoint.

Testing also lets us have confidence before we commit to production and launch, where the costs are vastly greater.

Most of all it this feedback loop helps us get closer to the most important stakeholder of all - our players!

ps - if you want to get in touch - my

Taking on Clash of Clans - Gods of Olympus

How to kill the king?

It’s a question many midcore studios have struggled with.  

For 3 years has stubbornly stuck to the top of the grossing charts, despite the efforts of many.

There have been the dozens of with the same basic formula - build a base, mine resources, produce armies and attack others while walling up and defending your own.   There have been different themes and increasing production values, but so far no one has been able to knock Clash with its strong community, off its throne.  

In fact the only pvp game to play at the top has been Game of War, which took a very different path by setting out to be the best hardcore strategy game, re-inventing a deep warfare formula that did so well for Kabam and others in prior years.   

So is the only viable option in the genre to abandon casual accessibility and go hardcore?

Maybe not.

One new title, from Aegis Interactive, has gone the other way - stripping out complexity and going for mass accessibility.

As a game designer it’s a fun title to analyse.   

The game eschews many “proven” mechanics and feels like a fresh take on a much-loved genre.   

Let’s take a look at it’s more interesting elements:

Ditching army production

One thing a player in Gods will notice within the first few minutes is that there are no resources to mine and store, no factories to build nor build queues to produce armies.  

In fact there are no armies to produce at all.

Instead, you have a small number of Gods to fight with and upgrade.

This has a number of advantages, especially for more casual players:

• bases become simpler to build and understand
  
• no need additional time and mental effort to plan troop production each session
  
• no need for resource management, allowing you to dive into the fun of the battles
  

Ditching walls

When it comes to defense, in Gods as in other games you can build fixed structures like towers, and also fixed structures that permanently house defenses, for example archers.

They can be upgraded but don’t consume resources - they are always “full” for defenses and there is no production required - they act autonomously in your defense.  

There are also no walls in Gods.  

Whilst it’s fun to build up intricate defenses around storage units and towers, it’s also a big resource investment to commit to making (or upgrading a wall).  Players have to spend days or weeks (or more) to finish one for a full base, during which time they feel weaknesses of their build.  This of course helps a lot in monetization, but for those majority who never pay it can reduce the fun.

Instead of walls, Gods introduces houses which give proximity-based benefits to other structures, which gives fun strategy and decision making with a much lighter commitment than walls.

MOBA style attacks

With no offensive armies, attacks on other players feels more like a MOBA.

You attack with only a very small number of Gods, with full real-time control of movement, attack and special moves.   

You win when you destroy all the enemy temples - your reward is the same regardless of what else is destroyed or left standing.

This has a number of benefits:

• attacking is easier to understand - you can concentrate on your few Gods (large and unique) instead of dozens of army units
  
• your goal is simpler - destroy enemy temples.  You don’t need to think of multiple objectives - steal resources vs destroy base etc
  
• attacking feels more satisfying as you your control of the Gods is what brings you success.  You cannot rely on the AI to do well in this game, and will learn new strategies during battles
  
• you feel powerful - by having your gods viscerally tower over and destroy lots of enemy troops (reminding me of Zynga’s )
  

No backing out of attacks

One feature of traditional battle games is scoping enemies and deciding whether to attack or skip.  This can often lead to large amount of time clouding to find an ideal target, during which time you are steadily consuming resources and getting bored.  Instead, Gods lets you decide whether to face an easy, normal or hard opponent then then selects one you must attack - leaving the battle counts as a defeat.

With no armies or lasting hero health bars, there is no penalty to just giving it your best.  Whist this approach can sometimes feel a unfair when you must fight overpowered enemies (matchmaking still feels rough), on the positive side it forces you to adapt to different situations and defence builds, rather than choose the ones you are more comfortable with.

This adds to the variety and the satisfaction of mastery.

Instant actions

Whereas other battle games tend to monetize in many ways including heavy use of slots and timers (see the approach to opening Gacha boxes), Gods appears to have made a conscious decision to go a different path.

• Building is instant
  
• Upgrading structures is instant
  
• Clearing is instant
  
• etc
  

It’s a happy surprise when building your first structure to have it complete instantly, with a resounding sound effect.  The positive feedback loop is strong.

The single timer then that remains in the game is the slow stamina (nectar) recharge timer. ie- the freemium block:

Everything else is independent of time and by doing that the game feels “on your side”.  One good example is clearing environment structures, which is not only instant but also consumes no resources or workers, and also grants resources as result.

This approach seems to be helping - with positive reviews and high ratings, despite the main recharge timer being currently very strict - allowing 4 battles every 4 hours only (each lasting a minute or two).

Real-time guild co-op

With such a strict stamina charger, there is a lot of spare time between attacks on other bases. In Gods this is put to excellent use through a well designed live alliance system.

Unlike other games, right from level 1 you can instantly join an alliance and once in, you will see in real-time attacks by and on your alliance members, and even better - jump in and help:

In just seconds you are in the middle of the conflict, and can support by placing down supporting troops with cooldown - fixed troops that appear of equal strength, I think regardless of level (not sure about this).

There is no costs to the troops - they do not come out of your army or resources, they just appear and help in the defense or offense.  

More than one alliance member can come to the aid in a single battle:

This system works very well in practice:

• Newbies feel valuable - by spending their time to help they can turn the tide of battle and feel like they are contributing
  
• In turn this makes elder players more welcoming and fosters a more positive spirit
  
• Players can do light collaboration to maximise their chances - eg, announcing an attack on alliance chat 30 seconds before, and have 3 others come to join in and make a material impact in the tide of a battle.  In fact, these situations become memorable moments that are shared immediately after in alliance chat and strengthens bonds
  
• It gives something fun and worthwhile to do while waiting.  While you don’t get an spoils of the wins, you do get to feel part of something, learn new tactics and enjoy the entertainment.  It feels almost like being a spectator but where you get to get involved and help (like watching a live stream)
  

No loss from being attacked and defeated

When others attack your base and prevail, you don’t lose anything apart from some minor ranking impact.  In fact if you defeated any of their heroes in your loss, you still get a minor bonus granted.  

Whilst this reduces the intense revenge feeling when someone attacks in other games, in Gods this means you avoid the feeling of wanting to give up totally on coming back to a wrecked base, or feel it’s not fair when paying users take your effort away from you.  

This adds up to the positive impression of the game and community.

Test attacking your own and alliance bases

Another smart feature is that Gods lets you, for much lower stamina cost and no rewards, attack your own base and test your own builds out.  

You can also do the same with alliance mates (spar) which can give a lot of entertainment and relationship building while waiting.  

This provides fun additional gameplay without affecting the balancing.

Free story mode

Continuing the theme of feeling friendly, there is a free story mode that consumes no resources, and gives progressively more difficult attack and defend scenarios.  If you lose, you can instantly try again.

It’s another way to ease more casual players into the gameplay and the world of tactics - you get to play with heroes apart from your own as well.

Summary

Gods of Olympus is a well designed game that has taken an interesting approach.

It has chosen to strip out many extras and focus on its core - fun and accessible real-time battles with alliance friends.

Whilst the lack of mechanics such as timers and armies might cause a lower payer LTV compared to games like Clash, I believe it gives itself a better chance to attract and retain players.  This in turn would make a strong foundation to then introduce elder gameplay to its community over time.

So will it really challenge Clash?  It’s too early to tell - but it’s a fresh take and I think has real potential.  Now, back to playing!
 
ps - I don’t have any connection with Aegis, this is just my take as a game designer and player.  If you want to get in touch - my