A/B testing your icon (good, better, best)
Your app icon is like a shop front, on a very busy retail street.
Warm and welcoming*, interesting and attractive and you’ll encourage people to wander inside and check it out Drab or shabby (drabby!) and few will bother to look.
* I mean seriously, who could resist that cute a baby ?
It’s clearly something people think is worth investing in. A co-founder of a top game company told me it took them 3 months to come up with the icon for one of their monster RPGs. For our first game we tested over 64 different versions on the way to identifying a winning icon.
One of the many tests we did over the course of several weeks
So how do you actually conduct an effective a/b test to come up with a great icon?
There are several ways - here are a few approaches in order of increasing accuracy.
1/ Good
One easy way to run a very fast and cheap test is to use a service like PickFu.com. They do simple A vs B tests, typically as a front end to a crowdsource platform like Amazon Mechanical Turk, where the participants are asked which they prefer.
Pros:
- Cheap
- Very fast
- Ability to filter by demographic
Cons:
- Bias of tester self selection
- Subject to participant self awareness
- Lack of real intent and context
2/ Better
Next up you can use a platform like . This let’s you deploy multiple versions in parallel out in front of many eyeballs and measure click through percentages.
Pros:
- Fast
- Ability to target by interests, demographics
- Less bias
- Flexible sample sizes
Cons:
- Intent and context is closer but not same as real intent
- Can cost a lot depending on sample size to get statistical significance
3/ Best
The best way to test is to actually test with a real audience in the context they would discover your app once launched (ie ads and appstore charts). For iOS and Android this can mean ads on networks including AdMob, iAds, inMobi, Millenial Media etc You don’t want to use incentivized channels like offerwalls as they aren’t an accurate measure of actual self-generated user interest.
Beyond just clickthroughs, you can also measure true conversion to installs and app opening, by integrating the ad networks’ SDKs. Ie- do the players feel your game description and landing page reflects what they were interested in, which was the icon. (A whole another story on whether the game matches to what they expected when they installed)
What if you app isn’t already launched? Then you can setup a separate test account and run the ads to point there. Android is good that way. Also you can restrict the test app and campaign to certain geographies if you want to keep it fairly low key.
Pros:
- Truly unbiased and representative audience
- Accurate context
- Flexible sample sizes
- Measure actual conversion
Cons:
- Takes time to integrate SDKs, approve creatives and measure - cycles are longer
- Can cost a lot depending on sample size to get statistical significance
How do you come up with a great icon? Let me know by or