Customer development ain’t cheap

New bootstrapped startup?

Building a new app?

Ready your credit card.

Back of the envelope costs

• Surveys to your target market $1000
• Starbucks focus groups and usability tests: $200
• Buy 1000 quality users to your new app $2000
• x 5 rounds of product iteration = $16,000

Based on

• $1 per response x 5 questions x 200 qualified respondents 
• $20 coupons x 10 people (random & Craigslist arranged)
• $2 targeted CPI x 1000 users

Conclusions

• If you’re running lean, early on it’s probably your biggest cost
• Good news: still cheaper than months of wasted design & dev effort 

Footnote

• Chances are you’ll need multiple iterations (maybe it’s just me)
• Figures excludes dev, launch marketing, hosting, pizza etc.
• Asking friends for feedback is cheap, but biased 
• You’ll 1000s of users to test LTV in a freemium product

Question - is there a cheaper way?

What’s your take?  or let me know on 

Death of a feature

Recently we bid farewell to a new feature…

It was a cute feature, one we liked.  But at our morning metrics meet we made the grim call.  When the daily Android build went out hours later, it was gone.

Sunsetting a feature - how to decide?

“Stella’s Assistant” was designed as a basic plant & harvest currency mechanic.  Players would start a job from 5 minutes to a day, and get paid once done.

   

It was a small feature, an MVP hacked together on an assumption it would increase retention, helping remind players to return to collect their coins from Stella.  

Other retention features we had launched performed well.  Yet after launching it to our usual A/B test we found retention actually decreased in the group that had Stella’s Assistant, vs the control:

And to top it off, monetization also dipped:

We first verified the test was statistically significant.  Next we thought it might have something to do with economy balance and tried with different payout values. Yet in no A/B test was the result positive.

Should I stay or should I go?  Some tips to decide

When a feature under-performs like this, there’s a binary decision to make: iterate it or drop it.   

How do you decide?  Some questions to consider…

Is the feature meeting its stated goal?

It’s good to have a goal for each feature, preferably a specific number. eg- lift 7 day retention by 5%.  Then when you release it you can see clearly whether it was successful.

In our case, the answer was no.  The numbers don’t lie and for a feature designed to improve retention, to have it dip down, albeit slightly was pretty damning.

What is the underlying reason for its under-perfomance?

Often, numbers tell the what, but not the why.  

In talking with players we found Stella’s Assistant just didn’t add much to the game.  Whilst extra coins were nice, it felt like a weird farming sim feature tacked on to a fashion game.  It also made the UI busier, and introduced an extra menu layer and more taps, pulling the overall experience down.

What is the opportunity cost of iterating?

As a startup, every developer hour counts, and the question is whether trying to improve the feature would have a bigger impact on metrics vs using that time to develop something else.  

Apart from metrics impact, you have to weigh up the ongoing additional cost of testing and support.

In this case we weren’t convinced iterating it would move the needle much, and we had several other features that would have a much better “bang for buck”.

Does it have strategic value beyond metrics?

Some features may not improve the numbers, but may still have strategic value.  

For example, a device registration system to stop cheating, an icon promoting a partner, or a parental safety feature that stops inadvertent payments in a kids game and builds long term trust.

In our case though, there just just wasn’t one.

Final tip - stay positive!

Abandoning something you have worked hard on, whether for a few hours or a few weeks is never fun.  Us humans deal pretty badly when it comes to loss, including that at our own hands.

The key though is to stay positive, and realize you may have lost a feature but you gained a real learning - one that can only improve you and your startup going forward.  

So, Stella’s Assistant rides the slow boat to Valinor and for us, the adventure continues :)

How do you decide when to kill a feature? I’d love your feedback. or .