How to use A/B tests to maximize your app’s revenue

January 19, 2024 by Annyce Davis 0 Comments

Show the annual plan first? Or the monthly? Which should you choose? It depends. That’s why you must use A/B tests to prove or disapprove your assumptions about any changes to your app’s checkout experience. Let’s see how it’s done.

Define a goal

The first step in this process is to define your goal. When you’re considering changes to your app’s checkout experience it’s in service of one or more of the following:

An increase in one metric could likely lead to a decrease in the other. Therefore, you need to understand your desired outcome. For instance, your subscriber LTV helps set marketing budgets and ensures that you can target the most effective users.

Whereas, your recognized revenue is an accounting standard that helps to ensure an apples-to-apples comparison can be made between companies when reviewing the income statement. Additionally, this value is often passed down from your finance department as it relates to quarterly and yearly financial targets for your team.

Create a hypothesis

Let’s say your goal is to increase subscriber LTV. One small optimization could be the order in which the subscription plans are presented. That brings us back to the opening question. Monthly or annual plan first?

Perhaps your gut is telling you that showing the monthly plan first will increase LTV more than having the annual plan first. Then that becomes your hypothesis. That’s exactly what my team and I thought after we saw the total number of subscribers decline after introducing an annual plan.

Screenshot of the two variations of the A/B test

Set up your A/B test

To prove or disprove your hypothesis, you should run an A/B test. If you’re not familiar with A/B testing yet, this is a straightforward article on the subject. There are various solutions you can use to run an A/B test on native apps. If you need something simple and free, go with Firebase Remote Config Experiments. If you have money and want a robust solution, LaunchDarkly is a solid choice.

At my company, we have an in-house application. However, most A/B testing solutions have the following setup:

1. Name the experiment: 1024-test-apps-plan-order
2. Define the split between your variants: 50/50
3. Choose what percentage of your users you want to be enrolled: 100%
4. Add a call in your app to enroll your user and determine which variant they will receive
5. Track the behavior of users

val isInExperiment = experimentService.getExperiment(
  "1024-test-apps-plan-order", userId
)

if (isInExperiment) {
  // show monthly plan first
} else {
  // show annual plan first
}

Analyze the results

Variant	# Enrolled	# Trialists	Conversion	Lift
control	33,726	785	2.33%	–
variant	33,985	830	2.44%	4.93%

Final A/B test results

From the results, we saw a 4.93% increase in the number of trialists by placing the monthly plan first. Seems like we have our winner. Not so fast!

After digging into the total number of users who selected the annual plan vs. the monthly plan, we decided to stick with the control. Why is that? The LTV.

Variant	Blended LTV	Lift
control	$50.04	–
variant	$46.63	-6.81%

Impact on the LTV based on the plan mix

After analyzing the plan mix we saw a -6.81% lift in the Blended LTV. This is the opposite of our desired goal. Our goal was to increase the LTV, not the number of trialists or MRR. Having the monthly plan first is clearly not what we should do.

What have we learned? It’s imperative that you A/B test changes to your app’s checkout experience. Going with your gut could cause you to lose out on meaningful revenue in the long run.