Engineers on Yelp’s Consumer Growth team work closely with product managers, data scientists, and designers to increase user acquisition, engagement and retention to fuel the rest of the business. The team is central to growing the number of active users on the Yelp platform.

In-App Update, introduced in Android Pie, is a feature that allows Android users to update the app by showing a prompt while they’re using the app and keeps users in the app as the update is happening in the background. We thought it would be a valuable feature for consumers on Yelp and began researching and using data to determine how we would allocate our time and effort into supporting this new project. In this blog post, I’ll explain the process behind how the Consumer Growth team goes from idea to implementation to feature rollout.

Proposals: PEP and CEP

At Yelp, many product and technical projects are initiated by engineers, and engineers have two main mediums to share their ideas: Product Enhancement Proposal (PEP) and Code Enhancement Proposal (CEP) documents. A PEP is a way of sharing critical info about your feature with others on the product team such as designers, UX researchers, data scientists, and product managers. It should help guide the conversation about the impact of the feature, design, and measurement strategy instead of inefficiently putting that context in emails and issue tracking systems. A CEP is a collaboration vehicle for significant changes made to the Yelp code stack. It is an approach for building consensus around technical direction - a way of communicating how you addressed a problem with stakeholders who will not be involved in the details of the implementation (for example, senior devs or tech leads of other teams). It is left to the discretion of each engineering team to determine if something is “CEP-worthy,” depending on the technical complexity of the project.

PEP for In-App Update

Going back to the example of In-App Update mentioned earlier, our PEP covered the user/business problem, hypothesis, description, and how to measure success. The following are the core topics discussed in the In-App Update PEP.

User/business problem

Being taken away from the app for updating (e.g., going to Play Store) could lead to user churn, and In-App Update allows us to prompt a user to update their app without being taken away from the app itself. Also, based on research in the past, we’ve seen that updating the Yelp app improved the Lifetime Value of the user. Considering these two points, we presumed In-App Update would increase the monthly active users (MAU) of the app, which is a metric important to the Consumer Growth team.

Description

Based on the developer website, there are two types of updates apps can adopt: flexible and immediate.

• We want to use a flexible update mostly but add support for an immediate update in case of an emergency.
• Only show a flexible update prompt to the user if the user app version is at least four versions behind the latest released app version.
• Given our weekly release cycle, add the functionality to show the update prompt whenever we want by triggering the update from the backend.
• If the user discards the prompt, we’re not going to ask them to update again until four weeks later.

Hypothesis

The PEP hypothesis was that implementing In-App Update will increase MAU and provide a platform to trigger app updates at any time. The feature ensures that more users will take advantage of the latest features and bug fixes of the Yelp Android app.

Measuring Success

In a PEP, we must determine how we measure success. At Yelp, we use an internal tool called Bunsen to run experiments, log events, and measure success for a feature. We decided to run an A/B experiment with 50% of the users in the status quo cohort and 50% in the treatment cohort - meaning only 50% of the users will experience In-App Update during the experiment run. We usually don’t release features to 100% until we’re sure the users in treatment cohorts are not being affected negatively. Moreover, we wanted to monitor update vs. dismiss rate (participation) for the prompt to make the right trade-offs on to whom and how often we want to show the prompt. To better understand how user activity and MAU are affected, we decided to monitor the average sessions of users per day metric using Bunsen. Choosing the primary metric and deciding on the success criteria is determined by discussions among engineers, PMs, and data scientists. We said, “if In-App Update increases average sessions of users per day and does not have any negative impact on other vital metrics during the experiment run, we ship it to 100%.”

CEP for In-App Update

As CEPs tend to be more technical and intended to be shared among engineers, for the In-App Update CEP, we explained technical requirements, milestones and issues regarding performance, monitoring, cost, security and privacy, if need be.

Requirements

One of the main product requirements that would drive the proposed solution is for In-App Update to be backend-driven. Since we want to prompt the user for flexible or immediate updates whenever we want, we need help from backend developers to create an API. This API receives the latest released version code and the user app version code, then tells the client if a flexible or immediate update is needed in a single response. Additionally, we need to figure out a way to calculate if a user is four (or X) versions behind by reading our app version code.

Milestones

Eventually we need to go from having a list of things to do, to having a list of cost & time estimates for each thing to do, and then to having some idea of what the schedule might look like for delivering. We settled on the following project milestones:

Milestone Description Owner ETA M0 Write PEP & CEP EngineerA 1/1 M1 Make it easier to calculate how many app versions behind a user is from the latest update by following semantic versioning EngineerB 1/7 M2 Implement the backend API. This endpoint gets the user app version and latest released app version and determines if a flexible or immediate update is needed EngineerC 1/10 M3 On client, implement InAppUpdateManager class in the Support module to handle everything related to updates in Android codebase EngineerA 1/10 M4 Set up experiment and event logging EngineerA 1/12 M5 Measure success EngineerA, PM 1/20

Implementation

The client-side implementation details are outlined in the developer website and are straightforward. The In-App Update flexible prompt for Yelp is shown below.

On the home screen, we make a call to our backend API with the user’s app version and latest app version available on the Play Store. The backend figures out if the update prompt should be shown or not, and lets the client know in a single response. It’s also the backend that decides whether immediate or flexible prompts should be shown, making it a backend-driven feature which gives us the flexibility to change to whom we want to show the prompts and when without an app release.

Result

We monitored user interaction with the feature over 30 days using Bunsen by running an A/B experiment. We created two cohorts, with 50% of our users assigned in the status quo cohort and the other 50% in the treatment cohort. For users that were in the treatment cohort and interacted with the In-App Update feature, we saw a lift in our primary metric (average sessions of users per day) in 30 days.

By manual testing the feature, we realized In-App Update would provide a better user experience for users when updating the app by presenting a reliable flow in which they don’t have to leave the app. The experimentation process gave us greater confidence in shipping the feature to 100 percent. It also informed our decision to limit showing the prompt to more select users so as not to inconvenience their app experience. For example, we now only show the prompt to users who are at least eight app versions behind. Doing so was super easy, and was achieved by changing server config files in the backend.

Acknowledgments

In this blog post we talked about how engineers at Yelp Consumer Growth team use CEPs and PEPs to present their ideas to the team and go from ideation to implementation, and gave you some insight on how we use data-driven development to measure success for a project.

Shoutouts to the following Yelpers for their support during the project: Aditya Vaidyanathan, David Kuo, Matthew Page, Sam Faber-Manning, Rishi Goel, and Rajeev Sarvaria

Written by Arash Nase (@_arashism)

Back to blog