An end-to-end user research case study using mixed research methods.

Introduction:

EduWorld(name changed)an Educational App planned to expand its offerings, they were planning to introduce a few more apps into their product with a separate tablet specially customized for Indian kids. Hence there was a need to evaluate the usability and user experience of a new app addition among EduWorld Indian users (children) and their parents. The usability of the new type of tablet was conducted during another study. EduWorld aims to provide a variety of engaging experiences while ensuring usability and intuitiveness for young minds and their parents.

This research study, conducted by me and my research partner Nithyn Joseph, aimed to investigate the user experience of the AdventureReader app(name changed) which is an app more popular in Western countries among children aged 7–10 and their parents in India. The AdventureReader app serves as a vast digital library, offering a wide range of exciting content to fuel curiosity and boost reading confidence in young minds. It has more than 40,000 books, audiobooks, learning videos, and various other engaging resources. The app provides a safe and interactive platform for children to freely explore their interests and discover new worlds.

This would be one of the apps that would be added to the Eduworld ecosystem. By conducting an independent research study, valuable insights were gained directly from the target users, ensuring a deeper understanding of their needs and expectations.

Research Objectives:

1. To understand how users naturally explore the new EduWorld app additions.
2. To evaluate the navigation flows, ease of use, and general satisfaction of children and parents using the app.
3. To identify delighters and pain points in the user experience of the app.
4. To assess the level of parental involvement required in the use of these applications.
5. To explore other behavioral aspects associated with each application during natural usage.

Assumptions:

Assumptions to validate during research:

1. Children will find the Epic App engaging and enjoyable.
2. Parents will have specific goals and expectations for their children when using the Epic App.
3. The app’s usability is a key factor in children’s engagement and satisfaction.
4. In-app communications effectively guide children and provide relevant information.
5. There may be pain points or areas of improvement in the app’s user experience.
6. Parents’ perspectives and opinions play a role in their child’s app usage.

Research Questions:

1. What are the navigation challenges faced by children and parents while using the AdventureReader app?
2. How can personalization and content recommendations be improved to cater to individual interests and reading levels?
3. What is the impact of interactive elements on user engagement and enjoyment, and how can they be tailored to meet user preferences?
4. What features and indicators are important for tracking reading progress, achievements, and rewards within the app?
5. How can parental involvement and controls be effectively implemented to ensure a safe and supportive reading environment for children?

Research Participants:

The participants in this research were existing Eduworld users in the age group of 3.5 to 9 years spread across pan India. Our screener ensured that they belong to different levels when it comes to reading and digital literacy.

Participant selection criteria

Research Approach:

The research approach can be broken down into 5 Phases.

5 Phases of the research project — Context immersion, Research Planning, Research Execution, Research Data Analysis and Deliverables

a) Context Immersion

We wanted to understand the context, some background, and the business goals behind conducting this research. We wanted to understand what we want out of this research and why. To understand more about the product we used the app thoroughly keeping our eyes open to minute details and writing down any question related to any feature as an assumption or something to be tested during research. Then we conducted stakeholder interviews with the internal team at Eduworld(product managers, head of design, VP of Product)as well as the teams at AdventureReader to get a broader perspective of the mission before planning what to do. This helped in forming the right objectives and research questions. The idea was to align everyone

b) Research Planning & Preparation

At this phase, we knew what we need. It was time to make the plan and prepare for the research. In this phase research plan was created, participant selection criteria were decided and the recruitment plan, preparation, and execution were performed. A screener was used to screen the participants. A pilot test of the usability test was conducted with 5 participants to ensure that everything is running smoothly — technicalities, participant orientation, overall flow, and outcomes. Iterations were made to the research plan based on the pilot tests. Methodologies and logistics were decided. The discussion guides for Usability testing as well as for the Diary study were made.

c) Research Execution(Steps, Methodologies, Tasks and Metrics)

Snapshot from a moment from the Remote Usability testing.

Data Analysis started from the beginning:

Screen recording was done with a software called tldv. The recordings were stored in Google Drive. Observation and interview notes were taken in Google spreadsheets with columns dedicated to tracking individual users' behaviors, abilities, expectations, and pain points when they interact with the features while completing the tasks. Two separate columns were dedicated just to adding the Diary study learnings and the behavioral data. We were adding questions to be asked and things to be observed for each participant during the Diary study which will follow right after the usability tests.

Remote Usability Testing: Things we did during the sessions.

Remote usability testing post some rapport building with warmup questions was conducted with 15 children across the country. Post the usability test follow up follow-up interviews were conducted to understand the child's behavior, motivations, pain points, and expectations while they were using the app to perform the tasks. We also tried to gauge the perceived usefulness of the app and its features and tried to understand the potential app usage behavior as well as what can motivate or trigger the children to use the app on a regular basis.

Usability testing tasks and metrics used:

Scenario-based tasks were given to the participants. That enables the participants to imagine, use their creativity, and use the app as they would use it to fulfill different tasks while using it in their daily lives. At first, the children were given 5 minutes of time to use their app on their own while thinking out loud so that we can observe and take notes of their natural app usage without any probes.

These are the five broad areas that the usability tests were based upon —

1. Navigation and Information Accessibility:

• Task: Find a specific book in the app’s library and add it to the reading list.
• Task Metric: Time taken to locate and add the book, number of incorrect attempts, ease of finding relevant information.

2. Personalization and Content Recommendations:

• Task: Customize the app’s recommendations based on reading preferences and interests.
• Task Metric: Number of successfully personalized recommendations, ease of customization, and user satisfaction with recommended content.

3. Interactive Elements and Engagement:

• Task: Interact with the interactive elements within a book, such as animations or sound effects.
• Task Metric: User engagement level during the interaction, perceived enjoyment, ease of interacting with the elements.

4. Progress Tracking and Rewards:

• Task: View and understand the progress tracking features and rewards system within the app.
• Task Metric: Ease of understanding progress indicators, user perception of reward system effectiveness, satisfaction with progress tracking features.

5. Parental Involvement and Controls:

• Task: Set up parental controls to monitor and manage a child’s reading activities.
• Task Metric: Ease of setting up parental controls, perceived effectiveness of control features, and parent satisfaction with the level of control provided.

Diary Study:

An 8 Day Diary Study was conducted with Behavioral Data tracking. We asked the same children to keep using the app for 8 more days. We wanted the parents of the children to help with filling up a daily survey which has some questions to understand what the children did on that particular day on the app, what were their motivation level, anything new that they have observed or done, and pain points if any faced while using the app.

Why we did do the Diary Study? During usability testing, we observed that some children could not discover certain features, and could not understand how to use some of them. We also wanted to understand the motivation levels of children to use the app when left alone with the app for the next 8 days. We asked the parents to not nudge their children to use the app so that they use it only when they feel like it.

What and how? Every 3rd Day we had a Check-in with the participants to understand how they have been using the app, we didn’t give any task but asked them questions based on their behavioral data which we used to continuously monitor every day when the children used the app. Exit interviews were held on the 8th day — To build the questions to be asked in the exit interview we made sense of both Usability Testing, Diary Study, and Behavioral Data.

d) Research Data Analysis — Affinity mapping, thematic analysis

In this phase, we made sense of the combined data collected through Usability Testing, Diary Studies, and Behavioral Quantitative Data as well as the Exit Interviews Qualitative Data. From the spreadsheet, we exported everything into Miroboard and we did an Affinity mapping where we tried to understand what actionable insights we can derive out of this huge chunk of data that we have compiled over a period of almost 2 weeks — identified patterns, themes, and connections within the data — different levels of insights — Sharing some examples here:

Affinity Mapping using Miro board

Level 1: Gathering the Surface-Level Insights: In this stage, we gathered raw data from the entire research. — All the notes, verbatim statements, individual user behavioral, motivational, barriers, expectations related notes. For example, they might gather responses like:

• “ I don’t like reading books. I use my tablet only to play games”.
• “ Whatever topic my child is interested in he would go into Google and search for it”
• “The English is difficult to understand. It needs to be in Hindi or Indian accent”.

Level 2: Forming Descriptive Insights: We did this by grouping and categorizing the data based on similar themes and patterns. We identified recurring behaviors and preferences. For instance, we found clusters like:

• Cluster 1: Motivations to use the tablet
• Cluster 2: Children's behaviors to gain knowledge
• Cluster 3: Barriers to reading on the app.

Level 3: Interpretive Insights were formed: This we did to analyze the relationships and connections between the clusters to derive deeper insights. We tried to understand the reasons behind the observed behaviors and preferences. We wanted to know the why behind it. For example:

• Cluster 1: Children use the tablet only to play games because they already have lots of online learning going on since covid times, so they do not want to use the tab to read — rather for entertainment.
• Cluster 2: Children generally know how to use Google and most of them know how to use voice search as well. Even if they don’t know how to type search they would use voice search to search anything that they want to learn, read or watch. They would need a strong reason to use an app like AdventureReader.
• Cluster 3: Children who find it difficult to read or comprehend what they see in the app would not use the app for learning. They may use it for entertainment. They need the English to be of an Indian accent.

Level 4: Forming Conceptual Insights: Based on interpretive insights, researchers generate concepts or ideas for addressing the identified needs and pain points. For example:

• Concept 1: Create an effective hook within the app by introducing rewards for children to read books on the app and form the habit of reading on the app.
• Concept 2: Create content that is not easily available on Google or make children feel why reading and learning on AdventureReader is better than learning from Google.
• Concept 3: Let there be a feature on the app that can allow children to switch the content from vernacular language to Indian Accent English and then to American English.

Level 5: Strategic Insights: We provided strategic recommendations based on the insights derived from the study. Some examples are:

• Improving existing features and technologies in the app to allow kids to be able to use the app more comfortably — to understand the content, and make better use of individual features by easily discovering them and understanding to use them.
• Conducting user research to understand what kind of a reward system would be able to form reading habits for the existing Eduworld users.
• Developing marketing strategies that emphasize the unique selling points related to reading and learning using the AdventureReader app.

There were many other insights that we got from this study which of course can’t be added here in this article as you will start cursing me for making you go through 100 insights. We didn’t share everything even in our final presentation but shared them on a Google spreadsheet with all the stakeholders. In the spreadsheet, we added a prioritization level to all the insights — using the prioritization rubric framework of — relevance, severity, frequency, and effort. Later we also did an effort vs impact matrix exercise.

Building the Emerging Archetypes

Our user archetypes are a combination of both the child and their parent. Parents played a very important role in the app usage behavior of their children. But every parent had a different level when it comes to their motivations and ability to make their child use the app. The children also were different in their abilities, motivations, and abilities. We spoke to different combinations of these parents and children across India hence we thought of building some emerging archetypes out of them.

Doing an archetype analysis to form the emerging archetypes using the Miro board

We made a child's Motivation Vs Parent's ability graph and plugged in all the individual users into them — to understand where each of them falls in terms of their motivations and abilities. This helped us to understand the archetypes at a deeper level — understanding their deeper needs, what may make them use or not use the product, and their expectations, and needs.

e) Presenting the Deliverables — A Google slide presentation to the stakeholders with an Executive summary, Prioritization rubric-based insights, emerging archetypes, journey maps, and actionable insights(key takeaways and recommendations, video recordings of the usability tests and diary studies, and usage data analysis.

Some of the Deliverables

Sharing some screenshots from the final presentation that we presented to the stakeholders on the day of the presentation.

Just like we discussed earlier this is how we shared most of our insights — on the presentation we created sections for insights that come under — Pain Points of different levels of priority to solve, expectations of different levels to fulfill, insights related to perceived usefulness and usability and interesting user behaviors.

Actionable insights

Screenshot from the original presentation showing a user pain point while using the app to check the child’s learning progress with a rubric prioritization framework, user expectations, and recommendations.

Here is another screenshot from the presentation showing an important interesting user behavior where users use the download option to favorite a book instead of clicking on the favorite(heart) icon.

Screenshot from the original presentation displaying an interesting user behavior.

For some of these insights, we asked our stakeholders questions which ignited creativity in the whole team to solve problems, deep thinking, and do lots of brainstorming to impact overall growth by making the individual features of the app contribute to business growth. Questions like “What might happen if users are confused while using this feature or users use these features interchangeably” were asked.

Emerging Archetypes

There were four emerging archetypes namely — The subscriber, the outlaw, The Reading Laggards, and The Aspirers. The archetypes were formed based on their combined motivations, ability, and trigger to use or buy the app. Each of them had their individual preferences of features, behaviors, and expectations when they use the AdventureReader app.

The subscribers are the highly motivated ones to buy the app. They also have a high ability to use the app.

Screenshot from the original presentation showing the most motivated emerging archetype of all — the subscriber

The outlaws are those archetypes who look for something more innovative or out of the box. They are usually very advanced in terms of using the app.

The Aspirers are very motivated but they lack the ability and skills to use the app to learn and grow in life. They have high ambitions for the app. But they may just not have the ability to afford or even use the app.

The subscribers — expectations, value, affinity to purchase and challenges.

The laggards are those users who do not or have very low ability to use the app. In most cases, the parents also do not have the ability to help them to use the app.

We also shared stat cards like these of individual child users which gives us an idea about who the child is, their background, age, ability, motivation, and app usage behavior.

User stat cards.

Here is a user journey showing the user's app usage journey into 4 individual phases — what they do, pain points or barriers, and potential drop-off points.

A app usage user jouney map

We clearly discuss why and how a particular archetype may drop off from using the app. We brainstormed solutions to stop them from dropping off.

Important Note(just for your information): Not all participants started their Diary studies immediately post the usability tests and all the usability tests didn’t happen on the same day. There were lots of no-shows and network issues that delayed the process. Hence the entire project took us around four weeks. We kept sharing interesting daily debriefs with the stakeholders so that they are on the same page.

Other Key Findings(At a High Level):

There were many other findings. We segregated them into levels of priority — high, medium, and low so that the stakeholders know which problem or user expectation to look at or solve first.

1. Navigation and Information Accessibility:

• Participants encountered difficulties in navigating the app, especially younger children.
• Improving the visibility and accessibility of essential information, such as the bookshelf and reading log, would enhance the user experience.

2. Personalization and Content Recommendations:

• Participants expressed a desire for personalized book recommendations based on their interests and reading levels.
• Implementing a robust recommendation system that considers user preferences and allows customization would enhance engagement and satisfaction.

3. Interactive Elements and Engagement:

• Users appreciated interactive elements, such as animations and sound effects, as they added to the overall engagement and enjoyment of the app.
• However, participants also emphasized the importance of providing the option to disable or adjust the level of interactive elements to cater to individual preferences.

4. Progress Tracking and Rewards:

• Participants desired clear indicators of their reading progress, achievements, and rewards within the app.
• Implementing a comprehensive and visually appealing progress-tracking system would motivate users and provide a sense of accomplishment.

5. Parental Involvement and Controls:

• Parents emphasized the need for effective parental controls to ensure a safe and age-appropriate reading environment.
• Providing features for parents to monitor their child’s reading activities and offer feedback or recommendations would enhance their involvement and support.

Conclusion:

The independent research study conducted by me and my partner uncovered valuable insights into the user experience of the AdventureReader app. The findings highlighted challenges in navigation, the importance of personalization and content recommendations, the value of interactive elements, the need for comprehensive progress tracking, and the significance of parental involvement and controls. By incorporating these findings into the app’s design and functionality, the AdventureReader app can be enhanced to provide

Hi guys, thanks for reading this article. I know it was long but if you found this article with this case study helpful please follow me on Medium. I will keep writing similar articles for you sharing with you my opinions on topics like User Experience and Business, case studies, and other interesting topics like branding and marketing.

You can also reach out on LinkedIn. Here is my profile link.