Product Tank Bellevue recently hosted a live-streamed session offering a glimpse into the world of data-centric product management. The event, facilitated by ProductTank organisers Cortney Jacobsen and Stephen Wang, featured John Mora, Chief AI Officer at Zefr. Watch the video in full, or read on for the key points discussed.

Jon has spent over 12 years leading and building data products. He shared some key learnings from leading a data team at dating app eHarmony.

From 2013-2016, eHarmony had a robust machine-learning infrastructure. Jon explained how its matching system was based on three metrics: Compatibility matching, affinity matching, and match distribution. “This system was created through linear models by a group of psychologists we employed,” he said.

In affinity matching, which Jon worked in. The team defined success as 7-day, 2-way communication. This means that within 7 days, if there were at least one message back and forth, this would be considered a successful affinity match. Finally, Match Distribution was introduced to ensure that everyone using the app would at least get a few matches.

Challenges and problems

Jon explained how this matching system had its challenges. For example, just because there was a lack of communication between a match, it didn’t necessarily mean that there was a lack of interest. A user might have just been busy during the 7 days.

When dealing with user feedback, Jon explained that “the number one complaint we got from the Customer Success team was that he was not as tall as he said he was. The number two complaint was, ‘why did you match me with this person?’” He said how the team always dug in hard to change user issues, but they were never deemed ‘good enough’. “We were trying to solve the most complex concept in the world, human love. People will always find holes,” he says.

Currently at brand advertising company Zefr, Jon’s biggest challenge is that ground truth is difficult to trust. Labelling products is a tedious task and difficult to do well. Training and developing new models is a weekly ops requirement, “if we’re not performing these weekly, then we’re losing out on insights in the world,”

How to manage data teams

When managing data-centric teams, Jon explained that the role of the product manager is slightly different to a more customer-facing product role. “They have to love the technical details on the interfaces and must produce useful documentation,” he said. Jon advocated an agile approach with two-week sprints, ensuring consistent and well-documented results—this system has worked well for the data team at Zefr.

He also shared valuable resources for product managers to learn about data, and understand how to empathise with data teams, including:

Additionally, Jon explained that data-centric product managers should be able to easily partner with the data engineering team, understand how data ties to customer needs, and have a strong empathy for the customer. “Teams will always go deep on the data, but can often lose sight of the customer,” Jon said, “it’s important for the product manager to always ask ‘does this make sense for the user?’ This is a fundamental value for any product manager, regardless of whether they are working with data teams or not,”

Finally, the product manager has to be able to tell great stories with data while also knowing when to trade off research and development. Getting into the technical weeds is imperative.

Employee personas

To help the audience better understand a standard data team hierarchy, Jon shared a helpful pyramid explaining the key roles in a data team.

• Data engineers: These are people who will live and breathe data sources and databases.
• ML Engineer: This is the glue between the produced data and the ML algorithms. A lot of the tools around ML have become approachable. They understand the tool and LLMs like Chat GPT
• Data Scientists: Finally, the data scientists are essentially researchers at heart. One of the hardest things for the product manager is ensuring that data scientists are balancing research while also being focused on business outcomes.

How we organise and manage teams is the way to build effective products. Jon notes Conway’s Law. Under this law, everyone can talk to everyone, has a mission, and all of the complexities are fed to the application whilst not being surfaced to the user.

Working with LLMs

Large language models are everywhere, Jon explained. “LLMs aren’t new for data scientists. We’ve been thinking about this for ten years,” he said, “When we think about LLMs in product management, I’m nervous because we are at the top of the hype cycle,” he explained to think about LLMs product in a user-centric way is key to surviving. LLMs can’t do everything – they can’t solve problems. Treat them as a tool, not as a solution, he added.

“The data field is moving faster than ever. Please, please, please try out LLMs, but don’t live and die by them,”

ProductTanks are informal meetups, created by Mind the Product, to bring local product people together and to enable speakers to share amazing product insights. Find your local ProductTank meetup to enjoy incredible product talks and connect with new product people – absolutely free! Use our map to get started. Take me to the map