We look at the likely impact of the latest AI bot ChatGPT on product work and find out how product people are using it

---

So… chatbot software ChatGPT. Have you played with it? Looked at how you might use it in your product? Do you welcome it or worry about its impact on the work you do?

Since launching in late 2022, ChatGPT has attracted lots of noise, from supporters and detractors in equal measure.  In December Y Combinator’s Paul Graham tweeted that “the striking thing about the reaction to ChatGPT is not just the number of people who are blown away by it, but who they are. These are not people who get excited by every shiny new thing. Clearly something big is happening”. But musician Nick Cave called it “a grotesque mockery of what it is to be human” in response to ChatGPT lyrics “written in the style of Nick Cave”.

Haven’t we been here before, or somewhere like it? Whenever there’s a technology development or breakthrough that captures our collective imagination – the internet, smartphones, blockchain, whatever – people get excited. Some will hyperbolise and others will catastrophise about its future impact. But any technology only becomes a breakthrough when it crosses the chasm between being cool and being useful, comments Pat Osorio, Co-founder and CRO at customer feedback analytics platform Birdie.ai. “The metaverse and NFT, for instance, aren’t there yet. AI [artificial intelligence] definitely is.”. ChatGPT has been incredibly smart in combining the ‘cool’ with the ‘useful’, she says.

Uses in product development

Let’s be clear, ChatGPT, and large language models like it, are tools: very good language processing tools that could be very useful in product development. But they’re no more than that. They can’t have all that deep knowledge of users, their needs and pain points, that a good product manager has. Adds Pat: “For 90% of the use cases (the cool ones), it will probably be a buzz and go away. But for 10% of them, it will change how people do things as it will bring a lot of efficiencies to the table.”

Anna Buldakova, CEO and Co-founder at mentorship startup Vektor AI comments: “To do a good job, product managers need to be aware of the new technologies on the market as well as their limitations. Similarly to how PMs had to learn about mobile development, then ML products, crypto products and so on, they will need to play around with ChatGPT and decide whether it will be able to improve their product or their process.”

Graham Paterson, Product Partner at Connect Ventures, has done exactly that and has been using ChatGPT as he’s been building a new product. He says: “I’ve played around and tested it by asking a few questions. You can see its limitations and what it’s really good at relatively quickly.” We know it will, for example, very confidently give incorrect answers to simple Maths questions, so we can’t yet trust it to make decisions. He says: “It can give us something that sounds like it was well thought out by a person, but we can’t tell the accuracy.”

Graham also sees the software bringing efficiencies and being useful to speed up much of the repetitive work that comes with many product management jobs. For example, he’s currently working on a product where he’s using ChatGPT to gather information from websites. “It’s not always accurate but it’s 95% accurate,” he says, “ so my job becomes fact-checking. And fact-checking is a lot quicker than the original research.” He points out that product people often have imposter syndrome because they don’t code and they don’t design. But the important  elements of the product role, like prioritisation, are strategic and there’s little risk of them being automated.

Other product managers are adding ChatGPT to their work. Chris Wade, Senior Product Manager at Smartnumbers, has found it useful for giving him different ways to look at existing information – feeding in an article and asking it to summarise the key points for social media, for example, or feeding in information about a company and its value proposition to get ideas for a tagline or mission statement. He adds: “ChatGPT can only talk generically about things it’s already been trained on. So I wouldn’t use it to write user stories, no matter how confidently it presents the information.”

Pat Osorio says they’re using AI and NLP(natural language processing) at Birdie.ai to process qualitative user feedback at scale and identify emerging patterns in the data. She says: “We’re embedding ChatGPT as an output option in our tool where our users can ask for different prompts related to user feedback and its relationship with different aspects of product management, like the reach of a feature, opportunities for discovery and so on.”

Product people have imposter syndrome, because we can’t code we can’t design. There’s always this what do we do? And there’s this fear of, okay, but what if you can automate priorities? What if you can automate this? I think that’s quite low risk, because it’s very strategic work.

What will it change?

It will change what is expected from product managers when it comes to both hard skills and soft skills, says Pat. Product managers can use it to automate multiple small tasks in their routines and become more efficient, and skills like product intuition, stakeholder management and prioritisation will be more valued than ever.

She says: “ChatGPT will be able to summarise things, answer high-level questions, and bring context to specific problems – almost like a superpower that gives more information to support decision-making. It will also become, at some point, a requirement in terms of hard skills, just like Cloud Computing is today. We’re calling it  “Cloud Thinking”, which leads to the second point: it will change how we build products. I can see multiple products being built on top of ChatGPT or using ChatGPT as an enabler to a piece of their value delivery, just like multiple low-code or no-code solutions that are available today.”

The bigger picture

What about later on? The use of AI is an ongoing enormous and complex debate that takes intellectual property considerations, plagiarism, bias and more. But there’s no doubt that ChatGPT and other AI will have a huge impact on many knowledge-worker and creative roles over time. Says Graham Paterson: “I don’t think it’s going to happen with the AI we have today. But with the AI that we’ll have in a couple of years’ time, if it keeps getting exponentially better, it will be very interesting. It will always have limitations, but those limitations will shrink.”

For example, ChatGPT is great for coding. It can be used to write and debug code in a number of languages, and to translate one programming language to another. But as this Medium post, ChatGPT: The End of Programming (As We Know It) points out, it’s not a software developer or an engineer, but these roles will inevitably shift in focus if  – when – the use of ChatGPT and other large language models takes hold.

Creatives are understandably upset at the potential threat of AI to their livelihoods – its image generation capabilities are already very good, for example, and it will knock out a serviceable if uninspiring blog post for you in minutes. There are some interesting observations and concerns on the impact of AI for knowledge workers and creatives in this post from The Conversation, AI and the future of work.

ChatGPT’s future in product management

That’s the big picture, what about product management? It will change what is expected from product managers when it comes to both hard skills and soft skills, says Pat Osorio. Product managers can use it to automate multiple small tasks in their routines and become more efficient, and skills like product intuition, stakeholder management and prioritization will be more valued than ever.

She says: “ChatGPT will be able to summarise things, answer high-level questions, and bring context to specific problems – almost like a superpower that gives more information to support decision-making. It will also become, at some point, a requirement in terms of hard skills, just like cloud computing is today.  We’re calling it  “Cloud Thinking”, which leads to the second point: it will change how we build products. I can see multiple products being built on top of ChatGPT or using ChatGPT as an enabler to a piece of their value delivery, just like multiple low-code or no-code solutions that are available today.”

What does ChatGPT say?

Finally, what does ChatGPT itself think about its use in product management? We’ve asked it a few questions about how it could affect product management here, so do judge for yourselves. We think it highlights the software’s limitations as well as its potential uses. Let us know what you think!