Prompt Engineer, the hottest new job in tech, is already almost obsolete

(© LeoWolfert - Canva.com)

It was foretold last year that tech jobs would shift to include working alongside generative AI. The details at the time were nebulous, but it's becoming clear AI is a fantastic tool that requires proper facilitation and management.

Times have changed even from only nine months ago, when generative AI promised to solve each company's every problem with almost zero additional effort required. According to a report by Forbes Advisor in April 2023, a modest 97% of the 600 businesses surveyed believed ChatGPT would improve their operations in at least one capacity in the near future. Funnily enough, almost half of the respondents, 43%, worried they would become too reliant on generative AI, and 35% of entrepreneurs are anxious about the technical abilities needed to use AI efficiently.

A look at recent job postings demonstrates that fears of overreliance and knowledge gaps were founded. The search for 'Prompt Engineers', serving as human-AI go-betweens, has begun — even though the existence of the role itself speaks to how unnecessary it is. After all, they're responsible for filling a large communication gulf between AI and companies when there was never meant to be one.

Discarding data scientists for Prompt Engineers

In a recent piece on ZDNet, contributing writer Joe McKendrick surveyed the current AI landscape to uncover which new jobs have the highest demand. To his surprise, it wasn't 'data scientist'. It was 'Prompt Engineer', a job that involves interfacing with a company's generative AI to ensure it's producing the best, most relevant results. The work of a Prompt Engineer, McKendrick explains, requires a multidisciplinary approach that includes conversational language fluency, an understanding of programming concepts, the ability to problem-solve on-the-fly, and more.

The existence of, and mad dash for, Prompt Engineers speaks to how generative AI isn't the magic solution everyone first thought it was. They were promised technology that paired effortlessly with what already existed to elevate work beyond what a team of humans could accomplish — in a fraction of the time, no less. Instead, companies are made to feel major FOMO on generative AI but provided no way to actually rectify that.

Now, a year on, what we do know is: generative AI isn't going to solve all of a company's problems.

Leave aside, for a minute, the fact that prompt engineers represent a stop-gap solution until folks chat with AI as one might with C3-PO, and consider that McKendrick notes in his article that companies have begun posting job openings requiring years of experience (never mind AI, in its current form, is still in its infancy) in what is undoubtedly a nascent corner of the industry. Anyone showing up for a job interview with a resume boasting a decade in the prompt engineer trenches is, at best, being far, far too generous with how they describe their previous employment.

What really makes for a good Prompt Engineer

If experience as a Prompt Engineer isn't a valuable metric in evaluating candidates for Prompt Engineer roles, what is? Companies will want to lean into this fact and consider candidates with zero tech background — linguists, psychologists, behavior specialists, and teachers. These folks hold a deep curiosity to research how humans communicate and are provided ample opportunities to test their theorems for immediate feedback.

AI is meant to be simple and intuitive, so the last thing any company would want is to place the responsibility for doing so in the hands of someone with an engineering mindset and buzzword-y vernacular. It's possible to train new hires on particular software and industry best practices, but less so to instill a people-first mindset in someone rooted firmly in technology. It's encouraging to learn that over 70% of organizations had specific training programs planned to ensure their workforces were well prepared to use generative AI tools last year, according to Accenture.

The job of a Prompt Engineer solves today's problem but leaves little room for tomorrow's solutions. Essentially, they are translating a language that hasn't fully been codified with full understanding that, when complete, they may no longer have a job. Much of this is unprecedented in tech, but the history of language spans millennia and the study of the human mind is never-ending. They are drawing from a deep knowledge pool from which to navigate these uncharted realms.

Prompt engineers are likely not worth it

Truthfully, the existence of, and dash for, prompt engineers is a red herring. The decision companies need to make isn't about who they can get to fill this role, but rather, whether they want to manage their own generative AI at all.

For mid-market companies, the most reasonable way forward is to partner with the right technology vendor. At present, only the most resource-rich enterprises can afford to build their own LLM or proprietary generative AI; the rest use, or will soon be using, an out-of-the-box vendor solution if they want quick access to AI. These vendors are incentivized to keep up with the AI craze without buying into hype cycles — the more established have long memories for similar past examples — and open their products up to any and all users regardless of tech savvy.

In matching the trust companies place in them, tech providers that are undoubtedly shaping what AI in business will look like today and in the future need to take responsibility for installing Prompt Engineers, or an equivalent, to calibrate generative AI offerings to a wide variety of use cases. They can help users fill in vast knowledge gaps by including low-code capabilities, intuitive interfaces, and a unified system within which managers can provide checks-and-balances on how AI is being used. These vendor-level Prompt Engineers will play a central role in all of this, especially looking ahead and anticipating customer needs, as they're the ones who can translate these needs into features of the company's AI.

Do prompts really require engineering?

diginomica's Jon Reed argues that companies who apply a surgical approach to incorporating AI into their workflows, via a careful selection of vendors and partners, will be the ones winning 2024 as opposed to those who rush into the latest fads head-on (a practice he labels, "AI overreach") or decide to forego the technology completely until IP and governance catches up (which it never will).

I'd include Prompt Engineers into the AI overreach category — though it's a symptom, not a cause—and concede that a flexible hire in this position, with a dynamic and nontraditional background, can soften the blow should the technology perform a zig after so many zags.

A strong vendor represents a combination of both tangible and intangible advances in technology. Engineers and programmers can match today's functionality while Prompt Engineers guide priorities and scenario-plan via customer use-cases. There is no single employee who can effectively work alongside AI; the task requires a team.