Screenshot of a bunch of the comments left on the article a professional editor reviewed.

Look, I’ll be honest: I had kind of changed my mind about AI text generation.

When I first reviewed the best AI text generation tools, I was very annoyed about the hype because I asked four of the leading AI text generators to make a blog post and all four of them posted worthless garbage I wouldn’t give away to my worst nemesis. And they charge you for the pleasure!

Maybe due to the friendlier interface, maybe because I learned how to use AI better, but ChatGPT set itself apart. I liked it. I got along with it.

Finally, the day came when I used it to help me write a client article.

I asked ChatGPT to help me write a technical article

The client wanted me to write about the difference between a business analyst and a software architect*, which apparently ranks well on Google. While I have some knowledge in the field, I was struggling to really comprehend the differences between these two jobs. So I put ChatGPT to the test.

*I ghostwrite for this client, so I’ve changed the topic to remain anonymous. It was equivalent to this, though.

I spent three hours writing the article with the help of ChatGPT. ChatGPT helped with:

• Generating section ideas
• Providing some details about roles and responsibilities
• Offering useful similes and metaphors to grasp concepts
• Detailing various technological capabilities

With assistance, I:

• Actually wrote the article
• Edited it to incorporate the info in a readable and friendly way
• Did additional research on job listings

I was thrilled. It took me about half the time to write the article than it would have normally. There was just one niggling doubt I had.

Because I was so unfamiliar with the field, there was a small chance that I had just written a hot pile of incorrect flaming trash. And I’d never know. At least until the client got back to me to say I was fired.

I decided to hire Sally, a professional editor in the field I’d worked with before, to review the article for me.

Just to double-check.

Just to make totally sure.

Before I sent off a (potentially) reputationally damaging article to a client.

And thank goodness I did.

Screenshot of one of the comments I received, lol.

The editor was not impressed

I didn’t tell Sally that AI had helped me write the article. I wanted to see what she thought of it as though I’d written it.

And to put it bluntly, she was not impressed.

The writing style itself was fine because I’d written it. So it was engaging, grammatically correct, and interesting.

It was just wrong.

ChatGPT had fabricated facts. ChatGPT made incorrect analogies.

ChatGPT drew distinctions that didn’t exist.

ChatGPT assigned roles to the wrong jobs.

Not only that, but ChatGPT got specific technologies wrong.

My stomach sank when I got the document back, covered in red writing and strikethroughs and critical comments. I was so disappointed. I really thought I’d cracked the code and could use ChatGPT to help handle my client work. It was sad to realize this wasn’t the case.

But then! I was doused in a wave of relief. Thank the lord I hadn’t sent this to a client!

ChatGPT is not a reliable researcher

When I first started planning this article, it was going to be all about how you can use ChatGPT to research complex topics. But honestly, Sally proved that you can’t. Unless you know a lot about the industry already, and you’re able to fact-check, ChatGPT just can’t be trusted not to make stuff up. It needs a firm hand on the reins.

My experience isn’t solitary. Online marketer Gael Breton recently called out CNET for posting a bunch of AI-written articles under the “CNET Money Staff” tag.

At first, the news commentary was breathlessly impressed. Did this mean writers were out of a job? What did this signify for our livelihoods? “Now, looking at the entire explainers that CNET has generated using AI, it looks like that goalpost has already shifted — and may never return,” wrote Frank Landymore for Futurism.

And then it turned out that the AI hadn’t even managed to write error-free articles on simple, basic financial explainers. And also it had freely plagiarised. CNET didn’t say what technology it had used, but clearly, it was not up to spec.

“CNET has issued corrections for over half of the AI-written articles the outlet recently attributed to its CNET Moneyteam,” reported Igor Bonifacic for Engadget in a slightly more prosaic tone. “Some of the 77 articles were also edited for plagiarism.”

Yikes!

Why is ChatGPT unreliable?

ChatGPT, like all AI text generation technologies, is very good at eating content, mixing it up into a blender, then spitting it back out into a new form.

Fact-checked by ChatGPT itself

But it doesn’t “know” things. It can’t reason. It can’t imagine or create. It can just regurgitate, in the closest order to its training data.

The further you push it, the more you’ll realize this. The more details you ask, the more you demand, the less correct it gets.

This makes no sense! This is not how bloggers or vloggers work!

What it can do is bullsh*t with confidence.

This is fine if the content is in the hands of fact-checking experts. For instance, in the example above, I know it’s nonsense to say bloggers are like the actors of the world while vloggers are musicians. Blog posts are not longer than vlogs. Concerts are not more visually engaging than theater performances. This is all baloney.

But if this AI-generated content reaches laypeople who don’t know the difference, that’s when things get dangerous. And that is why I was so glad to have hired Sally.

ChatGPT is not a reliable writer or researcher.

Use ChatGPT, but use it with care

When I read this article aloud to my husband to get his feedback, he asked me a question:

“So you’re not going to use AI text gen anymore?”

I thought about it and realized that actually, I would. I’ve found it too useful to give up. I have used it successfully, with oversight, to:

• come up with section ideas
• generate a ton of title suggestions
• just get something on the page which I can then rewrite/edit
• give me summaries which I can then fact-check
• make me laugh by writing an intro to a data science article in the style of “My Immortal,” famously the worst fanfic ever written

“machine learnin”

I think it makes sense for writing professionals to get a feel for it, to understand how it works, what its limitations are, and how it can be used to speed up workflows.

As long as you don’t rely on ChatGPT for anything too sensitive, technical, client-facing, or out of your comfort zone, it can be a useful tool.

But if you’re ever in doubt, I recommend hiring a professional editor to fact-check its work. If nothing else, it’ll probably be funny.