Prompt Engineering: A Practical Example

You’ve used ChatGPT, and you understand the potential of using a large language model (LLM) to assist you in your tasks. Maybe you’re already working on an LLM-supported application and read about prompt engineering, but you’re unsure how to translate the theoretical concepts into a practical example.

Your text prompt instructs the LLM’s responses, so tweaking it can get you vastly different output. In this tutorial, you’ll apply multiple prompt engineering techniques to a real-world example. You’ll experience prompt engineering as an iterative process, see the effects of applying various techniques, and learn about related concepts from machine learning and data engineering.

In this tutorial, you’ll learn how to:

• Work with OpenAI’s GPT-3.5 and GPT-4 models through their API
• Apply prompt engineering techniques to a practical, real-world example
• Use numbered steps, delimiters, and few-shot prompting to improve your results
• Understand and use chain-of-thought prompting to add more context
• Tap into the power of roles in messages to go beyond using singular role prompts

You’ll work with a Python script that you can repurpose to fit your own LLM-assisted task. So if you’d like to use practical examples to discover how you can use prompt engineering to get better results from an LLM, then you’ve found the right tutorial!

Understand the Purpose of Prompt Engineering

Prompt engineering is more than a buzzword. You can get vastly different output from an LLM when using different prompts. That may seem obvious when you consider that you get different output when you ask different questions—but it also applies to phrasing the same conceptual question differently. Prompt engineering means constructing your text input to the LLM using specific approaches.

You can think of prompts as arguments and the LLM as the function that you pass these arguments to. Different input means different output:

>>> def hello(name):
...     print(f"Hello, {name}!")
...
>>> hello("World")
Hello, World!
>>> hello("Engineer")
Hello, Engineer!

While an LLM is much more complex than the toy function above, the fundamental idea holds true. For a successful function call, you’ll need to know exactly which argument will produce the desired output. In the case of an LLM, that argument is text that consists of many different tokens, or pieces of words.

The field of prompt engineering is still changing rapidly, and there’s a lot of active research happening in this area. As LLMs continue to evolve, so will the prompting approaches that will help you achieve the best results.

In this tutorial, you’ll cover some prompt engineering techniques, along with approaches to iteratively developing prompts, that you can use to get better text completions for your own LLM-assisted projects:

There are more techniques to uncover, and you’ll also find links to additional resources in the tutorial. Applying the mentioned techniques in a practical example will give you a great starting point for improving your LLM-supported programs. If you’ve never worked with an LLM before, then you may want to peruse OpenAI’s GPT documentation before diving in, but you should be able to follow along either way.

Get to Know the Practical Prompt Engineering Project

You’ll explore various prompt engineering techniques in service of a practical example: sanitizing customer chat conversations. By practicing different prompt engineering techniques on a single real-world project, you’ll get a good idea of why you might want to use one technique over another and how you can apply them in practice.

Imagine that you’re the resident Python developer at a company that handles thousands of customer support chats on a daily basis. Your job is to format and sanitize these conversations. You should also help with deciding which of them require additional attention.

Collect Your Tasks

Your big-picture assignment is to help your company stay on top of handling customer chat conversations. The conversations that you work with may look like the one shown below:

[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

You’re supposed to make these text conversations more accessible for further processing by the customer support department in a few different ways:

• Remove personally identifiable information.
• Remove swear words.
• Clean the date-time information to only show the date.

The swear words that you’ll encounter in this tutorial won’t be spicy at all, but you can consider them stand-ins for more explicit phrasing that you might find out in the wild. After sanitizing the chat conversation, you’d expect it to look like this:

[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

Sure—you could handle it using Python’s str.replace() or show off your regular expression skills. But there’s more to the task than immediately meets the eye.

Your project manager isn’t a technical person, and they stuck another task at the end of this list. They may think of the task as a normal continuation of the previous tasks. But you know that it requires an entirely different approach and technology stack:

Mark the conversations as “positive” or “negative.”

That task lies in the realm of machine learning, namely text classification, and more specifically sentiment analysis. Even advanced regex skills won’t get you far in this challenge.

Additionally, you know that the customer support team that you’re preparing the data for will want to continue working on it programmatically. Plain text isn’t necessarily the best format for doing that. You want to do work that’s useful for others, so you add yet another stretch goal to your growing list of tasks:

Format the output as JSON.

This task list is quickly growing out of proportion! Fortunately you’ve got access to the OpenAI API, and you’ll employ the help of their LLM to solve all of these challenges.

One of the impressive features of LLMs is the breadth of tasks that you can use them for. So you’ll cover a lot of ground and different areas of use. And you’ll learn how you can tackle all of them with prompt engineering techniques.

Prepare Your Tools

To follow along with the tutorial, you’ll need to know how to run a Python script from your command-line interface (CLI), and you’ll need an API key from OpenAI.

You’ll focus on prompt engineering, so you’ll only use the CLI app as a tool to demonstrate the different techniques. However, if you want to understand the code that you’ll be using, then it’ll help to have some experience with Python classes, defining your own Python functions, the name-main idiom, and using Python to interact with web APIs.

To get started, go ahead and download the example Python script that you’ll work with throughout the tutorial:

The codebase represents a light abstraction layer on top of the OpenAI API and exposes two functions that’ll be primarily interesting for the tutorial:

1. get_completion() interacts with OpenAI’s GPT-3.5 model (text-davinci-003) to generate text completions using the /completions endpoint.
2. get_chat_completion() interacts with OpenAI’s GPT-4 model (gpt-4) to generate responses using the /chat/completions endpoint.

You’ll explore both endpoints, starting with get_completion() and eventually moving on to the more powerful GPT-4 model with get_chat_completion(). The script also parses a command-line argument to allow you to conveniently specify an input file.

The input files that you’ll primarily work with contain made-up customer support chat conversations, but feel free to reuse the script and provide your own input text files for additional practice.

The heart of the codebase is settings.toml. This TOML settings file hosts the prompts that you’ll use to sharpen your prompt engineering skills. It contains different prompts formatted in the human-readable settings format TOML.

Keeping your prompts in a dedicated settings file can help to put them under version control, which means you can keep track of different versions of your prompts, which will inevitably change during development.

Your Python script will read the prompts from settings.toml and send them as API requests.

Alternatively you can also run all the text prompts directly in the OpenAI playground, which will give you the same functionality as the script. You could even paste the prompts into the ChatGPT interface. However, the results will vary because you’ll be interacting with a different model and won’t have the opportunity to change certain settings.

Set Up the Codebase

Make sure that you’re on Python 3.11 or higher, so that you can interact with TOML files using the standard library. If you haven’t downloaded the codebase yet, go ahead and click the link below:

Unzip the folder and use your CLI to navigate into the folder. You’ll see a handful of files. The most important ones are app.py and settings.toml:

./
├── LICENSE
├── README.md
├── app.py
├── chats.txt
├── requirements.txt
├── sanitized-chats.txt
├── sanitized-testing-chats.txt
├── settings.toml
├── settings-final.toml
└── testing-chats.txt

The file settings.toml contains placeholders for all the prompts that you’ll use to explore the different prompt engineering techniques. That’s the file that you’ll primarily work with, so open it up. You’ll use it to iteratively develop the prompts for your application.

The file app.py contains the Python code that ties the codebase together. You’ll run this script many times throughout the tutorial, and it’ll take care of pulling your prompts from settings.toml.

After you’ve downloaded and unpacked the codebase, create and activate a new virtual environment. Then use pip to install openai, which is the only required dependency:

(venv) $ python -m pip install openai

To run the script successfully, you’ll need an OpenAI API key with which to authenticate your API requests. Make sure to keep that key private and never commit it to version control! If you’re new to using API keys, then read up on best practices for API key safety.

To integrate your API key with the script and avoid leaking it publicly, you can export the API key as an environment variable:

(venv) $ export OPENAI_API_KEY="your-api-key"

After you’ve added your API key as an environment variable named OPENAI_API_KEY, the script will automatically pick it up during each run.

At this point, you’ve completed the necessary setup steps. You can now run the script using the command line and provide it with a file as additional input text:

(venv) $ python app.py chats.txt

The command shown above combines the customer support chat conversations in chats.txt with prompts and API call parameters that are saved in settings.toml, then sends a request to the OpenAI API. Finally, it prints the resulting text completion to your terminal.

From now on, you’ll primarily make changes in settings.toml. The code in app.py is just here for your convenience, and you won’t have to edit that file at all. The changes in the LLM’s output will come from changing the prompts and a few of the API call arguments.

Freeze Responses by Setting the Temperature to Zero

When you’re planning to integrate an LLM into a product or a workflow, then you’ll generally want deterministic responses. The same input should give you the same output. Otherwise, it gets hard to provide a consistent service or debug your program if something goes wrong.

Because of this, you’ll want to set the temperature argument of your API calls to 0. This value will mean that you’ll get mostly deterministic results.

LLMs do text completion by predicting the next token based on the probability that it follows the previous tokens. Higher temperature settings will introduce more randomness into the results by allowing the LLM to pick tokens with lower probabilities. Because there are so many token selections chained one after one the other, picking one different token can sometimes lead to vastly different results.

If you use the LLM to generate ideas or alternative implementations of a programming task, then higher values for temperature might be interesting. However, they’re generally undesirable when you build a product.

In the example codebase, you can adjust temperature right inside your settings.toml file:

# settings.toml

[general]
chat_models = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4"]
model = "text-davinci-003"
max_tokens = 2100
temperature = 0

The initial value is set at 0. All the examples in this tutorial assume that you leave temperature at 0 so that you’ll get mostly deterministic results. If you want to experiment with how a higher temperature changes the output, then feel free to play with it by changing the value for temperature in this settings file.

It’s important to keep in mind that you won’t be able to achieve true determinism with the current LLM models offered by OpenAI even if you keep temperature at 0:

An edge-case in GPT-3 with big implications: Inference is non-deterministic (even at temperature=0) when top-2 token probabilities are <1% different. So temperature=0 output is very close to deterministic, but actually isn’t. Worth remembering. (Source)

So, while you can’t entirely guarantee that the model will always return the same result, you can get much closer by setting temperature to 0.

Start Engineering Your Prompts

Now that you have an understanding of prompt engineering and the practical project that you’ll be working with, it’s time to dive into some prompt engineering techniques. In this section, you’ll learn how to apply the following techniques to your prompts to get the desired output from the language model:

• Zero-shot prompting: Asking the language model a normal question without any additional context
• Few-shot prompting: Conditioning the model on a few examples to boost its performance
• Using delimiters: Adding special tokens or phrases to provide structure and instructions to the model
• Detailed, numbered steps: Breaking down a complex prompt into a series of small, specific steps

By practicing these techniques with the customer chat conversation example, you’ll gain a deeper understanding of how prompt engineering can enhance the capabilities of language models and improve their usefulness in real-world applications.

Describe Your Task

You’ll start your prompt engineering journey with a concept called zero-shot prompting, which is a fancy way of saying that you’re just asking a normal question or describing a task:

Remove personally identifiable information, only show the date, and replace all swear words with “😤”

This task description focuses on the requested steps for sanitizing the customer chat conversation and literally spells them out. This is the prompt that’s currently saved as instruction_prompt in the settings.toml file:

# settings.toml

# ...

instruction_prompt = """
Remove personally identifiable information, only show the date,
and replace all swear words with "😤"
"""

If you run the Python script and provide the support chat file as an argument, then it’ll send this prompt together with the content of chats.txt to OpenAI’s text completion API:

(venv) $ python app.py chats.txt

If you correctly installed the dependencies and added your OpenAI API key as an environment variable, then all you need to do is wait until you’ll see the API response pop up in your terminal:

[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

[support_amy] 2023-06-15T14:45:35+00:00 : Hello! How can I assist you today?
[greg_stone] 2023-06-15T14:46:20+00:00 : I can't seem to find the download link for my purchased software.
[support_amy] 2023-06-15T14:47:01+00:00 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[greg_stone] 2023-06-15T14:47:38+00:00 : It's ********. Thanks for helping me out!

[support_louis] 2023-05-05T09:22:12+00:00 : Hi, how can I help you today?
[karen_w] 2023-05-05T09:23:47+00:00 : MY BLASTED ORDER STILL HASN'T ARRIVED AND IT'S BEEN A WEEK!!!
[support_louis] 2023-05-05T09:24:15+00:00 : I'm sorry to hear that, Karen. Let's look into this issue.
[support_louis] 2023-05-05T09:25:35+00:00: Can you please provide your order number so I can check the status for you?
[karen_w] 2023-05-05T09:26:12+00:00: Fine, it's ********.
[support_louis] 2023-05-05T09:26:45+00:00: Thank you, Karen. I see there was a delay in shipping. Your order will arrive within the next 2 days.

[support_jenny] 2023-06-18T17:35:28+00:00: Hello! How can I help you today?
[alex_harper] 2023-06-18T17:36:05+00:00: I accidentally placed an order twice, can you help me cancel one?
[support_jenny] 2023-06-18T17:36:25+00:00: Sure, Alex. Can you give me the order number you'd like to cancel?
[alex_harper] 2023-06-18T17:36:55+00:00: Yes, it's ********. Thank you!
[support_jenny] 2023-06-18T17:37:32+00:00: I've successfully canceled order number ********. You will receive a confirmation email shortly.

[support_ben] 2023-06-29T11:51:45+00:00: Good morning, what can I assist you with today?
[lisa_beck] 2023-06-29T11:52:20+00:00: Hi there, I received a damaged item in my order. Can you help me return it?
[support_ben] 2023-06-29T11:52:45+00:00: I'm sorry to hear that, Lisa. Can you provide your order number and specify the damaged item?
[lisa_beck] 2023-06-29T11:53:22+00:00: Sure, order number is ******** and the damaged item is a coffee mug.

[support_rachel] 2023-05-04T08:16:37+00:00: How can I help you today?
[mike_t] 2023-05-04T08:17:15+00:00: My coupon code isn't working at checkout. Can you help?
[support_rachel] 2023-05-04T08:17:38+00:00: Of course, Mike. Please provide the coupon code you're trying to use.
[mike_t] 2023-05-04T08:18:02+00:00: It's "HELLO10".
[support_rachel] 2023-05-04T08:18:37+00:00: I've checked the code, and it seems to have expired. I apologize for the inconvenience. Here's a new code for you to use: "WELCOME15".

[support_vincent] 2023-06-15T20:43:55+00:00: Good evening! How may I assist you?
[sara_winters] 2023-06-15T20:44:30+00:00: Hi there, I'm having trouble logging into my account. I've tried resetting my password, but it's not working.
[support_vincent] 2023-06-15T20:44:52+00:00: I'm sorry to hear that, Sara. Let me help you. Can you please confirm your email address?
[sara_winters] 2023-06-15T20:45:25+00:00: Sure, it's ********.

[support_david] 2023-06-24T16:28:43+00:00: Welcome! What can I do for you today?
[jane_d] 2023-06-24T16:29:16+00:00: Hi, I need to change my delivery address for my recent order.
[support_david] 2023-06-24T16:29:43+00:00: Alright, Jane. Please provide your order number.
[jane_d] 2023-06-24T16:30:11+00:00: It's ********. Thanks for your help!

The text above represents an example response. Keep in mind that OpenAI’s LLM models aren’t fully deterministic even with temperature set to 0, so your output may be slightly different.

In the example output, you can see that the prompt that you provided didn’t really do a great job at tackling the tasks. In the example output above, it managed to obfuscate some of the personally identifiable information from the text, replacing it with ********. Your results might not have tackled that. Overall, a lot is left undone:

• The names of the customers and the customer service agents are still visible.
• The text still contains the full ISO date-time stamp.
• The swear words are still uncensored.

If you’re new to interacting with LLMs, then this may have been a first attempt at outsourcing your development work to the text completion model. But these initial results aren’t exhilarating.

So you’ve described the task in natural language and gotten mixed results. But don’t fret—throughout the tutorial you’ll learn how you can get better, more deterministic responses.

One way to do that is by increasing the number of shots, or examples, that you give to the model. When you’ve given the model zero shots, the only way to go is up! That’s why you’ll improve your results through few-shot prompting in the next section.

Use Few-Shot Prompting to Improve Output

Few-shot prompting is a prompt engineering technique where you provide example tasks and their expected solutions in your prompt. So, instead of just describing the task like you did before, you’ll now add an example of a chat conversation and its sanitized version.

Open up settings.toml and change your instruction_prompt by adding such an example:

instruction_prompt = """
Remove personally identifiable information, only show the date,
and replace all swear words with "😤"

Example Input:
[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

Example Output:
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!
"""

Once you’ve applied the change, give the LLM another chance to sanitize the chat conversations for you by running the script again:

(venv) $ python app.py chats.txt

You’ll have to wait for the LLM to predict all the tokens. When it’s done, you’ll see a fresh response pop up in your terminal:

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's 1245789. Thanks for helping me out!

[Agent] 2023-05-05 : Hi, how can I help you today?
[Customer] 2023-05-05 : MY 😤 ORDER STILL HASN'T ARRIVED AND IT'S BEEN A WEEK!!!
[Agent] 2023-05-05 : I'm sorry to hear that, Karen. Let's look into this issue.
[Agent] 2023-05-05 : Can you please provide your order number so I can check the status for you?
[Customer] 2023-05-05 : Fine, it's 9876543.
[Agent] 2023-05-05 : Thank you, Karen. I see there was a delay in shipping. Your order will arrive within the next 2 days.

[Agent] 2023-06-18 : Hello! How can I help you today?
[Customer] 2023-06-18 : I accidentally placed an order twice, can you help me cancel one?
[Agent] 2023-06-18 : Sure, Alex. Can you give me the order number you'd like to cancel?
[Customer] 2023-06-18 : Yes, it's 1122334. Thank you!
[Agent] 2023-06-18 : I've successfully canceled order number 1122334. You will receive a confirmation email shortly.

[Agent] 2023-06-29 : Good morning, what can I assist you with today?
[Customer] 2023-06-29 : Hi there, I received a damaged item in my order. Can you help me return it?
[Agent] 2023-06-29 : I'm sorry to hear that, Lisa. Can you provide your order number and specify the damaged item?
[Customer] 2023-06-29 : Sure, order number is 5566778 and the damaged item is a coffee mug.

[Agent] 2023-05-04 : How can I help you today?
[Customer] 2023-05-04 : My coupon code isn't working at checkout. Can you help?
[Agent] 2023-05-04 : Of course, Mike. Please provide the coupon code you're trying to use.
[Customer] 2023-05-04 : It's "HELLO10".
[Agent] 2023-05-04 : I've checked the code, and it seems to have expired. I apologize for the inconvenience. Here's a new code for you to use: "WELCOME15".

[Agent] 2023-06-15 : Good evening! How may I assist you?
[Customer] 2023-06-15 : Hi, I'm having trouble logging into my account. I've tried resetting my password, but it's not working.
[Agent] 2023-06-15 : I'm sorry to hear that, Sara. Let me help you. Can you please confirm your email address?
[Customer] 2023-06-15 : Sure, it's **********.

You’ll probably notice significant improvements in how the names in square brackets are sanitized. The time stamp is also correctly formatted. The model even replaced a swear word in a later chat with the huffing emoji. However, the names of the customers are still visible in the actual conversations. In this run, the model even took a step backward and didn’t censor the order numbers.

So far, you’ve provided one example in your prompt. To cover more ground, you’ll add another example so that this part of your prompt truly puts the few in few-shot prompting:

instruction_prompt = """
Remove personally identifiable information, only show the date,
and replace all swear words with "😤"

Example Inputs:
[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

[support_amy] 2023-06-15T14:45:35+00:00 : Hello! How can I assist you today?
[greg_stone] 2023-06-15T14:46:20+00:00 : I can't seem to find the download link for my purchased software.
[support_amy] 2023-06-15T14:47:01+00:00 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[greg_stone] 2023-06-15T14:47:38+00:00 : It's 1245789. Thanks for helping me out!

Example Outputs:
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!
"""

You added a second example that contains both a customer name as well as an order number in the chat text body. The example of a sanitized chat shows both types of sensitive data replaced with a sequence of asterisks (********). Now you’ve given the LLM a good example to model.

After editing instruction_prompt in settings.toml, run your script again and wait for the response to print to your terminal:

Wait? Where did the output go? You probably expected to see better results, but it looks like you’re getting an empty result instead!

Adding more examples should make your responses stronger instead of eating them up, so what’s the deal? You can trust that few-shot prompting works—it’s a widely used and very effective prompt engineering technique. To help the model distinguish which part of your prompt contains the instructions that it should follow, you can use delimiters.

Use Delimiters to Clearly Mark Sections of Your Prompt

If you’re working with content that needs specific inputs, or if you provide examples like you did in the previous section, then it can be very helpful to clearly mark specific sections of the prompt. Keep in mind that everything you write arrives to an LLM as a single prompt—a long sequence of tokens.

You can improve the output by using delimiters to fence and label specific parts of your prompt. In fact, if you’ve been running the example code, then you’ve already used delimiters to fence the content that you’re reading from file.

The script adds the delimiters when assembling the prompt in app.py:

74# app.py
75
76# ...
77
78def assemble_prompt(content: str, settings: Settings) -> str:
79    """Combine all text input into a single prompt."""
80    return f">>>>>\n{content}\n<<<<<\n\n" + settings.instruction_prompt

In line 80, you wrap the chat content in between >>>>> and <<<<< delimiters. Marking parts of your prompt with delimiters can help the model keep track of which tokens it should consider as a single unit of meaning.

You’ve seen in the previous section that missing delimiters can lead to unexpected results. You might receive an empty response, like before. But you might also receive output that’s quite different from what you want! For example, imagine that the content you’re reformatting contains a question at the end, such as:

Can you give me your order number?

If this question is the last line of your prompt without delimiters, then the LLM will probably continue the imaginary chat conversation by answering the question with an imaginary order number. Give it a try by adding such a sentence to the end of your current prompt!

Delimiters can help to separate the content and examples from the task description. They can also make it possible to refer to specific parts of your prompt at a later point in the prompt.

A delimiter can be any sequence of characters that usually wouldn’t appear together, for example:

• >>>>>
• ====
• ####

The number of characters that you use doesn’t matter too much, as long as you make sure that the sequence is relatively unique. Additionally, you can add labels just before or just after the delimiters:

• START CONTENT>>>>> content <<<<<END CONTENT
• ==== START content END ====
• #### START EXAMPLES examples #### END EXAMPLES

The exact formatting also doesn’t matter so much. As long as you mark the sections so that a casual reader could understand where a unit of meaning begins and ends, then you’ve properly applied delimiters.

Edit your prompt in settings.toml to add a clear reference to your delimited content, and also include a delimiter for the examples that you’ve added:

instruction_prompt = """Remove personally identifiable information
from >>>>>CONTENT<<<<<, only show the date,
and replace all swear words with "😤"

#### START EXAMPLES

------ Example Inputs ------
[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

[support_amy] 2023-06-15T14:45:35+00:00 : Hello! How can I assist you today?
[greg_stone] 2023-06-15T14:46:20+00:00 : I can't seem to find the download link for my purchased software.
[support_amy] 2023-06-15T14:47:01+00:00 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[greg_stone] 2023-06-15T14:47:38+00:00 : It's 1245789. Thanks for helping me out!

------ Example Outputs ------
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

#### END EXAMPLES
"""

With these adaptations to your instruction_prompt, you now specifically reference the content as >>>>>CONTENT<<<<< in your task description. These delimiters match the delimiters that the code in app.py adds when assembling the prompt.

You’ve also delimited the examples that you’re providing with #### START EXAMPLES and #### END EXAMPLES, and you differentiate between the inputs and expected outputs using multiple dashes (------) as delimiters.

When you give your script with the updated prompt another go, you’ll see that the results are more promising than before:

[Agent] 2023-05-05 : Hi, how can I help you today?
[Customer] 2023-05-05 : MY 😤 ORDER STILL HASN'T ARRIVED AND IT'S BEEN A WEEK!!!
[Agent] 2023-05-05 : I'm sorry to hear that, ********. Let's look into this issue.
[Agent] 2023-05-05 : Can you please provide your order number so I can check the status for you?
[Customer] 2023-05-05 : Fine, it's ********.
[Agent] 2023-05-05 : Thank you, ********. I see there was a delay in shipping. Your order will arrive within the next 2 days.

[Agent] 2023-06-18 : Hello! How can I help you today?
[Customer] 2023-06-18 : I accidentally placed an order twice, can you help me cancel one?
[Agent] 2023-06-18 : Sure, ********. Can you give me the order number you'd like to cancel?
[Customer] 2023-06-18 : Yes, it's ********. Thank you!
[Agent] 2023-06-18 : I've successfully canceled order number ********. You will receive a confirmation email shortly.

[Agent] 2023-06-29 : Good morning, what can I assist you with today?
[Customer] 2023-06-29 : Hi there, I received a damaged item in my order. Can you help me return it?
[Agent] 2023-06-29 : I'm sorry to hear that, ********. Can you provide your order number and specify the damaged item?
[Customer] 2023-06-29 : Sure, order number is ******** and the damaged item is a coffee mug.

[Agent] 2023-05-04 : How can I help you today?
[Customer] 2023-05-04 : My coupon code isn't working at checkout. Can you help?
[Agent] 2023-05-04 : Of course, ********. Please provide the coupon code you're trying to use.
[Customer] 2023-05-04 : It's "********".
[Agent] 2023-05-04 : I've checked the code, and it seems to have expired. I apologize for the inconvenience. Here's a new code for you to use: "********".

[Agent] 2023-06-15 : Good evening! How may I assist you?
[Customer] 2023-06-15 : Hi, I need to change my delivery address for my recent order.
[Agent] 2023-06-15 : Alright, ********. Please provide your order number.
[Customer] 2023-06-15 : It's ********. Thanks for your help!

Great, the sanitized output looks close to what you were looking for in the sanitation step! It’s noticeable that the model omitted the two example data that you passed as examples from the output. Could that mean that your prompt generalizes well? You’ll take a look up ahead.

In this section, you’ve learned how you can clarify the different parts of your prompt using delimiters. You marked which part of the prompt is the task description and which part contains the customer support chat conversations, as well as the examples of original input and expected sanitized output.

Test Your Prompt Across Different Data

So far, you’ve created your few-shot examples from the same data that you also run the sanitation on. This means that you’re effectively using your test data to fine-tune the model. Mixing training, validation, and testing data is a bad practice in machine learning. You might wonder how well your prompt generalizes to different input.

To test this out, run the script another time with the same prompt using the second file that contains chat conversations, testing-chats.txt. The conversations in this file contain different names, and different—soft—swear words:

(venv) $ python app.py testing-chats.txt

You’ll keep running your script using testing-chats.txt for the rest of this section.

Once you’ve waited for the LLM to generate and return the response, you’ll notice that the result isn’t very satisfying:

[Agent] 2023-07-15 : Hello! What can I help you with today?
[Customer] 2023-07-15 : Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15 : My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Customer] 2023-07-15 : It's "********".

[Agent] 2023-07-24 : Good day! How can I help you?
[Customer] 2023-07-24 : Hi "********", I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24 : I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Customer] 2023-07-24 : Sure, you have all my 😤 data already anyways. It's ********.

[Agent] 2023-08-13 : Good morning! How may I assist you?
[Customer] 2023-08-13 : Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13 : I'm sorry to hear that, ********. Could you tell me what device you're using?
[Customer] 2023-08-13 : I have an iPhone ********.

[Agent] 2023-08-30 : Good evening! How may I assist you today?
[Customer] 2023-08-30 : Hi Lisa, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30 : I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Customer] 2023-08-30 : Definitely, it's ********.

[Agent] 2023-09-01 : How can I help you today?
[Customer] 2023-09-01 : Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01 : I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Customer] 2023-09-01 : It's saying "********".

[Agent] 2023-10-11 : Good morning! How may I assist you?
[Customer] 2023-10-11 : Hello, I'd like to know the status of my order.
[Agent] 2023-10-11 : Of course, ********. Could you please provide me with the order number?
[Customer] 2023-10-11 : It's ********.

[Agent] 2023-10-19 : Welcome! How can I assist you right now?
[Customer] 2023-10-19 : 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19 : Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Customer] 2023-10-19 : I'm using the 😤 website

[Agent] 2023-10-29 : Hello! What can I help you with today?
[Customer] 2023-10-29 : Hi Tony, I was charged twice for my last order.
[Agent] 2023-10-29 : I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Customer] 2023-10-29 : Sure, it's ********.

[Agent] 2023-11-08 : How can I help you today?
[Customer] 2023-11-08 : Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08 : Certainly, ********. Could you provide me the order number?
[Customer] 2023-11-08 : Yes, it's ********. Thanks!

The model had no issue with identifying and replacing the swear words, and it also redacted the order numbers. It even managed to replace the different names in the square brackets. However, it missed some names in the conversation texts.

So your engineered prompt currently doesn’t generalize all that well. If you built a pipeline based on this prompt, where new chats could contain new customer names, then the application would probably continue to perform poorly. How can you fix that?

You’ve grown your prompt significantly by providing more examples, but your task description is still largely just the question you wrote right at the beginning. To continue to get better results, you’ll need to do some prompt engineering on the task description as well.

Describe Your Request in Numbered Steps

If you break up your task instructions into a numbered sequence of small steps, then the model is a lot more likely to produce the results that you’re looking for.

Go back to your prompt in settings.toml and break your initial task description into more granular, specific substeps:

instruction_prompt = """
Sanitize the text provided in >>>CONTENT<<< in multiple steps:

1. Replace personally identifiable information (customer names, agent names, email addresses, order numbers) with `********`
2. Replace names in [] with "Agent" and "Client", respectively
3. Replace the date-time information to only show the date in the format YYYY-mm-dd
4. Replace all swear words with the following emoji: "😤"

#### START EXAMPLES

------ Example Inputs ------
[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

[support_amy] 2023-06-15T14:45:35+00:00 : Hello! How can I assist you today?
[greg_stone] 2023-06-15T14:46:20+00:00 : I can't seem to find the download link for my purchased software.
[support_amy] 2023-06-15T14:47:01+00:00 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[greg_stone] 2023-06-15T14:47:38+00:00 : It's 1245789. Thanks for helping me out!

------ Example Outputs ------
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

#### END EXAMPLES
"""

With these step-by-step instructions in place, you’re ready for another run of your script and another inspection of the newly generated output:

[Agent] 2023-07-15 : Hello! What can I help you with today?
[Customer] 2023-07-15 : Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15 : My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Customer] 2023-07-15 : It's "********".

[Agent] 2023-07-24 : Good day! How can I help you?
[Customer] 2023-07-24 : Hi "********", I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24 : I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Customer] 2023-07-24 : Sure, you have all my 😤 data already anyways. It's ********.

[Agent] 2023-08-13 : Good morning! How may I assist you?
[Customer] 2023-08-13 : Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13 : I'm sorry to hear that, ********. Could you tell me what device you're using?
[Customer] 2023-08-13 : I have an iPhone ********.

[Agent] 2023-08-30 : Good evening! How may I assist you today?
[Customer] 2023-08-30 : Hi Lisa, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30 : I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Customer] 2023-08-30 : Definitely, it's ********.

[Agent] 2023-09-01 : How can I help you today?
[Customer] 2023-09-01 : Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01 : I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Customer] 2023-09-01 : It's saying "********".

[Agent] 2023-10-11 : Good morning! How may I assist you?
[Customer] 2023-10-11 : Hello, I'd like to know the status of my order.
[Agent] 2023-10-11 : Of course, ********. Could you please provide me with the order number?
[Customer] 2023-10-11 : It's ********.

[Agent] 2023-10-19 : Welcome! How can I assist you right now?
[Customer] 2023-10-19 : 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19 : Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Customer] 2023-10-19 : I'm using the 😤 website

[Agent] 2023-10-29 : Hello! What can I help you with today?
[Customer] 2023-10-29 : Hi Tony, I was charged twice for my last order.
[Agent] 2023-10-29 : I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Customer] 2023-10-29 : Sure, it's ********.

[Agent] 2023-11-08 : How can I help you today?
[Customer] 2023-11-08 : Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08 : Certainly, ********. Could you provide me the order number?
[Customer] 2023-11-08 : Yes, it's ********. Thanks!

In this case, the output is still the same as before. Often, numbered steps can improve the performance of your desired task. However, here some names, such as Tony and Lisa, are still visible in the conversation text.

Maybe you weren’t specific and detailed enough!

Increase the Steps for More Specificity

In the previous run of your script, you noticed that some personally identifiable information can still slip through. To fix that, you can increase the specificity of your instructions.

Framing your tasks in even smaller and even more specific steps, you’ll generally get better results. Don’t shy away from some repetition:

instruction_prompt = """
Sanitize the text provided in >>>CONTENT<<< in multiple steps:

1. Replace personally identifiable information with `********`
2. Delete all names
3. Replace email addresses and order numbers with `********`
4. Replace names in [] with "Agent" and "Client", respectively
5. Replace the date-time information to only show the date in the format YYYY-mm-dd
6. Replace all swear words with the following emoji: "😤"

#### START EXAMPLES

------ Example Inputs ------
[support_tom] 2023-07-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2023-07-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2023-07-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2023-07-24T10:04:03+00:00 : Blast! You're right!

[support_amy] 2023-06-15T14:45:35+00:00 : Hello! How can I assist you today?
[greg_stone] 2023-06-15T14:46:20+00:00 : I can't seem to find the download link for my purchased software.
[support_amy] 2023-06-15T14:47:01+00:00 : No problem, Greg. Let me find that for you. Can you please provide your order number?
[greg_stone] 2023-06-15T14:47:38+00:00 : It's 1245789. Thanks for helping me out!

------ Example Outputs ------
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

#### END EXAMPLES
"""

Adding additional steps provides the model with more context, which usually leads to better results. It certainly does so in this case:

[Agent] 2023-07-15 : Hello! What can I help you with today?
[Customer] 2023-07-15 : Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15 : My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Customer] 2023-07-15 : It's "********".

[Agent] 2023-07-24 : Good day! How can I help you?
[Customer] 2023-07-24 : Hi ********, I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24 : I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Customer] 2023-07-24 : Sure, you have all my 😤 data already anyways. It's ********.

[Agent] 2023-08-13 : Good morning! How may I assist you?
[Customer] 2023-08-13 : Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13 : I'm sorry to hear that, ********. Could you tell me what device you're using?
[Customer] 2023-08-13 : I have an iPhone ********.

[Agent] 2023-08-30 : Good evening! How may I assist you today?
[Customer] 2023-08-30 : Hi ********, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30 : I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Customer] 2023-08-30 : Definitely, it's ********.

[Agent] 2023-09-01 : How can I help you today?
[Customer] 2023-09-01 : Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01 : I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Customer] 2023-09-01 : It's saying "********".

[Agent] 2023-10-11 : Good morning! How may I assist you?
[Customer] 2023-10-11 : Hello, I'd like to know the status of my order.
[Agent] 2023-10-11 : Of course, ********. Could you please provide me with the order number?
[Customer] 2023-10-11 : It's ********.

[Agent] 2023-10-19 : Welcome! How can I assist you right now?
[Customer] 2023-10-19 : 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19 : Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Customer] 2023-10-19 : I'm using the 😤 website

[Agent] 2023-10-29 : Hello! What can I help you with today?
[Customer] 2023-10-29 : Hi ********, I was charged twice for my last order.
[Agent] 2023-10-29 : I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Customer] 2023-10-29 : Sure, it's ********.

[Agent] 2023-11-08 : How can I help you today?
[Customer] 2023-11-08 : Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08 : Certainly, ********. Could you provide me the order number?
[Customer] 2023-11-08 : Yes, it's ********. Thanks!

Finally, the remaining customer names in the conversation text are also redacted. The results look good and also seem to generalize well, at least to the second batch of example chat conversations in testing-chats.txt, on which you applied your prompt.

You may have received slightly different output. Keep in mind that the results aren’t fully deterministic. However, with better prompts, you’ll move closer to mostly deterministic results.

At this point, you’ve created a prompt that successfully removes personally identifiable information from the conversations, and reformats the ISO date-time stamp as well as the usernames.

Assess When to Switch to a Different Model

You may have noticed how your prompt has continued to grow from a single-line task description to a long text with multiple steps and multiple examples.

If you keep growing your prompt, then you might soon hit the limit of the model that you’re currently working with. In this section, you’ll learn why that might happen and how you can switch to a different model.

Juggle the Number of Tokens in Your Prompt and Your Response

Iterative prompt engineering often means that you’ll keep increasing the context in your prompt, providing more text overall. Because of this, you might eventually run into an error once you exceed the model’s token limit:

openai.error.InvalidRequestError: This model's maximum context
⮑ length is 4097 tokens, however you requested 4111 tokens
⮑ (1911 in your prompt; 2200 for the completion).
⮑ Please reduce your prompt; or completion length.

Like the error message in the above traceback describes, you’ve exceeded the maximum content length of this model, which is 4097 tokens for text-davinci-003. The message also mentions two approaches for solving the issue at hand:

1. Shorter prompt: You can decrease the tokens in your prompt by reducing either the instructions or the content input that you send in the request.
2. Shorter response: You can decrease the number of tokens that you request as a response from the model.

For this example, you don’t want to reduce the tokens in your prompt. But your response might still have room to shrink. In your settings.toml file, you can reduce the number of tokens that you request as a response by editing the entry for max_tokens:

# settings.toml

[general]
chat_models = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4"]
model = "text-davinci-003"
max_tokens = 2000
temperature = 0

If you set that to a lower number, then you can send more tokens in your prompt. However, the response that you receive won’t give back all the conversation examples if the total number of tokens in the response would exceed the value set in max_tokens.

>>> import tiktoken
>>> encoding = tiktoken.encoding_for_model("text-davinci-003")
>>> tokens = encoding.encode("This is a sample text")
>>> len(tokens)
5

You can experiment with changing max_tokens to a low value, for example 100. Without changing the prompt, you’ve now severely curtailed your output:

[Agent] 2023-07-15 : Hello! What can I help you with today?
[Customer] 2023-07-15 : Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15 : My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Customer] 2023-07-15 : It's "********".

[Agent]

Running into token limits is a common issue that users face when working with LLMs. There’s a lot of development effort aiming to increase the context that an LLM can consider, so the token windows will likely keep increasing.

OpenAI also offers different models that can consider a much larger token window, such as gpt-3.5-turbo-16k and gpt-4. If you keep growing your prompt, and you hit the limit of the model that you’re currently working with, then you can switch to a different model.

Switch to a Chat Completions Model

At the time of writing, the GPT-3.5 model text-davinci-003 has the highest token limit on the /completions endpoint. However, the company also provides access to other GPT-3.5 and GPT-4 models on the /chat/completions endpoint. These models are optimized for chat, but they also work well for text completion tasks like the one you’ve been working with.

Nevertheless, you’ll need to access them through a different endpoint, so both the structure of the prompt that you send as well as the API request will be slightly different.

If you’re working with the provided script, then all you need to do is to pick a chat model from chat_models in settings.toml and use it as the new value for model:

# settings.toml

[general]
chat_models = ["gpt-3.5-turbo", "gpt-3.5-turbo-16k", "gpt-4"]
model = "gpt-4"
max_tokens = 2000 # Not used

Changing this setting will trigger a different function, get_chat_completion(), that’ll assemble your prompt in the way necessary for a /chat/completions endpoint request. Like before, the script will also make that request for you and print the response to your terminal.

For the rest of this tutorial, you’ll work with OpenAI’s latest version of the GPT-4 model. If you don’t have access to this model, then you can instead use any of the other models noted in chat_models. If you’ve been following along using ChatGPT, then you’ve used one of the chat models, probably gpt-3.5-turbo, all along. If you’re a ChatGPT Plus subscriber, then you can even change the model to GPT-4 on the website.

Without changing your prompt, run your script another time to see the different results of the text completion based on using a different LLM:

#### START SANITIZATION

[Agent] 2023-07-15: Hello! What can I help you with today?
[Client] 2023-07-15: Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Client] 2023-07-15: It's "SAVE20".

[Agent] 2023-07-24: Good day! How can I help you?
[Client] 2023-07-24: Hi "********", I can't update my darn credit card information. Do you want my darn money or not?
[Agent] 2023-07-24: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Client] 2023-07-24: Sure, you have all my darn data already anyways. It's ********.

[Agent] 2023-08-13: Good morning! How may I assist you?
[Client] 2023-08-13: Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13: I'm sorry to hear that, ********. Could you tell me what device you're using?
[Client] 2023-08-13: I have an iPhone 11.

[Agent] 2023-08-30: Good evening! How may I assist you today?
[Client] 2023-08-30: Hi ********, I've forgotten my friggin password and I can't login into my account.
[Agent] 2023-08-30: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Client] 2023-08-30: Definitely, it's ********.

[Agent] 2023-09-01: Hello! How can I assist you this morning?
[Client] 2023-09-01: Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Client] 2023-09-01: It's saying "Payment method not valid".

[Agent] 2023-10-11: Good morning! How may I assist you?
[Client] 2023-10-11: Hello, I'd like to know the status of my order.
[Agent] 2023-10-11: Of course, ********. Could you please provide me with the order number?
[Client] 2023-10-11: It's ********.

[Agent] 2023-10-19: Welcome! How can I assist you right now?
[Client] 2023-10-19: 😤! There's no option to change my profile picture. What kind of crikey joint are you running?
[Agent] 2023-10-19: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Client] 2023-10-19: I'm using the darn website

[Agent] 2023-10-29: Hello! What can I help you with today?
[Client] 2023-10-29: Hi ********, I was charged twice for my last order.
[Agent] 2023-10-29: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Client] 2023-10-29: Sure, it's ********.

[Agent] 2023-11-08: How can I help you today?
[Client] 2023-11-08: Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08: Certainly, ********. Could you provide me the order number?
[Client] 2023-11-08: Yes, it's ********. Thanks!

#### END SANITIZATION

You may notice that the request took significantly longer to complete than with the previous model. Some responses may be relatively similar to the ones with the older model. However, you can also expect to receive results like the one shown above, where most swear words are still present, and the model uses [Client] instead of the requested [Customer].

It’s important to keep in mind that developing for a specific model will lead to specific results, and swapping the model may improve or deteriorate the responses that you get. Therefore, swapping to a newer and more powerful model won’t necessarily give you better results straight away.

Additionally, it’s also helpful to keep in mind that API calls to larger models will generally cost more money per request. While it can be fun to always use the latest and greatest LLM, it may be worthwhile to consider whether you really need to upgrade to tackle the task that you’re trying to solve.

Work With the Chat Completions Endpoint and GPT-4

You’ve decided to switch to a newer model at the /chat/completions endpoint that OpenAI will continue to develop. In this section, you’ll learn how to work with GPT-4 and get to know additional techniques to improve your prompt engineering skills:

1. Role prompting: Using a system message to set the tone of the conversation, and using different roles to give context through labeling
2. Chain-of-thought prompting (CoT): Giving the model time to think by prompting it to reason about a task, then including the reasoning in the prompt

You’ll also use GPT-4 to classify the sentiment of each chat conversation and structure the output format as JSON.

Add a Role Prompt to Set the Tone

The /chat/completions endpoint offers an option that isn’t available for the older /completions endpoint: adding role labels to a part of the prompt. In this section, you’ll use the "system" role to create a system message, and you’ll revisit the concept later on when you add more roles to improve the output.

Role prompting usually refers to adding system messages, which represent information that helps to set the context for upcoming completions that the model will produce. System messages usually aren’t visible to the end user. Keep in mind that the /chat/completions endpoint models were initially designed for conversational interactions.

You can also use system messsages to set a context for your completion task. You’ll craft a bespoke role prompt in a moment. However, for this specific task, the role prompt is likely less important than it might be for some other tasks. To explore the possible influence of a role prompt, you’ll take a little detour and ask your model to play a role:

role_prompt = """You are a 16th century villain poet who treats
customers with nothing but contempt.
Rephrase every line spoken by an Agent with your unique voice."""

You keep instruction_prompt the same as you engineered it earlier in the tutorial. Additionally, you now add text to role_prompt. The role prompt shown above serves as an example for the impact that a misguided prompt can have on your application.

Unleash, thou shall, the parchment’s code and behold the marvels unexpected, as the results may stir wonderment and awe:

[Agent] 2023-07-15: Hail! What troubles bring you to my lair?
[Client] 2023-07-15: Greetings, my discount code seems to be as useless as a jester in a nunnery.
[Agent] 2023-07-15: A thousand pardons for this inconvenience, ********. Pray, what is this code you speak of?
[Client] 2023-07-15: It goes by the name "SAVE20".

[Agent] 2023-07-24: Good morrow! What can this humble servant do for you?
[Client] 2023-07-24: Listen here, "Peter", I can't seem to update my blasted credit card information. Do you desire my coin or not?
[Agent] 2023-07-24: My deepest regrets for this vexation, ********. Could you confirm the raven's address where we send our scrolls?
[Client] 2023-07-24: Indeed, you already possess all my secrets. It's ********.

[Agent] 2023-08-13: Good morn! How may I be of service?
[Client] 2023-08-13: Salutations, my mobile contraption seems to be as stable as a drunkard on a horse.
[Agent] 2023-08-13: My condolences for your plight, ********. Pray, what is this device you wield?
[Client] 2023-08-13: I possess an iPhone 11.

[Agent] 2023-08-30: Good eve! How may I serve you this night?
[Client] 2023-08-30: Hail Lisa, I've misplaced my blasted password and now I'm as locked out as a peasant at a royal feast.
[Agent] 2023-08-30: My regrets for your predicament, ********. Could you confirm your raven's address so we may reset your key to the kingdom?
[Client] 2023-08-30: Indeed, it's ********.

[Agent] 2023-09-01: Hail! How may I serve you this morn?
[Client] 2023-09-01: Greetings, I'm attempting to make a purchase but it's proving as successful as a cat herding mice.
[Agent] 2023-09-01: My deepest regrets for your trouble, ********. Can you tell me what message of doom you receive?
[Client] 2023-09-01: It proclaims "Payment method not valid".

[Agent] 2023-10-11: Good morn! How may I be of service?
[Client] 2023-10-11: Salutations, I seek knowledge of my order's journey.
[Agent] 2023-10-11: Indeed, ********. Could you provide the number that marks your order?
[Client] 2023-10-11: It bears the mark 717171.

[Agent] 2023-10-19: Welcome! How may I assist you in this moment?
[Client] 2023-10-19: Fudge! There's no option to change my visage on your profile. What kind of establishment are you running?
[Agent] 2023-10-19: Allow me to guide you, ********. Are you attempting this change on our mobile contraption or our web of information?
[Client] 2023-10-19: I'm using your blasted web of information.

[Agent] 2023-10-29: Hail! What troubles bring you to my lair?
[Client] 2023-10-29: Greetings Tony, it seems you've taken my coin twice for my last order.
[Agent] 2023-10-29: My deepest regrets for your plight, ********. Could you share the number that marks your order so I may investigate this matter?
[Client] 2023-10-29: Indeed, it bears the mark 333666.

[Agent] 2023-11-08: How may I serve you this day?
[Client] 2023-11-08: Salutations, I made an order last week but I need to change the size.
[Agent] 2023-11-08: Certainly, ********. Could you provide the number that marks your order?
[Client] 2023-11-08: Yes, it bears the mark 444888. I am in your debt!

As you can see, a role prompt can have quite an impact on the language that the LLM uses to construct the response. This is great if you’re building a conversational agent that should speak in a certain tone or language. And you can also use system messages to keep specific setup information present.

For completion tasks like the one that you’re currently working on, you might, however, not need this type of role prompt. For now, you can give it a common boilerplate phrase, such as You’re a helpful assistant.

To practice writing a role prompt—and to see whether you can release your customer chat conversations from the reign of that 16th century villain poet—you’ll craft a more appropriate role prompt:

role_prompt = """You are a helpful assistant with a vast knowledge
of customer chat conversations.
You diligently complete tasks as instructed.
You never make up any information that isn't there."""

This role prompt is more appropriate to your use case. You don’t want the model to introduce randomness or to change any of the language that’s used in the conversations. Instead, you just want it to execute the tasks that you describe. Run the script another time and take a look at the results:

[Agent] 2023-07-15: Hello! What can I help you with today?
[Client] 2023-07-15: Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Client] 2023-07-15: It's "SAVE20".

[Agent] 2023-07-24: Good day! How can I help you?
[Client] 2023-07-24: Hi "********", I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Client] 2023-07-24: Sure, you have all my 😤 data already anyways. It's ********.

[Agent] 2023-08-13: Good morning! How may I assist you?
[Client] 2023-08-13: Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13: I'm sorry to hear that, ********. Could you tell me what device you're using?
[Client] 2023-08-13: I have an iPhone 11.

[Agent] 2023-08-30: Good evening! How may I assist you today?
[Client] 2023-08-30: Hi ********, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Client] 2023-08-30: Definitely, it's ********.

[Agent] 2023-09-01: Hello! How can I assist you this morning?
[Client] 2023-09-01: Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Client] 2023-09-01: It's saying "Payment method not valid".

[Agent] 2023-10-11: Good morning! How may I assist you?
[Client] 2023-10-11: Hello, I'd like to know the status of my order.
[Agent] 2023-10-11: Of course, ********. Could you please provide me with the order number?
[Client] 2023-10-11: It's ********.

[Agent] 2023-10-19: Welcome! How can I assist you right now?
[Client] 2023-10-19: 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Client] 2023-10-19: I'm using the 😤 website

[Agent] 2023-10-29: Hello! What can I help you with today?
[Client] 2023-10-29: Hi ********, I was charged twice for my last order.
[Agent] 2023-10-29: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Client] 2023-10-29: Sure, it's ********.

[Agent] 2023-11-08: How can I help you today?
[Client] 2023-11-08: Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08: Certainly, ********. Could you provide me the order number?
[Client] 2023-11-08: Yes, it's ********. Thanks!

That looks much better again! Abide concealed in yonder bygone era, ye villainous poet!

As you can see from these examples, role prompts can be a powerful way to change your output. Especially if you’re using the LLM to build a conversational interface, then they’re a force to consider.

In the final section of this tutorial, you’ll revisit using roles and see how you can employ the power of conversation to improve your output even in a non-conversational completion task like the one you’re working on.

Classify the Sentiment of Chat Conversations

At this point, you’ve engineered a decent prompt that seems to perform quite well in sanitizing and reformatting the provided customer chat conversations. To fully grasp the power of LLM-assisted workflows, you’ll next tackle the tacked-on request by your manager to also classify the conversations as positive or negative.

Start by saving both sanitized conversation files into new files that will constitute the new inputs for your sentiment classification task:

(venv) $ python app.py chats.txt > sanitized-chats.txt
(venv) $ python app.py testing-chats.txt > sanitized-testing-chats.txt

You could continue to build on top of the previous prompt, but eventually you’ll hit a wall when you’re asking the model to do too many edits at once. The classification step is conceptually distinct from the text sanitation, so it’s a good cut-off point to start a new pipeline.

The sanitized chat conversation files are also included in the example codebase:

Again you want the model do the work for you. All you need to do is craft a prompt that spells out the task at hand, and provide examples. You can also edit the role prompt to set the context for this new task that the model should perform:

instruction_prompt = """
Classify the sentiment of each conversation in >>>>>CONTENT<<<<<
with "🔥" for negative and "✅" for positive:

#### START EXAMPLES

------ Example Inputs ------
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

------ Example Outputs ------
🔥
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

✅
[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

#### END EXAMPLES
"""
role_prompt = """You are a thoroughly trained machine learning
model that is an expert at sentiment classification.
You diligently complete tasks as instructed.
You never make up any information that isn't there."""

You can now run the script and provide it with the sanitized conversations in sanitized-testing-chats.txt that were the output of your previously engineered prompt:

(venv) $ python app.py sanitized-testing-chats.txt

You added another step to your task description and slightly modified the few-shot examples in your prompt. Not a lot of extra work for a task that would have required a lot more work without the help of an LLM. But can this really be sufficient? Run the script again and take a look at the output:

🔥
[Agent] 2023-07-15: Hello! What can I help you with today?
[Client] 2023-07-15: Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Client] 2023-07-15: It's "SAVE20".

🔥
[Agent] 2023-07-24: Good day! How can I help you?
[Client] 2023-07-24: Hi "********", I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Client] 2023-07-24: Sure, you have all my 😤 data already anyways. It's ********.

✅
[Agent] 2023-08-13: Good morning! How may I assist you?
[Client] 2023-08-13: Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13: I'm sorry to hear that, ********. Could you tell me what device you're using?
[Client] 2023-08-13: I have an iPhone 11.

🔥
[Agent] 2023-08-30: Good evening! How may I assist you today?
[Client] 2023-08-30: Hi ********, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Client] 2023-08-30: Definitely, it's ********.

✅
[Agent] 2023-09-01: Hello! How can I assist you this morning?
[Client] 2023-09-01: Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Client] 2023-09-01: It's saying "Payment method not valid".

✅
[Agent] 2023-10-11: Good morning! How may I assist you?
[Client] 2023-10-11: Hello, I'd like to know the status of my order.
[Agent] 2023-10-11: Of course, ********. Could you please provide me with the order number?
[Client] 2023-10-11: It's ********.

🔥
[Agent] 2023-10-19: Welcome! How can I assist you right now?
[Client] 2023-10-19: 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Client] 2023-10-19: I'm using the 😤 website

✅
[Agent] 2023-10-29: Hello! What can I help you with today?
[Client] 2023-10-29: Hi ********, I was charged twice for my last order.
[Agent] 2023-10-29: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Client] 2023-10-29: Sure, it's ********.

✅
[Agent] 2023-11-08: How can I help you today?
[Client] 2023-11-08: Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08: Certainly, ********. Could you provide me the order number?
[Client] 2023-11-08: Yes, it's ********. Thanks!

The output is quite promising! The model correctly labeled conversations with angry customers with the fire emoji. However, the first conversation probably doesn’t entirely fit into the same bucket as the rest because the customer doesn’t display a negative sentiment towards the company.

Assume that all of these conversations were resolved positively by the customer service agents and that your company just wants to follow up with those customers who appeared noticeably angry at the situation they were facing. In that case, you might need to tweak your prompt a bit more to get the desired result.

You could add more examples, which is generally a good idea because it creates more context for the model to apply. Writing a more detailed description of your task helps as well, as you’ve seen before. However, to tackle this task, you’ll learn about another useful prompt engineering technique called chain-of-thought prompting.

Walk the Model Through Chain-of-Thought Prompting

A widely successful prompt engineering approach can be summed up with the anthropomorphism of giving the model time to think. You can do this with a couple of different specific techniques. Essentially, it means that you prompt the LLM to produce intermediate results that become additional inputs. That way, the reasoning doesn’t need to take distant leaps but only hop from one lily pad to the next.

Using chain-of-thought (CoT) prompting techniques is one of these approaches. To apply CoT, you prompt the model to generate intermediate results that then become part of the prompt in a second request. The increased context makes it more likely that the model will arrive at a useful output.

The smallest form of CoT prompting is zero-shot CoT, where you literally ask the model to think step by step. This approach yields impressive results for mathematical tasks that LLMs otherwise often solve incorrectly.

Chain-of-thought operations are technically split into two stages:

1. Reasoning extraction, where the model generates the increased context
2. Answer extraction, where the model uses the increased context to generate the answer

Reasoning extraction is useful across a variety of CoT contexts. You can generate few-shot examples from input, which you can then use for a separate step of extracting answers using more detailed chain-of-thought prompting.

You can give zero-shot CoT a try on the sanitized chat conversations to embellish the few-shot examples that you’ll then use to classify the chat conversations more robustly. Remove the examples and replace the instructions describing the reasoning on how you would classify the conversations in more detail:

instruction_prompt = """
Classify the sentiment of each conversation in >>>>>CONTENT<<<<<
with "🔥" for negative and "✅" for positive.

Follow these steps when classifying the conversations:
1. Does the customer use swear words or 😤?
2. Does the customer seem aggravated or angry?

If you answer "Yes" to one of the above questions,
then classify the conversation as negative with "🔥".
Otherwise classify the conversation as positive with "✅".

Let's think step by step
"""

You spelled out the criteria that you want the model to use to assess and classify sentiment. Then you add the sentence Let’s think step by step to the end of your prompt.

You want to use this zero-shot CoT approach to generate few-shot examples that you’ll then build into your final prompt. Therefore, you should run the script using the data in sanitized-chats.txt this time:

(venv) $ python app.py sanitized-chats.txt

You’ll get back a reference to the conversations, with the reasoning spelled out step by step to reach the final conclusion:

1. Conversation 1: The customer uses the 😤 emoji and seems aggravated, so the sentiment is negative. 🔥
2. Conversation 2: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅
3. Conversation 3: The customer uses the 😤 emoji and seems aggravated, so the sentiment is negative. 🔥
4. Conversation 4: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅
5. Conversation 5: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅
6. Conversation 6: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅
7. Conversation 7: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅
8. Conversation 8: The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅

The reasoning is straightforward and sticks to your instructions. If the instructions accurately represent the criteria for marking a conversation as positive or negative, then you’ve got a good playbook at hand.

You can now use this information to improve the few-shot examples for your sentiment classification task:

instruction_prompt = """
Classify the sentiment of each conversation in >>>>>CONTENT<<<<<
with "🔥" for negative and "✅" for positive.

#### START EXAMPLES

------ Example Inputs ------
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!
The customer uses the 😤 emoji and seems aggravated, so the sentiment is negative. 🔥

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!
The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive. ✅

------ Example Outputs ------
🔥
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!

✅
[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!

#### END EXAMPLES
"""

You’re using the same examples as previously, but you’ve enhanced each of the examples with a short chain of thought that you generated in the previous call. Give your script another spin using sanitized-testing-chats.txt as the input file and see whether the results have improved:

✅
[Agent] 2023-07-15: Hello! What can I help you with today?
[Client] 2023-07-15: Hey, my promo code isn't applying the discount in my cart.
[Agent] 2023-07-15: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?
[Client] 2023-07-15: It's "SAVE20".

🔥
[Agent] 2023-07-24: Good day! How can I help you?
[Client] 2023-07-24: Hi "********", I can't update my 😤 credit card information. Do you want my 😤 money or not?
[Agent] 2023-07-24: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?
[Client] 2023-07-24: Sure, you have all my 😤 data already anyways. It's ********.

✅
[Agent] 2023-08-13: Good morning! How may I assist you?
[Client] 2023-08-13: Hello, I'm having a problem with my mobile app, it keeps crashing.
[Agent] 2023-08-13: I'm sorry to hear that, ********. Could you tell me what device you're using?
[Client] 2023-08-13: I have an iPhone 11.

🔥
[Agent] 2023-08-30: Good evening! How may I assist you today?
[Client] 2023-08-30: Hi ********, I've forgotten my 😤 password and I can't login into my account.
[Agent] 2023-08-30: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?
[Client] 2023-08-30: Definitely, it's ********.

✅
[Agent] 2023-09-01: Hello! How can I assist you this morning?
[Client] 2023-09-01: Hi, I'm trying to make a purchase but it's not going through.
[Agent] 2023-09-01: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?
[Client] 2023-09-01: It's saying "Payment method not valid".

✅
[Agent] 2023-10-11: Good morning! How may I assist you?
[Client] 2023-10-11: Hello, I'd like to know the status of my order.
[Agent] 2023-10-11: Of course, ********. Could you please provide me with the order number?
[Client] 2023-10-11: It's ********.

🔥
[Agent] 2023-10-19: Welcome! How can I assist you right now?
[Client] 2023-10-19: 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?
[Agent] 2023-10-19: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?
[Client] 2023-10-19: I'm using the 😤 website

✅
[Agent] 2023-10-29: Hello! What can I help you with today?
[Client] 2023-10-29: Hi ********, I was charged twice for my last order.
[Agent] 2023-10-29: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?
[Client] 2023-10-29: Sure, it's ********.

✅
[Agent] 2023-11-08: How can I help you today?
[Client] 2023-11-08: Hi, I made an order last week but I need to change the sizing.
[Agent] 2023-11-08: Certainly, ********. Could you provide me the order number?
[Client] 2023-11-08: Yes, it's ********. Thanks!

Great! Now the first conversation, which was initially classified as negative, has also received the green checkmark.

In this section, you’ve supported your examples with reasoning for why a conversation should be labeled as positive vs negative. You generated this reasoning with another call to the LLM.

At this point, it seems that your prompt generalizes well to the available data and classifies the conversations as intended. And you only needed to carefully craft your words to make it happen!

Structure Your Output Format as JSON

As a final showcase for effective prompting when incorporating an LLM into your workflow, you’ll tackle the last task, which you added to the list youself: to pass the data on in a structured format that’ll make it straightforward for the customer support team to process further.

You already specified a format to follow in the previous prompt, and the LLM returned what you asked for. So it might just be a matter of asking for a different, more structured format, for example JSON:

instruction_prompt = """
Classify the sentiment of each conversation in >>>>>CONTENT<<<<<
as "negative" and "positive".
Return the output as valid JSON.

#### START EXAMPLES

------ Example Input ------

[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!
The customer uses the 😤 emoji and seems aggravated, so the sentiment is negative.

[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!
The customer does not use any swear words or 😤 emoji and does not seem aggravated or angry, so the sentiment is positive.

------ Example Output ------

{
  "negative": [
    {
      "date": "2023-07-24",
      "conversation": [
        "A: What can I help you with?",
        "C: I CAN'T CONNECT TO MY 😤 ACCOUNT",
        "A: Are you sure it's not your caps lock?",
        "C: 😤! You're right!"
      ]
    }
  ],
  "positive": [
    {
      "date": "2023-06-15",
      "conversation": [
        "A: Hello! How can I assist you today?",
        "C: I can't seem to find the download link for my purchased software.",
        "A: No problem, ********. Let me find that for you. Can you please provide your order number?",
        "C: It's ********. Thanks for helping me out!"
      ]
    }
  ]
}

#### END EXAMPLES
"""

In your updated instruction_prompt, you’ve explicitly asked the model to return the output as valid JSON. Then, you also adapted your few-shot examples to represent the JSON output that you want to receive. Note that you also applied additional formatting by removing the date from each line of conversation and truncating the [Agent] and [Customer] labels to single letters, A and C.

You’re still using example chat conversations from your sanitized chat data in sanitized-chats.txt, and you send the sanitized testing data from sanitized-testing-chats.txt to the model for processing.

In this case, you receive valid JSON, as requested. However, the previously misclassified message that you just fixed hops back into the negative bucket. Also, the model doesn’t apply all the additional requested formatting:

{
  "negative": [
    {
      "date": "2023-07-15",
      "conversation": [
        "[Agent] 2023-07-15: Hello! What can I help you with today?",
        "[Client] 2023-07-15: Hey, my promo code isn't applying the discount in my cart.",
        "[Agent] 2023-07-15: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?",
        "[Client] 2023-07-15: It's \"SAVE20\"."
      ]
    },
    {
      "date": "2023-07-24",
      "conversation": [
        "[Agent] 2023-07-24: Good day! How can I help you?",
        "[Client] 2023-07-24: Hi \"********\", I can't update my 😤 credit card information. Do you want my 😤 money or not?",
        "[Agent] 2023-07-24: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?",
        "[Client] 2023-07-24: Sure, you have all my 😤 data already anyways. It's ********."
      ]
    },
    {
      "date": "2023-08-30",
      "conversation": [
        "[Agent] 2023-08-30: Good evening! How may I assist you today?",
        "[Client] 2023-08-30: Hi ********, I've forgotten my 😤 password and I can't login into my account.",
        "[Agent] 2023-08-30: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?",
        "[Client] 2023-08-30: Definitely, it's ********."
      ]
    },
    {
      "date": "2023-10-19",
      "conversation": [
        "[Agent] 2023-10-19: Welcome! How can I assist you right now?",
        "[Client] 2023-10-19: 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?",
        "[Agent] 2023-10-19: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?",
        "[Client] 2023-10-19: I'm using the 😤 website"
      ]
    }
  ],
  "positive": [
    {
      "date": "2023-08-13",
      "conversation": [
        "[Agent] 2023-08-13: Good morning! How may I assist you?",
        "[Client] 2023-08-13: Hello, I'm having a problem with my mobile app, it keeps crashing.",
        "[Agent] 2023-08-13: I'm sorry to hear that, ********. Could you tell me what device you're using?",
        "[Client] 2023-08-13: I have an iPhone 11."
      ]
    },
    {
      "date": "2023-09-01",
      "conversation": [
        "[Agent] 2023-09-01: Hello! How can I assist you this morning?",
        "[Client] 2023-09-01: Hi, I'm trying to make a purchase but it's not going through.",
        "[Agent] 2023-09-01: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?",
        "[Client] 2023-09-01: It's saying \"Payment method not valid\"."
      ]
    },
    {
      "date": "2023-10-11",
      "conversation": [
        "[Agent] 2023-10-11: Good morning! How may I assist you?",
        "[Client] 2023-10-11: Hello, I'd like to know the status of my order.",
        "[Agent] 2023-10-11: Of course, ********. Could you please provide me with the order number?",
        "[Client] 2023-10-11: It's ********."
      ]
    },
    {
      "date": "2023-10-29",
      "conversation": [
        "[Agent] 2023-10-29: Hello! What can I help you with today?",
        "[Client] 2023-10-29: Hi ********, I was charged twice for my last order.",
        "[Agent] 2023-10-29: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?",
        "[Client] 2023-10-29: Sure, it's ********."
      ]
    },
    {
      "date": "2023-11-08",
      "conversation": [
        "[Agent] 2023-11-08: How can I help you today?",
        "[Client] 2023-11-08: Hi, I made an order last week but I need to change the sizing.",
        "[Agent] 2023-11-08: Certainly, ********. Could you provide me the order number?",
        "[Client] 2023-11-08: Yes, it's ********. Thanks!"
      ]
    }
  ]
}

Despite the small hiccups, this output is quite impressive and useful! You could pass this JSON structure over to the customer support team, and they could quickly integrate it into their workflow to follow up with customers who displayed a negative sentiment in the chat conversation.

You could stop here, but the engineer in you isn’t quite satisfied yet. The format isn’t exactly what you wanted, and one of the messages really shouldn’t be classified as negative. Can you fix these two remaining issues as well before setting up your LLM-assisted pipeline and calling it a day?

Improve Your Output With the Power of Conversation

You switched to using a newer model on the /chat/completions endpoint earlier on, which also required you to assemble your prompt differently. You added a role prompt, but otherwise you haven’t tapped into the power of conversations yet.

In this final section, you’ll learn how you can provide additional context to the model by splitting your prompt into multiple separate messages with different labels.

In calls to the newer /chat/completions endpoint, a prompt is split into several messages. Each message has its content, which represents the prompt text. Additionally, it also has a role. There are different roles that a message can have, and you’ll work with three of them:

1. "system" gives context for the conversation and helps to set the overall tone.
2. "user" represents the input that a user of your application might provide.
3. "assistant" represents the output that the model would reply with.

So far, you’ve provided context for different parts of your prompt all mashed together in a single prompt, more or less well separated using delimiters. When you use a model that’s optimized for chat, such as GPT-4, then you can use roles to let the LLM know what type of message you’re sending.

For example, you can create some variables for your few-shot examples and separate variables for the associated CoT reasoning and outputs:

[prompts]
instruction_prompt = """
Classify the sentiment of each conversation in >>>>>CONTENT<<<<<
as "negative" and "positive".
Return the output as valid JSON.
"""
role_prompt = """You are a thoroughly trained machine learning model
that is an expert at sentiment classification.
You diligently complete tasks as instructed.
You never make up any information that isn't there."""
positive_example = """
[Agent] 2023-06-15 : Hello! How can I assist you today?
[Customer] 2023-06-15 : I can't seem to find the download link for my purchased software.
[Agent] 2023-06-15 : No problem, ********. Let me find that for you. Can you please provide your order number?
[Customer] 2023-06-15 : It's ********. Thanks for helping me out!
"""
positive_reasoning = """The customer does not use any swear words or 😤 emoji
and does not seem aggravated or angry, so the sentiment is positive."""
positive_output = """
{
  "positive": [
    {
      "date": "2023-06-15",
      "conversation": [
        "A: Hello! How can I assist you today?",
        "C: I can't seem to find the download link for my purchased software.",
        "A: No problem, ********. Let me find that for you. Can you please provide your order number?",
        "C: It's ********. Thanks for helping me out!"
      ]
    }
  ]
}
"""
negative_example = """
[Agent] 2023-07-24 : What can I help you with?
[Customer] 2023-07-24 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[Agent] 2023-07-24 : Are you sure it's not your caps lock?
[Customer] 2023-07-24 : 😤! You're right!
"""
negative_reasoning = """The customer uses the 😤 emoji and seems aggravated,
so the sentiment is negative."""
negative_output = """
{
  "negative": [
    {
      "date": "2023-07-24",
      "conversation": [
        "A: What can I help you with?",
        "C: I CAN'T CONNECT TO MY 😤 ACCOUNT",
        "A: Are you sure it's not your caps lock?",
        "C: 😤! You're right!"
      ]
    }
  ]
}
"""

You’ve disassembled your instruction_prompt into seven separate prompts, based on what role the messages have in your conversation with the LLM.

The helper function that builds a messages payload, assemble_chat_messages(), is already set up to include all these prompts in the API request. Take a look into app.py to check out the separate messages, with their fitting roles, that make up your overall prompt:

# app.py

# ...

def assemble_chat_messages(content: str, settings: Settings) -> list[dict]:
    """Combine all messages into a well-formatted dictionary."""
    return [
        {"role": "system", "content": settings.role_prompt},
        {"role": "user", "content": settings.negative_example},
        {"role": "system", "content": settings.negative_reasoning},
        {"role": "assistant", "content": settings.negative_output},
        {"role": "user", "content": settings.positive_example},
        {"role": "system", "content": settings.positive_reasoning},
        {"role": "assistant", "content": settings.positive_output},
        {"role": "user", "content": f">>>>>\n{content}\n<<<<<"},
        {"role": "user", "content": settings.instruction_prompt},
    ]

Your prompt is now split into distinct parts, each of which has a certain role label:

• Example input has the "user" role.
• Reasoning that the model created has the "system" role.
• Example output has the "assistant" role.

You’re now providing context for how user input might look, how the model can reason about classifying the input, and how your expected output should look. You removed the delimiters that you previously used for labeling the example sections. They aren’t necessary now that you’re providing context for the parts of your prompt through separate messages.

Give your script a final run to see whether the power of conversation has managed to improve the output:

{
  "positive": [
    {
      "date": "2023-07-15",
      "conversation": [
        "A: Hello! What can I help you with today?",
        "C: Hey, my promo code isn't applying the discount in my cart.",
        "A: My apologies for the trouble, ********. Could you tell me the promo code you're trying to use?",
        "C: It's \"SAVE20\"."
      ]
    },
    {
      "date": "2023-08-13",
      "conversation": [
        "A: Good morning! How may I assist you?",
        "C: Hello, I'm having a problem with my mobile app, it keeps crashing.",
        "A: I'm sorry to hear that, ********. Could you tell me what device you're using?",
        "C: I have an iPhone 11."
      ]
    },
    {
      "date": "2023-09-01",
      "conversation": [
        "A: Hello! How can I assist you this morning?",
        "C: Hi, I'm trying to make a purchase but it's not going through.",
        "A: I'm sorry to hear that, ********. Can you tell me what error message you're receiving?",
        "C: It's saying \"Payment method not valid\"."
      ]
    },
    {
      "date": "2023-10-11",
      "conversation": [
        "A: Good morning! How may I assist you?",
        "C: Hello, I'd like to know the status of my order.",
        "A: Of course, ********. Could you please provide me with the order number?",
        "C: It's ********."
      ]
    },
    {
      "date": "2023-10-29",
      "conversation": [
        "A: Hello! What can I help you with today?",
        "C: Hi ********, I was charged twice for my last order.",
        "A: I'm sorry to hear that, ********. Could you share your order number so I can look into this for you?",
        "C: Sure, it's ********."
      ]
    },
    {
      "date": "2023-11-08",
      "conversation": [
        "A: How can I help you today?",
        "C: Hi, I made an order last week but I need to change the sizing.",
        "A: Certainly, ********. Could you provide me the order number?",
        "C: Yes, it's ********. Thanks!"
      ]
    }
  ],
  "negative": [
    {
      "date": "2023-07-24",
      "conversation": [
        "A: Good day! How can I help you?",
        "C: Hi \"********\", I can't update my 😤 credit card information. Do you want my 😤 money or not?",
        "A: I'm sorry for the inconvenience, ********. Can you please confirm your account's email?",
        "C: Sure, you have all my 😤 data already anyways. It's ********."
      ]
    },
    {
      "date": "2023-08-30",
      "conversation": [
        "A: Good evening! How may I assist you today?",
        "C: Hi ********, I've forgotten my 😤 password and I can't login into my account.",
        "A: I'm sorry for the trouble, ********. Could you confirm your email address so we can reset your password?",
        "C: Definitely, it's ********."
      ]
    },
    {
      "date": "2023-10-19",
      "conversation": [
        "A: Welcome! How can I assist you right now?",
        "C: 😤! There's no option to change my profile picture. What kind of 😤 joint are you running?",
        "A: Let me help you with this, ********. Are you trying to update it from the mobile app or the website?",
        "C: I'm using the 😤 website"
      ]
    }
  ]
}

This JSON structure is looking legitimately great! The formatting that you wanted now shows up throughout, and even the stray conversation is again labeled correctly as positive. You can feel proud to pass on such a useful edit of the customer chat conversation data to your coworkers!

Key Takeaways

You’ve covered common prompt engineering techniques, and here, you’ll find a few questions and answers that sum up the most important concepts that you’ve covered in this tutorial.

You can use these questions to check your understanding or to recap and solidify what you’ve just learned. After each question, you’ll find a brief explanation hidden in a collapsible section. Click the Show/Hide toggle to reveal the answer. Time to dive in!

Next Steps

In this tutorial, you’ve learned about various prompt engineering techniques, and you’ve built an LLM-assisted Python application along the way. If you’d like to learn more about prompt engineering, then check out some related questions, as well as some resources for further study below:

The field of prompt engineering is quite new, and LLMs keep developing quickly as well. The landscape, best practices, and most effective approaches are therefore changing rapidly. To continue learning about prompt engineering using free and open-source resources, you can check out Learn Prompting and the Prompt Engineering Guide.

Have you found any interesting ways to incorporate an LLM into your workflow? Share your thoughts and experiences in the comments below.