Sarcasm and ChatGPT Simple Sentiment Analysis

I had fun with some sentiment analysis testing using the OpenAI gpt-3.5-turbo model. This came as a tangent as I was working through the Prompt Engineering for Developers class offered by deeplearning.ai. The idea is to scan through prompts on a product review site – think eBay or customer generated listings on eCommerce sites. Without needing to create a bunch of ML models, we can get a pretty amazing outcome:

Text to summarize – a made up product review

The Enchanted Cottage Dollhouse is for busy
moms of energetic 8-year-olds. While the wait for its
arrival was two days longer than expected, the beautifully
crafted design and vibrant colors are certainly appealing.
My daughter spent hours captivated by arranging the
furniture and creating stories. However, some of the smaller
accessories and furniture pieces seem a bit delicate and may not
withstand rough play. I had to have one part replaced, which arrived
the next day. While the overall construction is
durable, it would be helpful if the manufacturer could improve
the sturdiness of the smaller components. In conclusion, the
dollhouse has brought joy and memories to our home, but be prepared
for the possibility of some minor breakages.

Generic product review

import openai
import os

from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) # read local .env file

openai.api_key  = os.getenv('OPENAI_API_KEY')

def get_response(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0,         # degree of randomness of the model output
        max_tokens=60,         # max tokens to generate in the completion.
        frequency_penalty=0.0, # how likely to repeat the same line
        presence_penalty=0.0   # penalize new tokens based on whether they appear in the text so far
    )
    return response.choices[0].message["content"]

prompt = f"""
What is the sentiment of the following product review, 
which is delimited with triple backticks?

Review text: '''{prod_review}'''
"""
response = get_response(prompt)
print(response)

GPT3.5 provided the following classification to the review:

The sentiment of the product review is generally positive, with some minor concerns about the sturdiness of smaller components.

Now I’m curious… how small a change can we make to tip the review to negative? How about some sarcasm?

The Enchanted Cottage Dollhouse is for busy
moms of energetic 8-year-olds. While the wait for its
arrival was two days longer than expected, the beautifully
crafted design and vibrant colors are certainly appealing.
My daughter spent hours captivated by arranging the
furniture and creating stories. However, some of the smaller
accessories and furniture pieces seem a bit delicate and may not
withstand rough play. I had to have one part replaced, which arrived
the next day. Quality, ha! While the overall construction is
durable, it would be helpful if the manufacturer could improve
the sturdiness of the smaller components. In conclusion, the
dollhouse has brought joy and memories to our home, but be prepared
for the possibility of some minor breakages.

Just adding those two words tipped off the model that something more is amiss. I reran the classification and got the result:

The sentiment of the product review is generally positive, with some minor criticisms.

Still not tipping into negative, but I was surprised that the little bit of sarcasm, even though “quality” is usually a positive word, would be detected as less than a positive. I am only scratching the surface, but I honestly have the feeling I’ve met a new person and I’m getting to know how they think and view the world. Very exciting!