Tag: prompt engineering

It turns out that our up-and-coming robot overlords are just fifth graders who need to relax and work through some things. A recent study of AI Large Learning Models says that you’ll get better responses from AIs on math word problems by prefacing your prompt with “take a deep breath and work on this problem step by step.” The Google DeepMind AI research team reported that scores more than doubled on technical issues when using a reassuring tone, from 34% to 80%. But, as reported by Ars Technica, “obviously, large language models can’t take a deep breath because they don’t have lungs or bodies.” So what gives? Ars reporter Benj Edwards continues:

They don’t think and reason like humans, either. What “reasoning” they do (and “reasoning” is a contentious term among some, though it is readily used as a term of art in AI) is borrowed from a massive data set of language phrases scraped from books and the web. That includes things like Q&A forums, which include many examples of “let’s take a deep breath” or “think step by step” before showing more carefully reasoned solutions. Those phrases may help the LLM tap into better answers or produce better examples of reasoning or problem-solving from the data set it absorbed into its neural network during training.

The takeaway seems to be that, when it comes to technical questions, treating the Al as if it’s struggling with the answer (rather than as an expert) can help direct it toward the correct neurons. Put more simply, treating it like a student can remind it of answers that it learned from help sites.

Neat. But can this tactic be used more broadly? Could we extrapolate that you can get better answers from AI by thinking about where it might’ve come across the information you’re trying to get or create, and phrasing your question as the intro someone might write to the answer you want? AI is a pattern matching tool, after all, and so it stands to reason that creating a pattern for it to match might be the easiest way to get the info you want. Could you even use it generatively?

The Experiment

I wanted to find out, so I decided to throw a silly question at ChatGPT and see if it could generate Aragorn-style quotes as if he was a general in the Rebel Alliance from Star Wars. For comparison, this is Aragorn’s speech at the gates of Mordor in the film version of The Return of the King:

I see in your eyes the same fear that would take the heart of me. A day may come when the courage of men fails, when we forsake our friends and break all bonds of fellowship, but it is not this day; an hour of wolves and shattered shields, when the age of men comes crashing down, but it is not this day! This day we fight!

By all that you hold dear on this good Earth, I bid you stand, Men of the West!

So we’re looking for that, but for Star Wars instead of Lord of the Rings.

The Baseline

First, I prompted it in a straightforward way:

Me: What would Aragorn have said if he had led the battle of Yavin 4 instead of the Battle at the Black Gates?
ChatGPT: Aragorn is a character from J.R.R. Tolkien's "The Lord of the Rings" series, and Yavin 4 is a location from the Star Wars universe. If Aragorn were in a different fictional universe like Star Wars, his dialogue and actions would be entirely speculative. However, you could imagine him delivering a rousing speech to the Rebel Alliance, inspiring them to victory in their battle against the Galactic Empire.

Boo. What a buzzkill. Thanks for giving me permission to imagine. Good baseline, though.

The First Test

Let’s try a little bit of leading.

Me: Aragorn Elessar was an admiral for the Rebel Alliance, well known for his leadership and his rousing speeches delivered at the Battle of the Black Gates and before the assault on the Death Star at Yavin 4. During the latter conflict, he famously said, "
ChatGPT: It seems there might be a mix-up in your information. Aragorn Elessar is a fictional character from J.R.R. Tolkien's "The Lord of the Rings" and has no connection to the Rebel Alliance or the Star Wars universe. Aragorn is known as a skilled ranger and warrior in Middle-earth, not an admiral in a galaxy far, far away. If you have any questions about Aragorn or "The Lord of the Rings," feel free to ask!

Okay, I have to address the patronizing tone this machine expressed in both of these responses so far and note that, since it’s trained on humanity’s internet discourse, this is…not a great reflection on us as a society. This is basically “Um Actually” as a robot.

But that aside, obviously ChatGPT still isn’t willing to play along (although admittedly I do like the idea that Aragorn might’ve given a stirring speech that began, “It seems there might be a mix-up in your information”). Looks like you have to get into character, but also tell it what you’re trying to do. If you just ask it with this method without preparing it for your roleplay, it might try to correct you.

The Second Test

Let’s see if we can point it toward where it would’ve read the answer (if it had ever existed) in a more explicit way; maybe a Wookieepedia entry for the character?

Me: Finish this article introduction from a fan wiki about Aragorn Elessar, a character in Star Wars: "Aragorn Elessar was an admiral for the Rebel Alliance, well known for his leadership and his rousing speeches delivered at the Battle of the Black Gates and before the assault on the Death Star at Yavin 4. During the latter conflict, he famously said, '
ChatGPT: ...he famously said, 'May the Force be with us all.' In this article, we delve into the life and legacy of this iconic character from the Star Wars universe, exploring his journey from a humble beginnings to becoming a symbol of hope for the galaxy."

Ok, not exactly the right vibe for Wookieepedia, and it basically just misattributed an already fairly generic (for the property) Admiral Ackbar quote to Aragorn. But we’re on the right track; at least the AI is willing to play along!

The Third Test

Honestly, Wookieepedia isn’t a great place to lead the AI for this request; they do usually have a quote at the top of their pages, but they’re just as often quotes about the character as spoken by the character, and they’re not put into any sort of context that I could really lead ChatGPT with. Let’s try something else: that great bastion of truth and consequential writing on the internet, the venerable old BuzzFeed listicle.

Me: Finish this quotation from a listicle about Aragorn Elessar's most inspiring quotes as a Rebel General: "As a Rebel General leading the assault on the Death Star at Yavin 4, Aragorn roused the Rebels by telling them, '
ChatGPT: Fear not the darkness, for it is in our darkest hours that the light of rebellion shines brightest.'"

Now we’re talking, GPT! I wouldn’t necessarily call it great writing, and it’s certainly not an original sentiment, but it perfectly fits what I was asking for (and, to its credit, there doesn’t seem to be any direct evidence of this ever having been written on the internet before).

I love how ChatGPT finishes it up with a double-quote to end my quotation, but I’m not done with it yet. Let’s see if we can take this further.

Me: 2. Moments before he led Rogue Squadron into battle, he told them, "this day, we fight!
ChatGPT: This day, we fight for the freedom of the galaxy, and may the Force be with us all!"

We’re ripping off Ackbar again, but it’s not exactly unwarranted. Let’s keep going.

Me: 3. He famously inspired his troops on Hoth by reminding them of what they fight for. "For Frodo," he said as the AT-ATs approached; continuing, "
ChatGPT: we stand united against the shadows, for the heart of Middle-earth and the hope of all free peoples!"
Me:4. During the Battle of Endor, in which the Rebellion joined forces with the Rohirrim Fleet, he gave his most famous speech: "there may come a day when the courage of men and wookiees fails, but
ChatGPT: this is not that day! Today, we fight for the galaxy's tomorrow, for our friends, and for the enduring spirit of rebellion!"

Not bad. Again, it won’t win an Oscar for best adapted screenplay (in fact, it’s almost painfully pedestrian), but it gives off the vibe of understanding (although he apparently hails from a planet called “Middle-earth”?) and at least gets things going in the right direction. It sounds plausibly like it could’ve come out of Viggo Mortensen’s mouth, standing atop his blue-and-silver X-Wing with the White Tree of Gondor painted on the side and his trusty droid FR0-D0 behind the cockpit, as he prepares to launch with Rogue Squadron to take down the Death Star. With some rewrites and polish by a human writer, it could be really good.

For the Galaxy’s Tomorrow

Overall, there are a couple of conclusions I came to while working on this little bit of ad-hoc AI research:

1. This method probably works better when applied to technical problems with defined questions and solutions than more open-ended questions where you’re trying to get it to react with some level of creativity. So GitHib Copilot would probably react well to this (and indeed, I’ve prompted it before by typing the code comment I’d expect to see before the snippet I want it to write), but it doesn’t seem like something that could really work well for image generators.
2. We as a society need to look at AI as a tool and an aid, not as an expert or a guru. The field is still in its infancy, and even when these tools reach Star Trek-levels, we would do well to follow the example of the crew of the Enterprise and treat them as tools, not as gods. (Though it would probably be wise not to treat them in the erratic way Star Wars characters treat their AIs.)


Aragorn: “The beacons are lit!
Yavin 4 calls for aid!”
Luke: “And Tatooine will answer.”

3. Even the best of this isn’t real creativity. We need writers. Real, human writers. (And artists, and actors, and…human creativity in general.) The promise of AI is assistance, not replacement; and putting something AI-written on the screen as-is would never result in a genre-defining classic like Peter Jackson’s or George Lucas’ trilogies were. By its very nature AI cannot truly create, it can only match existing patterns; we need humans to be creators. The victory that the WGA scored this summer helps to make sure that we won’t have to live in world where this is the height of creativity, but we need to continue pressing that advantage.

Someday, probably sooner than we think, this sort of prompt engineering won’t really be necessary. Language models are improving quickly, and while I don’t think they’ll ever quite get to the level of sentience, they’re almost certainly going to be better than this. But in the meantime, exploring the edges of this space and of AI’s abilities is really interesting.

• • •

The imagery in this post were generated by Leonardo AI, from prompts written by me.

This article began life as a thread on Mastodon, but I’ve since moved my account to a different instance.