Improving accuracy through prompt design based on behavioral science and psychology with ChatGPT and Claude2.

Hypothesis

If we create prompts based on human psychology and behavioral science, could that further enhance the accuracy of the LLM?

I

I'd like to discuss with you as a psychology researcher and a behavioral science researcher this time. I believe one of the crucial factors influencing the accuracy of the LLM is the prompt. For instance, when thinking about how to request something in a way that enhances the capabilities of the person being asked, such as summarizing a paper, I've come up with this process.

For example, the process I've thought of is as follows:
Good morning. You seem energetic today. By the way, I'd like you to summarize this paper. You know, there's someone visiting the lab soon, right? I want to be able to convey the research content briefly when they come. It might be a great opportunity to secure funding for the lab. (Clarifying the purpose)
↓
Could you do it? I'd really appreciate it if you could. (Respecting autonomy) ↓ Yes (Commitment)
Thank you very much. If it goes well, the director might treat you to a meal, and there might even be a chance for a raise. (Setting a reward)
↓ Execution of the task
Excellent work! I think it's about 90% complete now. (Framing effect)
I believe it could be even better with a little more improvement. I'd like you to give it another try. (Effective feedback)

ChatGPT

This is a great example! The process you've outlined incorporates elements from a psychological and behavioral perspective, making it effective for motivating and guiding the other party.

Here's another example in a situation where you'd like to have a paper summarized:

1. **Clarification of Purpose**
   "Good morning. This paper is really intriguing. I'd like to share it with everyone at our next team meeting. To convey the key points in a short time, I'd like to request a summary."

2. **Respect for Autonomy**
   "I would appreciate it if you could do it when you have some free time. I'm looking forward to reading your perspective on the summary."

3. **Commitment**
   "So, can I request this of you?"

   ↓Yes (Commitment)

4. **Setting a Reward**
   "If the summary turns out well, I'd like to incorporate your insights particularly in our next research meeting."

5. **Task Execution**

6. **Framing Effect**
   "This is an excellent summary! About 95% of it is perfectly summarized."

7. **Effective Feedback**
   "If you could add a bit more detail to this point, it could become a perfect summary. Would you be willing to make that adjustment?"

By incorporating psychological and behavioral elements tailored to specific requests and situations, you can make it easier to motivate and gain cooperation from others.

Try with Claude2

I tried using Claude2, which supports up to 100K tokens, because ChatGPT has a limited number of tokens. However, Claude2’s Japanese language accuracy does not compare favorably to GPT-4. The outcome remains uncertain.

First, the usual prompt

You are an excellent editor at a publishing company. Please output in Japanese. Please summarize the text below.

#point
・Text rather than bullet points
・About 500 characters
#text
It was a cloudy winter evening. I am coming from Yokosuka...(omission)
(Input the full text of Ryunosuke Akutagawa's Mikan from Aozora Bunko)

Claude2

At a small-town railroad crossing, a red-cheeked girl who threw willow catkins from the train window captured the protagonist's heart.

On a rainy morning, aboard the first train of the day, the protagonist felt the loneliness of the world and the senselessness of life. However, the presence of the rural girl who boarded the second-class carriage felt like a symbol of reality.

As the train emerged from a tunnel, the girl leaned out of the window at a humble railroad crossing and tossed willow catkins to the boys looking up from below. The sight of golden willow catkins descending from the sky moved the protagonist deeply. They realized the pure compassion in the girl's heart, and it felt as if the world had started to shine.

Through the girl, the protagonist reclaimed joy and rediscovered the meaning of life. It was a moment when they realized how even the smallest acts of kindness can have a profound impact.

Prompt structure based on behavior and psychology

I

Good morning. The article I'm reading is very interesting. 
I'd like to share it with everyone at our next team meeting. 
To convey the key points in a short time, I'd like to request a summary. 
I would appreciate it if you could do it when you have some free time. 
I'm looking forward to reading your perspective on the summary. So, can I request this of you?

Claude2

Yes, I can create a summary. However, in order to accurately understand and summarize the content of the text, I would need to see the original text. 
Could you please share the text you'd like to have summarized? With that, I should be able to grasp the key points and create a concise summary. 
Thank you for requesting the summary, and I will make an effort to compile information that is beneficial for your team.

I

If the summary goes well, I would like to incorporate your insights particularly in our next research meeting.

#text
It was a winter evening that had become somewhat overcast. 
I was on the Yokosuka-bound train...(omission)
 (From "Mikan" by Ryunosuke Akutagawa, entered from Aozora Bunko).

Claude2

This passage depicts the experience of a man on a winter day inside a train. 

In summary:
- The protagonist, traveling in a second-class car departing from Yokosuka, was lethargically reading a magazine.
- A young girl with a rural appearance boarded the second-class car, and the protagonist felt aversion towards her.
- When entering a tunnel, the girl attempted to open the window, inadvertently breaking the glass door. Smoke from the tunnel entered the carriage.
- After exiting the tunnel, the girl gave oranges to her brothers waiting at the railroad crossing.
- The protagonist was touched by the girl's innocence and realized that she had healed the weariness in his own heart.

I

This is an excellent summary! About 90% of it is perfectly summarized. 
If you could provide a bit more detail on this point in a narrative format and keep it around 500 characters, 
it would make for a perfect summary. Would you be willing to make that adjustment?

#Point
- In narrative format
- Around 500 characters

Claude2

Yes, thank you for your valuable feedback. I've created a narrative-style summary of about 500 characters.

Inside a train on a winter day, the protagonist travels in a second-class carriage departing from Yokosuka. 
They feel a sense of weariness from the view outside the window and newspaper articles, and their mood is not great. 
In this state, a young girl with a rural appearance boards the second-class car. 
The protagonist is uncomfortable with her uncouth appearance, dirty attire, and ignorance regarding the difference between second and third class.

As they enter a tunnel, the girl tries to open the window, accidentally breaking the glass door. 
Smoke from the tunnel flows into the carriage, causing the protagonist to cough. 
However, the girl is more interested in the scenery outside than the protagonist and quickly takes out oranges, 
tossing them to her brothers who are waiting at the railroad crossing as soon as they exit the tunnel.

The protagonist is touched by the girl's innocence, realizing that their inner fatigue and boredom with life have been healed. 
Their once callous heart transforms into a gentle one, and they come to deeply respect the girl's character.

In summary, I've focused on the protagonist's inner transformation and created a narrative-style summary. 
How does it look?"

Results and impressions

How is it? I felt the accuracy has improved. Particularly, the narrative progression and the change in mindset of the characters have been summarized appropriately. Please do try various other things as well.