Lab 3: LLM-Based Human-Robot Dialogue


Learning Goals

Slides

You can find the slides from today's lab here.

Lab Setup

Preparing Your Development Environment

Beyond the steps you've already taken in Lab 1, this lab requires a few additional setup steps. Follow these in order.

  1. Create your Lab 3 folder: Inside EECS690_HRI_Lab, create a folder named lab_3_LLM_based_human_robot_dialogue for the lab code: 
    cd EECS690_HRI_Lab
  mkdir lab_3_LLM_based_human_robot_dialogue
  2. Install additional Python packages: Activate your virtual environment and run: 
    source venv/bin/activate
  pip install openai
  pip install dotenv
  pip install paramiko
    You should already have the qi module installed from Lab 1.
  3. Store API keys in a .env file: Create a file named .env inside the baseEECS690_HRI_Lab folder, and paste the OpenAI API keys you were provided via email/Canvas. The .env file should look like this: 
    OPENAI_API_KEY=your_openai_api_key_here
  4. Add the starter code: Inside EECS690_HRI_Lab either use our template repo or manually copy the starter files into the lab_3_LLM_based_human_robot_dialogue folder.
    The starter code is available on GitHub: Starter Code
  5. Use this folder structure: 
    EECS690_HRI_Lab/
  ├── .env
  ├── venv/
  ├── lab_1_nao_introduction/
  │   └── nao_introduction.py
  ├── lab_2_nao_woz_gui/
  │   └── nao_woz_gui.py
  └── lab_3_LLM_based_human_robot_dialogue/
      ├── llm_based_human_robot_dialogue.py
      ├── three_good_things_system_instruction.txt
      ├── test_dependencies.py
      ├── motion_test.py
      └── gen_ai_test.py
    Important: your lab folders and your .env file should both be directly inside EECS690_HRI_Lab.
  6. Test your setup: From within your lab folder (with the virtual environment activated) run: 
    python3 test_dependencies.py
    If everything is configured correctly, the script should exit without errors.

Working in Groups

During this lab, you will work with a different group than you worked with for Lab 2. Similar to Lab 2, each individual will turn in their own set of deliverables.

Lab 3 Deliverables & Submission

With the starter code we've provided, in Lab 3 you are expected to:

Your are expected to upload the following to Canvas after you have completed the lab:

To receive credit for this lab, you will need to submit your video, code, and system instructions to Canvas by Thursday, September 17, 2026 at 11:59pm.

An Overview of the Starter Code

The starter code contains several files:

Running the Python code

To run the main Python code, you will need to have the necessary dependencies installed and your OpenAI API key configured.

You can then execute the script by running: python llm_based_human_robot_dialogue.py <robot_ip> Replace <robot_ip> with the IP address of your NAO robot.

The program will continue this process until the program is manually stopped in the terminal (e.g., by pressing Ctrl+C).

To see what stage of the dialogue the robot is in, you can check the terminal output. The code will print out the current stage of the dialogue, as well as any transcriptions and generated responses.

Prompt Engineering

The primary focus of this lab will be on prompt engineering. In the three_good_things_system_instruction.txt file, you will find a system instruction that is used to prompt the ChatGPT model to generate text for Nao. Right now, the system instruction guides the behavior of a robot receptionist in the CS department at KU. You will need to modify this system instruction to enable Nao to guide a human participant through the "Three Good Things" exercise.

If you want to test your system prompt independently from the Nao robot, you can do so by running gen_ai_test.py from the starter code in your terminal. This will allow you to communicate with the model only with text, enabling you to develop more quickly.

As a reminder, here is the desired interaction flow for the "Three Good Things" positive psychology exercise:

Robot Expressions

For this lab, you are asked to develop 5 additional custom actions for the robot. To develop these custom actions, we recommend you check out the following resources:

Your new robot expressions should be added to the change_expression definition in the python code file and in the <your_expression> tag within three_good_things_system_instruction.txt. The rest of this section delves into how the robot expressions are executed within the starter code.

How the Robot Expressions Work in the Starter Code

The robot expressions are defined in the change_expression function in the python code file. When the ChatGPT model generates a text response for the robot to speak, it will also generate an action expression for the robot that corresponds with that text (e.g., "hi", "listen"), which is then parsed from the JSON response inside the code.

These expressions can be generated by the ChatGPT model because the list of expressions the robot can execute are provided in the system instruction (three_good_things_system_instruction.txt): 

<your_expression>
Your expression should be one of the ones from this list. 
These expressions can represent how you are feeling or be a reaction to what the student has said.
Please refrain from choosing an expression multiple times in a row: [
'nod',
'hi',
'listen',
'happy'
]
</your_expression>

After the expression is generated by the ChatGPT model, it is looked up in the NaoProxy change_expression definition and executed on the robot in the main pipeline.

Talking Back-and-Forth with Nao: Speech-to-Text, Text Generation, Text-to-Speech

While it is not required to know how the dialogue code works in detail for the purposes of completing this lab, I want to provide a brief overview for those interested in how it enables Nao to have a back-and-forth conversation with a person. This conversation consists of three main steps: speech-to-text, text generation, and text-to-speech.

Speech-to-text: This lab provides an OpenAI implementation for transcribing the human participant's speech to text. It begins by turning Nao's LED green and opening a local microphone stream using Nao's ALAudioRecorder module. For this lab, the human participant will press one of Nao's foot bumpers to stop the recording, which will turn Nao's LED red and close the microphone stream. (It is also possible to implement a volume-based trigger for stopping the recording, but the lab classroom environment can be noisy, so we have opted for the manual foot bumper trigger instead.) The audio is then downloaded from the Nao robot and sent to the local device (your lab desktop) by paramiko. This .wav file is then sent to OpenAI's Whisper API, and the returned transcript is then passed to the text generation component within the code.

Text generation: The transcription from the speech to text section is bundled together with the system instruction and any chat history to form the input for the text generation component. The text generation component uses ChatGPT-5-nano to generate a text response for Nao to speak. Since the generated text is outputted in a JSON format (as instructed in the system instruction text file), it is then parsed and the text and any expressions are extracted for use in the text-to-speech component.

Text-to-Speech: The text generated by the ChatGPT-5-nano model is then converted to speech using OpenAI's text-to-speech API. This conversion occurs inside generate_speech_audio() in the OpenAIHandler class in the starter code and the resulting audio file is then saved, transfered to the robot via paramiko, and then played on the robot.


Instructor AI Usage Acknowledgment: This lab was partially created using Gemini AI and Github Copilot to do the following things:

© 2026 Sarah Sebo