i-spy

Play the I Spy game with the Kinova Gen3 Robot!

Main Software Requirements:

• Ubuntu 18.04.6
• ROS Melodic
  • rospy
  • roslaunch
  • smach
  • actionlib
  • sensor_msgs
  • cv_bridge
• darknet_ros
• ros_kortex_vision
• OpenCV
• opencv-python v4.2.0.32
• SpeechRecognition
  • PyAudio
• pyttsx3
  • espeak

Build

$ cd ~/catkin_ws/src
$ git clone https://github.com/soroush-msd/i-spy.git
$ cd ~/catkin_ws/
$ catkin_make

Run

In one terminal:

$ roscore

And in another terminal:

$ cd ~/catkin_ws/
$ source devel/setup.bash
$ rosrun i-spy stateMachine.py [f]emale/[m]ale [h]uman/[r]obot/[g]reeting

How To Play?

This is a voice-based interaction. You need to have three components connected to your system:

• A microphone through which you will speak to the robot.
• A speaker through which the robot will speak to you.
• The Kinova Gen3 Robotic Arm with a working camera to enable the robot to see and detect objects in the environment.

There are two/three-ish different scenarios you can play the game:

1. The robot looks around, chooses one random object, picks its first letter and then you will need to guess the matching object.
2. You look around, choose one random object within the robot's FOV and pick its first letter. The robot then asks you for the letter and tries to guess the matching object.
3. The robot greets you, introduces itself, and then starts scenario 1.

So, for example, if you run:

$ rosrun i-spy stateMachine.py f r

The robot will play scenario 1 with the female voice.

Note: f stands for female, and r stands for robot in the command line arguments. This means the robot will have a female voice and will be providing a letter for the user to guess.

Or, if you run:

$ rosrun i-spy stateMachine.py m h

The robot will play scenario 2 with the male voice.

Note: m stands for male, and h stands for human in the command line arguments. This means the robot will have a female voice, and you will be providing a letter for the robot to guess.

Similarly, if you run:

$ rosrun i-spy stateMachine.py m g

The robot will play scenario 3 with the male voice.

Note: g stands for greeting in the command line arguments. Refer to scenario 3 for the explanation.

Important Notes:

⚠️ Your microphone needs to remain mute except when you are required to speak to the robot. When ready to speak:

1. Unmute the microphone,
2. Provide input,
3. Mute back the microphone.

This might be due to the sensitivity of different microphones and the variety of energy level thresholds needed to adjust to ambient noise.


⚠️ When the robot asks you about the first letter of the object, you need to provide input by speaking in the following format:

• letter x, for example, if the object starts with x. Or
• letter y, for example, if the object starts with y.

State Machine Interactions

The diagram below demonstrates the interactions and transitions between state machines in different stages during the game:

Demo Videos

The videos below demonstrate the interactions between a human participant and the robot playing the game in different scenarios.

Scenario 3 with female voice
Related command:

$ rosrun i-spy stateMachine.py f g

1.mp4

Scenario 2 with female voice
Related command:

$ rosrun i-spy stateMachine.py f h

2.mp4

Scenario 1 with male voice
Related command:

$ rosrun i-spy stateMachine.py m r

3.mp4

Scenario 2 with male voice
Related command:

$ rosrun i-spy stateMachine.py m h

4.mp4

📚 More Info

For more information regarding this project and the whys and hows: CLICK ME