Getting Started with Furhat

Supported platforms

What is described below should run on Windows, Linux and Mac. I have tested it in the WASP VM but had problems which I have not yet quite come to grips with. The Virtual Furhat window described below was immediately closed. You are therefore advised to run this natively on your machine.

NOTE: Since this tutorial was written there has been a new release of the SDK. If you want to be able to run on the physical Furhat platform, use the new SDK. There are some differences which makes what is described below not work exactly the same. I encourage you to share experiences via the forum or by adding material to the Own contribution page.

Furhat SDK

You should have been sent an email with an invitation from Furhat Developer Team <info@furhat.io> with the subject "Invitation to the Furhat SDK". If you did not get this email, first look in your junk folder and similar and then ask patric@kth.se for help. Use the link in the email to sign up to download the Furhat SDK version 0.35. Move the SDK to a suitable place and unpack it. In the WASP VM/disk environment this could be

cd ~/software
unzip ~/Downloads/Furhat-SDK.zip

Installing the Community edition of IntelliJ IDEA

The skills for Furhat will be developed in IntelliJ IDEA. Download the community edition from https://www.jetbrains.com/idea/download Links to an external site.. You do not need to subscribe, add any email address or anything. IntelliJ IDEA takes about 1GB of disk space and runs on Windows, Mac and Linux.

Install and start IntelliJ IDEA

Linux
Unpack the archive file. Assuming you have downloaded ideaIU-2018.2.4.tar.gz and it is Downloads you would do, for example,
```
cd ~/software
tar xzf ~/Downloads/ideaIU-2018.2.4.tar.gz
cd ideaIU/bin/
./idea.sh
```
Do not import settings
Accept the conditions
In the window that asks for setup, press "Skip remaining and use defaults"
Activate a 30 day evaluation trial
Mac
Double click on the dmg file and drag IDEA into the Application folder
The first time you start the program you need to click a few buttons to get the actual program to start.
Do not import settings
Windows
Run the executable installer file and follow the instructions. I associated the .kt files with it and installed JetBeans during the installation process.
Do not import settings
Accept the conditions
In the window that asks for setup, press "Skip remaining and use defaults"

When IntelliJ IDEA has started the first time you should see a screen like this

Import Furhat-SDK into IntelliJ IDEA

Select "Import project" in the window shown above, navigate to the Furhat-SDK folder and hit Open. Then select Gradle as at the type of project to import as shown below and press Next. In the following screen push Finish and the Furhat-SDK will begin to build.

For some, "gradle jvm" is not set. In that case, you need to tell the system where java is installed on your machine (or install it first if you do not have java installed). You see an example of this below (search for jvm in this document).

You can monitor the build process by opening the build window by clicking on "Build" in the lower left corner.

The build process will take several minutes and includes downloading a number of packages so make sure to have network access.

Run Virtual Furhat Deamon

Open a terminal and navigate to the Furhat-SDK folder and run the Deamon

cd ~/software/Furhat-SDK
./gradlew run

On Windows you would execute .\gradlew run instead.

This should open up the Virtual Furhat window.

The Connect button in the upper right corner never toggles state.

You can use the icon in the upper left corner to change from where you are looking at Furhat.

Open a browser window and direct the browser to http://localhost:8080 and log in with the password admin (it can be hard to see what is written in the little orange window in the middle but that is where you need to enter the password) which should give you a window like below.

Test that your speaker and microphone is working. You can see the microphone response with the blue bar under the Gaze window. You can test the speakers by pressing Play next to the sentence under the Gaze window. Notice how the Virtual Furhat moves its lips with the spoken text.

Press in the Gaze window and you should see how the head moves in that direction.

Test the Gesture bottoms.

Furhat_gestures.mp4 Download Furhat_gestures.mp4 Play media comment.

Speech recognition

You do not need to use speech recognition for the assignment. You can type in the word, but it is more fun to use your voice. If you do not want to use speech, skip this step.

Google Speech API

NOTE: If you are part of the WASP course we will send you a json file that you can use directly without having to set things up like described below.

Go to https://cloud.google.com/speech-to-text/docs/quickstart-gcloud. Links to an external site.

Links to an external site.Press the blue button "SET UP A PROJECT" under item 1

Select a name for the project such as "WASP Furhat"

Select the role for the service account as "Project viewer"

Set up the billing information. You have 300 USD free use of the API. The cost should be 0.006 USD/15 seconds speech. You need to provide a credit card to prove that you are not a robot. Google says that you will not be charged before agreeing to change to a paid account.

Download the json file, open it and paste the content into the Speech Recognition Settings under the Settings tab in the Furhat Studio (localhost:8080 in browser).

Make sure Google is selected and press Save at the bottom of the settings window.

Restart the Deamon. Press ctrl-c in the terminal where you ran gradlew and then restart with ./gradlew run

You should not see a printout like this now

00:40:07 [ForkJoinPool.commonPool-worker-1: 59915] WARN  GoogleRecognizerProcessor:162 - Unable to start the Google recognizer, as no credentials could be found at /Users/patric/code/Furhat-SDK/properties/google_credentials.json

if things went well.

Creating a first skill

Go to the Furhat Studio browser tab

Click the Skills icon on the Left

Select a name like "TestSkill" and press Play to start the skill as shown below.

This will first build the skill and then run it.

To test it go to the Dashboard where you can add users to interact with Furhat. Note that the skill that is now running is shown at the top in the green area.

Double-click somewhere in the dark grey area and a person will appear. The virtual Furhat will look at the person and start the interaction. Either speak with Furhat or type in your replies in the bottom right text area. This skill template only accepts Yes and No as answers.

NOTE: You burn the credits quite quickly so I suggest that you test the speech recognition and then turn off the microphone to save credits.

Furhat_TestSkill.mp4 Download Furhat_TestSkill.mp4 Play media comment.

Configure IntelliJ IDEA

Make sure the Kotlin JVM version is used is 1.8 (File > Settings > Build, Execution, Deployment > Compiler > Kotlin Compiler and set to 1.8). This is shown in the image below. On the Mac this is reached via Preferences under the IntelliJ IDEA menu item.

Set the default Working directory for Kotlin run configurations (Run -> Edit configurations -> Defaults -> Kotlin and set to $MODULE_DIR$ )

Open TestSkill source in IntelliJ IDEA and Run

Open the Furhat-SDK in IntelliJ IDEA

Select File -> New -> Project from Existing Sources...

Navigate to ~/software/Furhat-SDK/skills/TestSkill and press Open

Import it as a Gradle project and press Next

IntelliJ import.png

You should now have settings similar to what is shown below. Press Finish. On my Windows machine I had to tell IntelliJ IDEA where the Java folder was (the entry for Gradle JVM).

Open it as a New Window

On the left upper side you will see the source code. Navigate to the file main.kt found in TestSkill -> src -> main -> kotlin -> furhatos.app.testskill

Double-click on it to open it.

Place the cursor at fun main and you should see a Play icon appear to the left. Press it to run the skill. It might take some time to build.

You should now be able to interact with the robot like you did before using the Dashboard in the Furhat Studio browser window.

Look at the files general.kt and interaction.kt that you find under TestSkill -> src -> main -> kotlin -> furhatos.app.testskill -> flow

Modify the skill to say something different when a person enters.

Testing additional skills

As described at the end of this page https://docs.furhat.io/skills Links to an external site., there are some pre-made skills to test. Download them with.

cd ~/software/Furhat-SDK/skills
git clone https://github.com/FurhatRobotics/example-skills.git

Now test the skills as you did above for your TestSkill. Ignore messages about adding files to git. Below is an example of running the PizzaOrder skill

Futhat_PizzaOrder.mp4 Download Futhat_PizzaOrder.mp4 Play media comment.