Building a smart Robot AI using Hugging Face 🤗 and Unity

A Robot obeying your orders

  • Unity Game Engine (2020.3.18.f1 and +).
  • The Jammo Robot asset, made by Mix and Jam.
  • Hugging Face 🤗

The power of Sentence Similarity 🤖

Before diving into the implementation, we need to understand how does the project work and what sentence similarity is.

How does the project work?

With this project, we want to give more liberty to the player. Instead of giving an order to a robot by just clicking a button, we want him to interact with it through text.

What’s Sentence Similarity?

Sentence Similarity is a language model able, given a source sentence and sentences, to calculate how much similar sentences are to the source sentence.

Step 1: Select the Sentence Similarity Model

Getting Started with HuggingFace 🤗

HuggingFace contains a lot of amazing language models and an API (Accelerated Inference API) to directly plug them on your projects.

Accelerated Inference API 🚀

Now that we have the API key, we can choose and try our model.

Step 2: Connect HuggingFace API 🤗 to Unity 🔗

Now, we need to connect the Hugging Face model API to Unity to be able to use it.

HFScore(): call the API

To call the API and handle the result, we use a Coroutine function since this type of function can wait for execution, and we want to wait for that API to return a response before continuing the execution.

  • First, we form the JSON for the POST Request, it looks like this:
  • Then we make the web request and return the response.
  • The response is a string that looks like this “[0.7777, 0.19, 0.01]”, to work with it, we need to transform it to an array of floats. That is what ProcessResult(data) is doing.

ProcessResult(): convert result to an array of floats and find the max score and max score index.

Because the API returns a string “[0.77, 0.18, 0.01,…]” we can’t work with it since we need to find the highest score and its index.

Fill the inspector

The last step before working on our robot behaviors is to fill your API key and model URL to the inspector.

Step 3: Build the Robot Behavior 🤖

Now that we’ve connected our Unity Project to the Hugging Face Model API, we need to define the behavior of our robot.

Let’s define the State Machine

In a state machine, each state represents a behavior, for instance, move to a column, saying hello, etc. Based on the state the agent is it will perform a series of actions.

Let’s define the Utility Function

Our action list looks like this:

  • Sentence is what will be fed to the API
  • Verb is the State
  • Noun (if any) is the object to interact with (Pillar, Cube, etc)

Step 4: Let’s interact with our Robot 🤖

In this step, you just need to click on the play button in the editor. And you can prompt some orders and see the results.

What’s Next? 🧭

How can I add more actions?

Let’s take an example:

  • Copy YellowPillar gameobject and move it
  • Change the name to GreenPillar
  • Create a new material and set it to green (RGB:)
  • Place the material on GreenPillar
  • Add Go to green column
  • GoTo
  • GreenColumn



Developer Advocate 🥑 at Hugging Face 🤗| Founder Deep Reinforcement Learning class 📚 |

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Thomas Simonini

Developer Advocate 🥑 at Hugging Face 🤗| Founder Deep Reinforcement Learning class 📚 |