1. Using Input Devices: Keyboard

1.1. Learning Objectives

This tutorial introduces how to use a keyboard as human input device (HID) for interacting with a simulation in Omniverse Isaac Sim. After this tutorial, you will know how to map inputs from a keyboard to operate a mobile robot in simulation.

5-10 Minute Tutorial

1.2. Getting Started

Prerequisites

  • Please review the Essential Tutorial series, especially Hello Robot, prior to beginning this tutorial.

This tutorial demonstrates how to drive a NVIDIA Jetbot robot using keyboard input. The Jetbot is an open-source two-wheeled robotic platform designed for makers, students, and enthusiasts. It supports multiple sensors and neural networks in parallel for object recognition, collision avoidance, and more. Let’s learn how to simulate and maneuver the Jetbot in Omniverse Isaac Sim.

Begin by opening the Jetbot Keyboard example: go to the top Menu Bar and Click Isaac Examples > Input > Jetbot Keyboard. Press the LOAD button to load a Jetbot into the scene.

../_images/isaac_tutorial_advanced_keyboard_control_0.png

A Jetbot robot and the window for the Jetbot Keyboard example extension should now be visible in the workspace. Next, let’s examine the source code for this example.

1.3. Keyboard Input

Click the Open Source Code button to launch the source code for editing in Visual Studio Code, and then open the file jetbot_keyboard.py.

The imports section of the code add the necessarily libraries for loading and controlling a Jetbot. The carb library is what provides the interface for keyboard input. The omni.isaac.jetbot library provides access to Jetbot functions and a differential controller.

1
2
3
4
5
6
7
8
9
import carb                                                                     # Used for input handling
import omni.ext
import omni.appwindow                                                           # Contains handle to keyboard
import gc
import numpy as np
from omni.isaac.jetbot import Jetbot                                            # Used for Jetbot related functions
from omni.isaac.jetbot.controllers import DifferentialController
from omni.isaac.examples.base_sample import BaseSample
from omni.isaac.core.utils.viewports import set_camera_view

The __init__ function initializes the default action commands for each Jetbot wheel.

1
2
3
4
    def __init__(self) -> None:
        super().__init__()
        self._controller = None
        self._command = [0.0, 0.0]                                              # Default Actions (No Motion)

The setup_scene function adds a Jetbot to the World.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def setup_scene(self):
    world = self.get_world()                                                    # Get the World
    self._jetbot = world.scene.add(                                             # Add a Robot to the World Scene
        Jetbot(
            prim_path="/jetbot",
            name="my_jetbot",
            position=np.array([0, 0.0, 2.0]),
            orientation=np.array([1.0, 0.0, 0.0, 0.0]),
        )
    )

The setup_post_load function add a Gamepad Event callback and a Physics Step callback.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
async def setup_post_load(self):
    world = self.get_world()                                                        # Get the World
    self._jetbot = world.scene.get_object("my_jetbot")                              # Get the Robot from the World
    self._controller = DifferentialController(name="simple_control")                # Create the Controller
    self._input = carb.input.acquire_input_interface()                              # Grab the Input handle
    self._keyboard = self._appwindow.get_keyboard()                                 # Get the Keyboard
    self._sub_keyboard = self._input.subscribe_to_keyboard_events(                  # Subscribe to Keyboard Events
        self._keyboard, self._sub_keyboard_event
        )
    self._world.add_physics_callback("jetbot_step", callback_fn=self._on_sim_step)  # Add Callback to Physics Step
    await self._world.play_async()                                                  # Start the Simulation

The setup_pre_reset and setup_post_reset functions reset the controller before reset is called on the scene as well as removing and adding the physics callback.

1
2
3
4
5
6
7
8
9
async def setup_pre_reset(self):
    self._controller.reset()  # resets the controller
    self._world.remove_physics_callback("jetbot_step") # Removes the physics callback
    return

async def setup_post_reset(self):
    self._world.add_physics_callback("jetbot_step", callback_fn=self._on_sim_step) # Adds the physics callback
    await self._world.play_async()
    return

The Keyboard Event callback _sub_keyboard_event maps W, S, A, D to rotate and move the robot.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
def _sub_keyboard_event(self, event, *args, **kwargs):
    if (event.type == carb.input.KeyboardEventType.KEY_PRESS
        or event.type == carb.input.KeyboardEventType.KEY_REPEAT):
        if event.input == carb.input.KeyboardInput.W:                               # Set the Action Command for each Wheel
            self._command = [20, 0.0]
        if event.input == carb.input.KeyboardInput.S:
            self._command = [-20, 0.0]
        if event.input == carb.input.KeyboardInput.A:
            self._command = [0.0, np.pi / 5]
        if event.input == carb.input.KeyboardInput.D:
            self._command = [0.0, -np.pi / 5]
    if event.type == carb.input.KeyboardEventType.KEY_RELEASE:
        self._command = [0.0, 0.0]

And finally the Physics Event callback _on_sim_step applies the Actions to the Jetbot’s differential controller.

1
2
3
4
def _on_sim_step(self, step):                                                       # When Simulating
    self._jetbot.apply_wheel_actions(                                               # Send the Actions to the Controller
        self._controller.forward(command=self._command)
        )

And with these few core functions, we can drive the Jetbot around in simulation using the W, S, A, D keys:

../_images/isaac_tutorial_advanced_keyboard_control_1.gif

Extra Practice

  • Use the + and - keys to change the velocity of the robot.

1.4. Summary

This tutorial covered the following topics:

  1. Using the carb API

  2. Using Event Callbacks

  3. Adding Robots from a Nucleus Server

1.4.1. Next Steps

See the other Input Devices tutorial in our Advanced Tutorials section, Using Input Devices: Gamepad Controller, to learn how to integrate gamepad input into simulations in Omniverse Isaac Sim.

1.4.2. Further Learning

For a more in-depth look into the concepts covered in this tutorial, please see the following reference materials:

Carbonite (carb)