1. Using Input Devices: Keyboard¶
1.1. Learning Objectives¶
This tutorial introduces how to use a keyboard as human input device (HID) for interacting with a simulation in Omniverse Isaac Sim. After this tutorial, you will know how to map inputs from a keyboard to operate a mobile robot in simulation.
5-10 Minute Tutorial
1.2. Getting Started¶
Prerequisites
Please review the Essential Tutorial series, especially Hello Robot, prior to beginning this tutorial.
This tutorial demonstrates how to drive a NVIDIA Jetbot robot using keyboard input. The Jetbot is an open-source two-wheeled robotic platform designed for makers, students, and enthusiasts. It supports multiple sensors and neural networks in parallel for object recognition, collision avoidance, and more. Let’s learn how to simulate and maneuver the Jetbot in Omniverse Isaac Sim.
Begin by opening the Jetbot Keyboard example: go to the top Menu Bar and Click Isaac Examples > Input > Jetbot Keyboard.
Press the LOAD
button to load a Jetbot into the scene.
A Jetbot robot and the window for the Jetbot Keyboard example extension should now be visible in the workspace. Next, let’s examine the source code for this example.
1.3. Keyboard Input¶
Click the Open Source Code button to launch the source code for editing in Visual Studio Code, and then open the file jetbot_keyboard.py
.
The imports section of the code add the necessarily libraries for loading and controlling a Jetbot.
The carb
library is what provides the interface for keyboard input. The omni.isaac.jetbot
library provides access to Jetbot functions and a differential controller.
1 2 3 4 5 6 7 8 9 | import carb # Used for input handling
import omni.ext
import omni.appwindow # Contains handle to keyboard
import gc
import numpy as np
from omni.isaac.jetbot import Jetbot # Used for Jetbot related functions
from omni.isaac.jetbot.controllers import DifferentialController
from omni.isaac.examples.base_sample import BaseSample
from omni.isaac.core.utils.viewports import set_camera_view
|
The __init__
function initializes the default action commands for each Jetbot wheel.
1 2 3 4 | def __init__(self) -> None:
super().__init__()
self._controller = None
self._command = [0.0, 0.0] # Default Actions (No Motion)
|
The setup_scene
function adds a Jetbot to the World.
1 2 3 4 5 6 7 8 9 10 | def setup_scene(self):
world = self.get_world() # Get the World
self._jetbot = world.scene.add( # Add a Robot to the World Scene
Jetbot(
prim_path="/jetbot",
name="my_jetbot",
position=np.array([0, 0.0, 2.0]),
orientation=np.array([1.0, 0.0, 0.0, 0.0]),
)
)
|
The setup_post_load
function add a Gamepad Event callback and a Physics Step callback.
1 2 3 4 5 6 7 8 9 10 11 | async def setup_post_load(self):
world = self.get_world() # Get the World
self._jetbot = world.scene.get_object("my_jetbot") # Get the Robot from the World
self._controller = DifferentialController(name="simple_control") # Create the Controller
self._input = carb.input.acquire_input_interface() # Grab the Input handle
self._keyboard = self._appwindow.get_keyboard() # Get the Keyboard
self._sub_keyboard = self._input.subscribe_to_keyboard_events( # Subscribe to Keyboard Events
self._keyboard, self._sub_keyboard_event
)
self._world.add_physics_callback("jetbot_step", callback_fn=self._on_sim_step) # Add Callback to Physics Step
await self._world.play_async() # Start the Simulation
|
The setup_pre_reset
and setup_post_reset
functions reset the controller before reset is called on the scene as well as removing and adding the physics callback.
1 2 3 4 5 6 7 8 9 | async def setup_pre_reset(self):
self._controller.reset() # resets the controller
self._world.remove_physics_callback("jetbot_step") # Removes the physics callback
return
async def setup_post_reset(self):
self._world.add_physics_callback("jetbot_step", callback_fn=self._on_sim_step) # Adds the physics callback
await self._world.play_async()
return
|
The Keyboard Event callback _sub_keyboard_event
maps W
, S
, A
, D
to rotate and move the robot.
1 2 3 4 5 6 7 8 9 10 11 12 13 | def _sub_keyboard_event(self, event, *args, **kwargs):
if (event.type == carb.input.KeyboardEventType.KEY_PRESS
or event.type == carb.input.KeyboardEventType.KEY_REPEAT):
if event.input == carb.input.KeyboardInput.W: # Set the Action Command for each Wheel
self._command = [20, 0.0]
if event.input == carb.input.KeyboardInput.S:
self._command = [-20, 0.0]
if event.input == carb.input.KeyboardInput.A:
self._command = [0.0, np.pi / 5]
if event.input == carb.input.KeyboardInput.D:
self._command = [0.0, -np.pi / 5]
if event.type == carb.input.KeyboardEventType.KEY_RELEASE:
self._command = [0.0, 0.0]
|
And finally the Physics Event callback _on_sim_step
applies the Actions to the Jetbot’s differential controller.
1 2 3 4 | def _on_sim_step(self, step): # When Simulating
self._jetbot.apply_wheel_actions( # Send the Actions to the Controller
self._controller.forward(command=self._command)
)
|
And with these few core functions, we can drive the Jetbot around in simulation using the W
, S
, A
, D
keys:
Extra Practice
Use the
+
and-
keys to change the velocity of the robot.
1.4. Summary¶
This tutorial covered the following topics:
Using the
carb
APIUsing Event Callbacks
Adding Robots from a Nucleus Server
1.4.1. Next Steps¶
See the other Input Devices tutorial in our Advanced Tutorials section, Using Input Devices: Gamepad Controller, to learn how to integrate gamepad input into simulations in Omniverse Isaac Sim.
1.4.2. Further Learning¶
For a more in-depth look into the concepts covered in this tutorial, please see the following reference materials:
Carbonite (carb)
See the Carbonite Overview docs for additional usage information.