Isaac Sim integration with ROS 2

Introduction to ROS

ROS (Robot Operating System) is an open source software development kit for robotics applications. It is not an OS, but a standard software platform to developers across industries that will carry them from research and prototyping all the way through to deployment and production.

Three components make up most robots:

• Actuators: components responsible for movement.
• Sensors: devices that collect data from the environment.
• Control system: systems that make decisions based on sensor input and desired outcomes.

ROS helps to build these three components and connect them together using topics and messages, which can be recorded for debugging purposes and sent for teleoperation purposes.

ROS potentially works with any component that has a software interface, and can be run in a Docker container. The Python package for ROS is rospy, which can be installed using pip install roslibpy.

ROS Concepts

ROS nodes

Nodes are individual processes that perform computation.

ROS topics

Nodes can publish and subscribe to messages on named buses called topics. They provide a way to exchange information in a decoupled manner. 

ROS messages

Nodes use messages as data structures to communicate. They can contain various types of data, such as integers, floats, strings, and arrays.

Messages are defined using a specific format, often in .msg files, which specify the types and names of the fields. When nodes send and receive messages, they use these predefined structures.

Message structure

ROS message structures play a key role in how nodes exchange data within the Robot Operating System.

Example: A Forklift with an Attached Camera for Pallet Detection

Nodes

• Camera Node: This node interfaces with the camera mounted on the forklift, capturing images.
• Pallet Detection Node: This node subscribes to the camera’s image topic, processes the incoming images to identify pallets, and may publish the results (e.g., pallet locations) to another topic.
• Control Node: This node receives pallet detection information and makes decisions on the forklift’s movements, such as navigating to pick up pallets.
• User Interface Node: This node provides a graphical interface for operators to monitor the forklift’s activities, showing the camera feed and detected pallets.

Topics

• /camera/image: This topic is where the Camera Node publishes raw images captured from the forklift’s camera.
• /detected/pallets: This topic is where the Pallet Detection Node publishes the results of its analysis, such as the locations and dimensions of detected pallets.
• /forklift/control: This topic allows the Control Node to receive commands based on pallet detection and send movement instructions to the forklift.
• /ui/status: This topic is used by the User Interface Node to receive updates, such as “Pallet detected” or “Moving to pallet.”

Messages

• Image Message: The message format for /camera/image might be defined using the standard ROS message type, like sensor_msgs/Image, containing fields such as:
  • width (int): Width of the image.
  • height (int): Height of the image.
  • data (byte array): The pixel data of the image.
• Pallet Detection Message: For /detected/pallets, you might define a custom message type (e.g., my_package/PalletDetection.msg) that includes:
  • pallet_id (string): A unique identifier for each detected pallet.
  • position (geometry_msgs/Pose): The position of the pallet in the camera’s coordinate system.
  • dimensions (float[3]): Width, length, and height of the detected pallet.
  • confidence (float): The confidence score of the detection.

A message example could be:

# RobotStatus.msg
float64 x          # Current X position of the robot
float64 y          # Current Y position of the robot
float64 z          # Current Z position of the robot (e.g., altitude)
float32 battery     # Battery level (0.0 to 100.0)
string status       # Operational status (e.g., "idle", "moving", "charging")

A node responsible for monitoring the robot’s status would populate an instance of this message (put x, y, z, etc in the message) and publish it to /robot/status. Other nodes, like a monitoring system, could subscribe to this topic. They would receive regular updates about the robot’s position, battery level, and operational status.

Data Flow

1. Image Capture:
   • The Camera Node captures images of the warehouse or storage area and publishes them to the /camera/image topic at regular intervals.
2. Pallet Detection:
   • The Pallet Detection Node subscribes to /camera/image, processes the images using computer vision algorithms (e.g., machine learning models trained to recognize pallets), and identifies any pallets in view.
   • When pallets are detected, the node constructs a message with their details (location, dimensions) and publishes it to the /detected/pallets topic.
3. Forklift Control:
   • The Control Node subscribes to /detected/pallets to receive real-time updates about the location of pallets.
   • Based on the detected pallet information, it calculates the forklift’s optimal path to approach the pallets and sends movement commands (like “move forward,” “turn left”) to the /forklift/control topic.
4. User Interface:
   • The User Interface Node subscribes to /camera/image and /detected/pallets, displaying the camera feed on a screen along with overlayed bounding boxes around detected pallets and relevant details (e.g., pallet IDs).
   • It also subscribes to /ui/status to provide feedback to the operator about the current state of the forklift, such as “Pallet detected” or “Navigating to pallet.”

Isaac Sim ROS2 integration

Isaac Sim can connect to ROS2 via ROS2 Bridge Extension. The system will remove ROS1 support in a future release because it has deprecated. Recommended distros are ROS2 Humble.

The ROS2 Bridge Extension enables users to publish and subscribe to several rostopics and rosservices commonly needed for robotic simulation. However, Isaac Sim’s Docker image does not include the pre-installed ROS2 extension, so we must create a custom image with both Isaac Sim and ROS2 installed.

The first step is create a dockerfile and put the following into it:

FROM nvcr.io/nvidia/isaac-sim:4.2.0

# Set the Debian frontend to noninteractive
ENV DEBIAN_FRONTEND=noninteractive

# Preconfigure the timezone
ENV TZ=America/New_York

# Install necessary dependencies
RUN apt-get update && apt-get install -y \
    curl gnupg2 lsb-release tzdata

# Add the ROS 2 GPG key and repository
RUN curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | apt-key add - \
    && echo "deb http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/ros2-latest.list

# Install ROS 2 Humble desktop version
RUN apt-get update && apt-get install -y ros-humble-desktop

# Install teleop_twist_keyboard package
RUN apt-get update && apt-get install -y ros-humble-teleop-twist-keyboard

# Install text editor 
RUN apt-get update && apt-get install -y nano

# Configure the ROS 2 environment
SHELL ["/bin/bash", "-c"]
RUN echo "source /opt/ros/humble/setup.bash" >> /root/.bashrc

# Set necessary environment variables
ENV ROS_VERSION=2
ENV ROS_PYTHON_VERSION=3
ENV ROS_DISTRO=humble

# Reset the Debian frontend
ENV DEBIAN_FRONTEND=dialog

# Set the default entry point
ENTRYPOINT ["/bin/bash"]

This Dockerfile creates a custom Docker image by combining NVIDIA Isaac Sim with ROS 2 Humble, adding essential tools for robotic simulation and development. Below is a detailed explanation of each section and its purpose:

1. Base Image: The image starts from nvcr.io/nvidia/isaac-sim:4.2.0, which provides the necessary environment to run Isaac Sim, a powerful simulation tool for robotics.
2. Non-Interactive Mode: By setting the Debian frontend to noninteractive, the Docker build process avoids prompts during package installations, ensuring smooth automation.
3. Timezone Configuration: The timezone is set to America/New_York to align the system’s time with a specific region, which is helpful for debugging logs and ensuring accurate timestamps.
4. Installing Basic Dependencies: Essential tools like curl, gnupg2, lsb-release, and tzdata are installed to enable repository management, secure downloads, and system configuration, which are prerequisites for installing ROS 2 and other packages.
5. Adding ROS 2 Repository: The official ROS 2 repository is added to the system along with its GPG key, enabling secure and authenticated installation of ROS 2 packages.
6. Installing ROS 2 Humble: The ROS 2 Humble desktop package is installed to provide the framework, tools, and libraries necessary for developing and running robotic applications.
7. Installing teleop_twist_keyboard: This package is added to allow robot control using keyboard input, which is a convenient way to test and debug robot movement without requiring additional hardware.
8. Installation of a Text Editor: Nano, a lightweight text editor, is installed to allow users to edit configuration files or scripts within the container quickly, which is useful for on-the-fly changes during development.
9. Configuring ROS 2 Environment: The container is set up to automatically source the ROS 2 environment setup script (setup.bash) on shell startup, ensuring ROS 2 commands and tools are always available without manual sourcing.
10. Setting Environment Variables: Key environment variables such as ROS version, Python compatibility, and ROS distribution are defined to configure the ROS 2 tools and libraries properly.
11. Resetting Debian Frontend: The Debian frontend is reset to dialog to enable normal interactive behavior if manual package installations or configurations are needed after the container is built.
12. Default Entry Point: The system sets the entry point to /bin/bash, which allows users to interact with the container using a familiar shell environment when it starts.

Building the Docker Image

To create the Docker image using the Dockerfile, run the following command in the terminal:

docker build -t isaac-with-ros2:v1.0 .

This command builds the image and assigns it the name isaac-with-ros2 with the version tag v1.0. The . indicates that the Docker build context is the current directory, where the Dockerfile is located. Users can now use this image to run Isaac Sim with ROS 2 and additional tools configured for robotic development.

The following command is used to run the Docker container created from the image isaac-with-ros2:v1.0 and access the container’s Bash shell. The command ensures the container uses the host’s GPU resources. It also accepts the NVIDIA Isaac Sim End User License Agreement (EULA) and privacy terms. Additionally, it mounts specific directories from the host to the container for persistent caching, logging, and data storage.

It also uses the –network=host option to enable network connectivity between the host and the container. 

docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" -e "PRIVACY_CONSENT=Y" --rm --network=host \
-v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
-v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
-v ~/docker/isaac-sim/documents:/root/Documents:rw \
isaac-with-ros2:v1.0

Once inside the Bash shell of the Docker container, you can start the Isaac Sim application in headless mode by running one of two commands, depending on the desired connection method to the Isaac Sim interface. To connect using the NVIDIA Omniverse Streaming Client, execute the following command (https://docs.omniverse.nvidia.com/streaming-client/latest/user-manual.html):

./runheadless.native.sh

Alternatively, if you want to connect to the Isaac Sim interface via a web browser (e.g., Google Chrome) using WebRTC, run the following command:

./runheadless.webrtc.sh

For the WebRTC connection, after running the command, open your browser and navigate to the following URL:

http://127.0.0.1:8211/streaming/webrtc-demo/?server=127.0.0.1

To demonstrate the connection and functionality between Isaac Sim and ROS 2 using Action Graph, two examples will be executed. The first example involves controlling a differential drive robot using ROS 2 nodes. Keyboard commands control the robot’s movement, demonstrating how Isaac Sim integrates with ROS 2 to respond to real-time inputs. The second example involves controlling a forklift within a factory scene. Unlike the first example, the forklift operates with four wheels, making the steering control more complex. This scenario highlights the versatility and capability of Isaac Sim and ROS 2 to handle different types of robotic systems and control architectures in simulated environments.

Controlling a differential robot

To begin, navigate to Isaac Assets within Isaac Sim. In the search bar, type “turtlebot_tutorial” and double-click on it to load the scene. This will open a preconfigured environment featuring a room, and inside it, you will find a differential drive robot named Turtlebot3_burger. This setup serves as the starting point for demonstrating the interaction and control of a differential robot using ROS 2 nodes.

After loading the scene, we will create the necessary nodes to establish communication between Isaac Sim and ROS 2. To do this, go to the “Create” menu, then select “Visual Scripting”, and finally click on “Action Graph”. This will allow you to build the logic and connections required for the simulation to interact seamlessly with ROS2.

In the Action Graph tab, you need to search for and drag the following nodes into the workspace:

1. On Playback Tick: This node triggers actions on each simulation tick during playback, ensuring the graph executes continuously while the simulation is running.
2. ROS2 Context: Establishes the connection to the ROS 2 ecosystem, enabling Isaac Sim to communicate with ROS 2 nodes.
3. ROS2 Subscribe Twist: Subscribes to ROS 2 Twist messages, which are used to convey velocity commands (linear and angular) for robot movement.
4. Scale To/From Stage Units: Converts the scale of velocity data between ROS 2 and Isaac Sim’s unit system to ensure compatibility.
5. Break 3-Vector (x2): Splits a 3D vector into its individual components (X, Y, and Z), allowing the velocity components to be processed separately.
6. Constant Token (x2): Defines fixed tokens or identifiers for controlling specific aspects of the simulation or robot.
7. Differential Controller: Simulates the control logic for a differential drive robot by mapping linear and angular velocities to the appropriate wheel speeds.
8. Make Array: Combines multiple values into an array format, often used to send a structured set of commands to another node.
9. Articulation Controller: Sends the final control commands to the robot’s joints or wheels, enabling movement within the simulation.

After adding all the nodes, connect them as shown in the provided diagram. These connections define the data flow and logic needed for Isaac Sim. They allow Isaac Sim to interpret and execute commands received from ROS 2. This effectively controls the robot in the simulation.

You must ensure that the “Constant Token” nodes are configured with the names of the robot’s wheels. One node should be set to “wheel_left_joint” and the other to “wheel_right_joint”. These values can be found by clicking on the Prim of the Turtlebot3_burger in the stage, then navigating through its hierarchy to locate the joints for the robot’s wheels. This step is crucial for mapping the control commands correctly to the robot’s wheels, as shown in the accompanying image.

Additionally, you need to configure the “Make Array” node by adding another input in its settings. This will allow you to connect both “Constant Token” nodes to its inputs. Furthermore, the “Articulation Controller” node must also be configured. In its settings, assign the Target Prim, which specifies the robot that will be controlled. The Target Prim should be set to the path corresponding to the Turtlebot3_burger

Ensure that the “Differential Controller” node is configured with the following parameters in its Input settings:

With everything configured, you can now click “Play” to start the scene. The next step is to open a new terminal and access the same Docker container using the following command: docker exec -it isaac-sim bash

Once inside the container’s Bash shell, execute the following command to verify that the ROS 2 nodes are accessible: ros2 topic list

This command lists all active ROS 2 topics available in the current environment. You should see the following output:

/cmd_vel

/parameter_events

/rosout

• /cmd_vel: This is the topic where velocity commands (linear and angular) are published.It controls the movement of the robot, specifically the Turtlebot3_burger in this case.
• /parameter_events: This topic provides information about parameter changes in the ROS 2 nodes, such as updates to configuration values.
• /rosout: This is a logging topic that collects output messages from all active ROS 2 nodes, useful for debugging and monitoring.

Using teleop_twist_keyboard to Control the Robot

The teleop_twist_keyboard command is a ROS 2 node that allows you to control a robot by publishing velocity commands based on keyboard inputs. This node reads keypresses from the keyboard and sends them as Twist or TwistStamped messages to the robot via the /cmd_vel topic. It is an intuitive way to manually control the robot, making it ideal for testing and debugging movement functionality in a simulation.

To launch the node, use the following command:

ros2 run teleop_twist_keyboard teleop_twist_keyboard

This node works best with a US keyboard layout. Below is an explanation of the controls and what each key does:

Keyboard Controls for Movement

Basic Motion Commands

The robot can move in 8 directions and stop, based on the following keys:

• i: Move forward.
• j: Turn left while moving forward/backward.
• l: Turn right while moving forward/backward.
• u: Move forward and slightly left.
• o: Move forward and slightly right.
• m: Move backward and slightly left.
• ,: Move backward.
• .: Move backward and slightly right.
• k: Stop the robot.

Speed Adjustment

To adjust the robot’s speed dynamically:

• q/z: Increase/decrease both linear and angular speeds by 10%.
• w/x: Increase/decrease only the linear speed by 10%.
• e/c: Increase/decrease only the angular speed by 10%.

Controlling a Four-Wheeled Forklift with Ackermann Steering

Now, let’s proceed with a second example, where we will control a robot with four wheels instead of two, as in the previous differential robot example. In this case, it will no longer be possible to use the Differential Controller node. Instead, we will use a node called Ackermann Controller. This node controls vehicles that use Ackermann steering geometry, commonly seen in cars and forklifts. It calculates the appropriate steering angles and wheel velocities to achieve smooth, accurate motion based on linear and angular velocity inputs.

For this example, navigate to the Isaac Assets section and search for a scene named warehouse_with_forklift. Double-click on it to load the scene. The forklift’s steering and movement require more complex logic compared to a two-wheel differential drive robot, showcasing the versatility of Isaac Sim in handling different robotic systems.
With the scene loaded, we will disable the forklift located in the center of the warehouse to make room for another forklift that we will import. This ensures there is enough space in the environment to work with the new forklift and avoid any interference with the existing model during the simulation.

To import the forklift that we will control, go back to Isaac Assets and search for forklift_c. Drag and drop this asset into the warehouse environment, placing it in the desired location within the depot.

With the scene ready, create a new Action Graph and import the following nodes:
On Playback Tick, Isaac Compute Odometry Node, ROS2 Subscribe Twist, Scale To/From Stage Units, two Break 3-Vector, Ackermann Controller, two Make Array, and two Articulation Controller.

Explanation of Key Nodes:

1. Isaac Compute Odometry Node:
   This node calculates the robot’s odometry based on its movements in the simulation. It computes the robot’s position and orientation in the world over time. This information can be used for navigation, mapping, and other ROS 2 tasks. This data is typically published to topics like /odom, providing feedback for ROS 2 nodes about the robot’s current state.
2. Ackermann Controller:
   This node handles the control logic for vehicles with Ackermann steering geometry, such as cars or forklifts. It translates linear and angular velocity inputs into appropriate steering angles and wheel velocities, ensuring smooth and accurate motion. The node accounts for the physical constraints of the vehicle, like the turning radius, to simulate realistic movements. This is particularly important for robots with more complex steering mechanisms than simple differential drive systems.Ensure that the Isaac Compute Odometry Node is configured with the correct chassisPrim path.
   You must also configure the parameters of the Ackermann Controller node as shown in the provided image.
   Finally, you must configure the Articulation Controller nodes by setting the jointNames for the wheels responsible for steering and the wheels responsible for traction. Assign the appropriate joint names for each group of wheels based on the forklift’s structure.With the Action Graph fully configured, the final step is to click Play to start the scene. Once the simulation is running, follow the steps outlined in the Using teleop_twist_keyboard to Control the Robot section. This will allow you to use keyboard commands to send velocity inputs to the forklift. You can control its movement and steering within the warehouse simulation. This demonstrates the successful integration of Isaac Sim with ROS 2. It enables control of a four-wheel robot with a more complex steering mechanism.d

Conclusion

This blog provides a comprehensive overview of ROS concepts, workflows, and its integration with NVIDIA Isaac Sim. Together, they enhance robotics applications. By combining ROS’s modular design with Isaac Sim’s advanced simulation capabilities, developers can create and test robotic systems more effectively. These tools allow for optimization in various workflows, whether for a forklift system or more complex industrial tasks. They pave the way for innovation and efficiency in robotics development.

For similar blogs please visit Marvik Blogs.