Exploring NVIDIA Omniverse and Isaac Sim

This blog provides an overview of the NVIDIA Omniverse ecosystem, with a particular emphasis on NVIDIA Isaac Sim.

Introduction

NVIDIA Omniverse™ is a platform of APIs, SDKs, and services. It helps developers use Universal Scene Description (OpenUSD) and NVIDIA RTX™ rendering tools. These tools can be added to current software and simulation processes to create AI systems.

Moreover, NVIDIA Omniverse includes key applications that address various aspects of 3D design, simulation, and collaboration.

It is important to note that Omniverse supports many ways of doing the same thing. For instance, you can generate synthetic data using either the GUI or Python scripts. As a result, it is especially important to keep this in mind when looking for tutorials.

Omniverse Applications

USD Composer

USD Composer is an adaptable RTX-powered app for OpenUSD scene creation and editing. Users can build, light, test, and render large scenes with it. This tool is based on Pixar’s USD. Omniverse USD Composer takes advantage of the advanced workflows of USD like Layers, Variants, and Instancing. As a result, it enables real-time ray and path tracing as well as physics simulation of objects.

USD Connections 

OpenUSD Connections is a collection of importers, exporters, converters, and USD file format plug-ins. These tools enable various 3D applications, products, and file formats to exchange data using the Universal Scene Description (OpenUSD or USD) format. Moreover, it facilitates seamless data sharing across platforms.

Omniverse Nucleus

Omniverse Nucleus serves as the platform’s main server. Specifically, it manages storage, teamwork, and data sharing between users and apps. Acting as a hub for data and assets, it ensures that multiple users can work on the same project in real time.

For instance, users can share entire scenes or individual assets, and changes made by one user are instantly reflected for others. This ensures that all users have access to the latest versions of files and assets, reducing the risk of working with outdated information.

Aditionally, the platform supports various 3D file formats and applications, making it easy to import, export, and manage assets from different software tools. This interoperability helps streamline workflows across different teams.

NVIDIA Isaac

NVIDIA Isaac is a platform designed for robotics, combining hardware and software to enable the development and deployment of intelligent robots.  It includes Isaac ROS and Isaac Sim.

Isaac ROS

This integration makes it easier to develop and use robotic applications. It lets users smoothly simulate and test ROS-based robots. It also supports compatibility between different robotics tools, making it easier to move from simulation to real-world use.

Key features of Isaac ROS include:

• Isaac ROS Visual SLAM: visual simultaneous localization and mapping. It uses one or more stereo cameras and optionally an IMU sensor.
• Isaac ROS nvBlox: uses RGB-D data to create a dense 3D map, including unforeseen obstacles, to generate a temporal costmap for navigation.
• Isaac ROS Pose Estimation: foundation model for 6D pose estimation and tracking.
• Isaac ROS cuMotion: CUDA-accelerated library for solving robot motion planning problems at scale by running multiple trajectory optimizations simultaneously to return the best solution.

Isaac Sim

Isaac Sim is a powerful simulation tool for robotics and AI development. For instance, developers can use it to create realistic 3D environments for modeling robot behavior, testing algorithms, and viewing sensor data.
Additionally, advanced physics, rendering, and AI tools allow users to build detailed and realistic simulations. This capability is especially useful for training machine learning models, validating robotics designs, and testing scenarios without physical prototypes. Moreover, the platform supports simulations of multiple robots or agents interacting in shared environments, enabling studies on teamwork and coordination.

Key features of Isaac Sim include:

• Physics Simulations: simulate realistic physical interactions, including collision detection and friction modeling support.
• Rendering: offers real-time, high-fidelity rendering capabilities using NVIDIA RTX technology. Supports physically-based rendering (PBR) techniques, enabling photorealistic visuals with accurate lighting, shadows, and reflections.
• Environment creation: create 3D environments with obstacles and interactive elements. It comes with pre-built assets (robots, sensors, objects).
• Sensor Simulation: simulates a range of sensors, including cameras, LiDAR, IMUs, and more.

Installing Isaac Sim

There are two methods to install Isaac Sim:

• Container Installation: recommended for remote headless servers of the Cloud, using a Docker container.

• Workstation Installation: alternatively, it can be installed from Omniverse Launcher and is recommended for Workstation users.

Launching Isaac Sim from Omniverse Launcher

From Omniverse Launcher launch Isaac Sim. This opens the Isaac Sim App Selector, which allows to start one of three Isaac Sim modes.

1. Isaac Sim: opens the GUI.
2. Isaac Sim (Headless Native): for livestreaming via Omniverse Streaming Client.
3. Isaac Sim (Headless WebRTC): to use Isaac Sim with WebRTC Browser Client.

Isaac Sim Workflows

There are three main workflows when developing in Isaac Sim: GUI, extensions, and standalone Python. Here is a quick summary of the key features and their recommended usages:

Isaac Sim via GUI Interface

Here we outline the UI elements that are unique to Isaac Sim.

1. Create > Isaac – Opens the Isaac Create Menu
2. Utilities – Opens the Isaac Utilities Menu
3. Examples – Opens the Isaac Examples Menu
4. Replicator – Opens the Replicator Menu
5. Isaac Assets Panel – Quick Access to all Isaac Sim library assets

Viewport

The viewport displays real-time, high-quality renderings of 3D scenes. Specifically, it uses advanced tools like NVIDIA RTX for ray tracing.

In the viewport, users can move through the simulation, change camera angles, zoom in and out, and adjust objects. It allows for detailed inspection of simulations with accurate lighting, shadows, and reflections, enabling users to analyze and refine their robotic models and environments effectively.

1. Viewport settings (eg. navigation settings, grid settings)
2. Render mode (select renderer and render settings)
3. Interface visibility 
4. Camera: allows you to select a camera from a preconfigured perspective (e.g top, front, right) or a camera prim. Extra camera controls include lens, zoom, focal distance, exposure and ISO settings. 

Isaac Sim via Extension

Extensions are individually built application modules. The main feature of this workflow is that the application runs asynchronously. This enables the extension applications to interact with the USD stage without blocking rendering and physics stepping. It also allows for hot reloading, so you can change the application code while Isaac Sim is running and then see the reflected changes in your application after saving the file, without shutting down or restarting Isaac Sim.

To get started using this workflow, you have to use the Isaac Sim Extension Template Generator to populate a UI-based extension on a local machine. The template gives a starting point for building a custom UI tool. For more information, go to the docs.

Isaac Sim via Jupyter Notebook Extension

With Isaac Sim open, navigate to the “Window” tab and select “Extensions.” Once there, use the search bar to type “notebook” and click on the extension called “Jupyter Notebook Integration“. Activate this extension, and as an optional (but recommended) step, check the “Autoload” option to ensure the extension is always active when you run Isaac Sim.

With the extension active, go to the desktop browser on the EC2 instance where Isaac Sim is installed and enter the following address to access Jupyter Lab: http://127.0.0.1:8228/lab. If you want to access Jupyter Lab remotely, use the public IP address of the EC2 instance along with port 8228, like this: http://<PUBLIC_IP>:8228/lab.

Isaac Sim via Standalone Python

It can be run in standalone Python mode, which is ideal for customizing simulations directly through Python scripts. This setup allows developers to control the simulation environment, interact with objects in the scene, and configure physics and rendering options with greater flexibility. Running it this way is especially useful for automated testing, prototyping, and real-time simulations where minimal graphical overhead is preferred.

Setting Up and Running Isaac Sim Docker Container

To get started, launch the Docker Container with the following command, ensuring all necessary configurations and mount points are set up:

docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
    -e "PRIVACY_CONSENT=Y" \
    -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
    -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
    -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
    -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
    -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
    -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
    -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
    -v ~/docker/isaac-sim/documents:/root/Documents:rw \
    nvcr.io/nvidia/isaac-sim:4.2.0

Once inside the container, you can explore available directories by running ls. Key folders include:

• standalone_examples: Contains sample scripts demonstrating various Isaac Sim features.
• python.sh: Used to execute Python scripts with the necessary environment variables for Isaac Sim.

Key Standalone Python Examples

Below are some useful standalone Python scripts to get started with Isaac Sim.

Time Stepping
This example demonstrates how to start the Omniverse Kit Python app and create callbacks for each rendering frame and physics timestep. It also illustrates different methods to control physics and rendering steps.

./python.sh standalone_examples/api/omni.isaac.core/time_stepping.py

Load USD Stage
This example loads a USD stage and initiates a simulation. Specify the usd_path to your desired location on the Nucleus server.

./python.sh standalone_examples/api/omni.isaac.kit/load_stage.py --usd_path /Isaac/Environments/Simple_Room/simple_room.usd

Livestream
This example enables livestreaming while running Isaac Sim in native Python mode, compatible with the Omniverse Streaming Client backend. Review the livestream documentation for more details on setting up the client.

./python.sh standalone_examples/api/omni.isaac.kit/livestream.py

Omniverse Extensions

Omniverse Extensions are the core building blocks of Omniverse Kit-based Apps. An extension is a uniquely named and versioned package loaded at runtime. Extensions are plugins that can be used to extend the functionality of existing apps.

To install/enable/disable an extension in Isaac Sim, go to Window>Extensions.

Extension Categories

Here are some key categories of extensions, but there are more in the Omniverse Extension documentation. Replicator Extension deserves a separated section.

Materials

This section houses Extensions that allow you to create, import/export and manage materials in Omniverse.

OmniGraph

OmniGraph is an engine for programmatically manipulating scenes with a graph-based visual scripting interface. There are two types of graph types you can create in OmniGraph: Action Graphs, which allow event driven behaviors and Push Graphs which evaluate nodes continuously.

An Action Graph is an OmniGraph that triggers actions in response to a particular event.

Omniverse Replicator Extension

The Omniverse Replicator (omni.replicator) lets developers build custom synthetic data generation (SDG) tools and pipelines. A more detailed description in the following section.

Core components

Replicator is composed of six components that enable synthetic data generation. Throughout generation of a dataset the most common workflow is to randomize a scene, select your annotators, and then write to your desired format. However, if needed for more customization you have access to omni.synthetic data.

Semantics Schema Editor 

Semantic annotations (data “of interest” pertaining to a given mesh) are required to properly use the synthetic data extension. These annotations inform the extension about what objects in the scene need bounding boxes, pose estimations, etc… The Semantics Schema Editor provides a way to apply these annotations to prims on the stage through a UI.

Visualizer

The Replicator visualizer enables you to visualize the semantic labels for 2D/3D bounding boxes, normals, depth and more.

Randomizers

Replicator’s randomization tools allow to create domain randomized scenes, sampling from assets, materials, lighting and camera positions.

Omni.syntheticdata

Lowest level component of Replicator. Provides low level integration with the RTX renderer and OmniGraph.

Annotators

The annotation system itself ingests the AOVs and other output from the omni.syntheticdata extension to produce precisely labeled annotations for deep neural network training.

Writers

Writers process the images and other annotations from the annotators, and produce specific data formats for training. They can output to local storage and to cloud based storage backends.

Conclusion