Isaac Sim robot workbench

dex-isaac-mcp

*Recorded headless with the server's own tools (sim_frame_robot, sim_set_backdrop, sim_record_start), driving the Allegro hand example. Each caption is the request and the tool call it became.*

An MCP server that lets an AI agent (Claude Code, or any MCP client) drive a live, persistent Isaac Sim session and launch Isaac Lab training runs.

Kit takes tens of seconds to boot. If every experiment is a fresh launch, most of your time goes to waiting. Here Kit starts once inside a daemon and stays up. Each tool call lands in that running session, so changing a gain, stepping physics or taking a screenshot costs a frame, not a relaunch.

 MCP client ──stdio──▶ dex_isaac_mcp (host, plain python)

│ newline-delimited JSON over a Unix socket ▼ scripts/simd.py (Isaac Lab container, Kit stays up) └─ one articulation, described by a robot JSON

• Any articulated robot. Point it at a USD and a small JSON config. Franka and Allegro examples are included.
• Normalized joint control. Targets are 0..1, where 0 is a joint's lower limit and 1 its upper limit, so agents don't need to know radians or meters.
• Named poses per robot (home, fist, …), which can be blended part-way.
• Headless capture and GIF recording from an auto-framed camera, with captions. The clip above was made this way.
• Props: spawn a table, a ball or a USD into the running scene and read back where they settle.
• Measurements: joint state, per-joint travel and tracking error, a range test that finds blocked joints, and parameter sweeps inside one session.
• Training control: each run is a detached docker compose run. You can poll its status, TensorBoard scalars, checkpoints and logs.

How it differs from other Isaac Sim MCP servers

NVIDIA's official Isaac Sim MCP is a documentation search for coding assistants, and works well alongside this one. Servers like isaacsim-mcp-server and omni-mcp/isaac-sim-mcp run inside the Isaac Sim GUI and cover scene building broadly. This one runs headless, as a daemon in Docker, and focuses on tuning, measuring and training one robot.

Pick an in-GUI server to build a scene by talking to it, interactively, with a wide robot and asset library.

Pick this one to have an agent tune, debug and train a specific robot (your own, from a USD), unattended or on a remote box, with numbers it can act on instead of only screenshots.

Requirements

• Linux with an NVIDIA GPU that Isaac Sim supports, plus the NVIDIA Container Toolkit
• Docker with the Compose plugin
• An NGC login to pull the Isaac Lab image: docker login nvcr.io (user $oauthtoken, password: your NGC API key)
• Python ≥ 3.10 on the host, for the MCP server only

Quick start

git clone <this repo> dex-isaac-mcp && cd dex-isaac-mcp

1. Build the image (Isaac Lab 2.3.2 base, pinned by digest)

cd docker && docker compose build && cd ..

2. Install the host-side server, editable so it finds this clone

pip install -e . # add [metrics] for train_metrics: pip install -e '.[metrics]'

3. Register it with Claude Code

claude mcp add isaac -- python -m dex_isaac_mcp

4. Allow the container to open windows (once per login; needed for screenshots)

xhost +local:docker

Then ask the agent something like *"start the sim with the Franka, move it to ready and show me a screenshot."* It will call sim_up, sim_set_pose, sim_step and sim_screenshot.

From PyPI instead: pip install dex-isaac-mcp. The daemon still runs from a clone (it needs docker/ and scripts/simd.py), so point the server at it:

claude mcp add isaac -e ISAAC_MCP_HOME=/abs/path/dex-isaac-mcp -- dex-isaac-mcp

Other MCP clients can launch python -m dex_isaac_mcp (or the dex-isaac-mcp script) over stdio.

The first sim_up takes several minutes: Kit builds its shader cache and downloads Nucleus assets. Later starts are much faster.

Running the daemon by hand

sim_up deletes its container (--rm) when the daemon exits, so a crash on startup takes its traceback with it. To see the error, run the daemon in the foreground:

cd docker

docker compose run --rm simd scripts/simd.py --gui --robot examples/robots/franka.json docker compose run --rm simd scripts/simd.py --headless --usd /path/in/container/robot.usd docker compose run --rm simd scripts/simd.py --sliders --robot examples/robots/allegro_hand.json

--sliders opens an omni.ui panel with one slider per driven joint. The socket stays live alongside it.

Tools

Session

Inspect

Drive

Scene

Capture and record

These work headless, with no GUI or viewport. They use a dedicated camera that is independent of the GUI view.

Tune

Train

Training runs do not depend on the daemon or on the MCP session. They keep going after the client disconnects.

Robot config

A robot is one JSON file. Only usd is required. Unknown keys are rejected, so a typo fails loudly instead of quietly falling back to a default.

{

"name": "franka", "usd": "{ISAACLAB_NUCLEUS_DIR}/Robots/FrankaEmika/panda_instanceable.usd", "fix_root_link": true, "spawn_pos": [0, 0, 0], "init_joint_pos": {"panda_joint4": -2.81, "panda_joint6": 3.04, ".*": 0.0}, "driven_joints": ["panda_joint[1-7]", "panda_finger_joint.*"], "passive_joints": [], "couplings": [{"leader": "joint_a", "follower": "joint_b", "ratio": 1.0}], "actuator": {"stiffness": 400, "damping": 40, "effort": 87, "velocity": null}, "solver": {"pos_iters": 32, "vel_iters": 4, "self_collisions": true}, "camera": {"eye": [1.8, 1.8, 1.4], "target": [0, 0, 0.4]}, "poses": {"ready": {"panda_joint[1357]": 0.5, "panda_finger_joint.*": 1.0}} }

| couplings | Software mimic joints: follower target = ratio × leader target. They are