Operator Tools¶

Small command-line tools used during day-to-day simulation work.

These are convenience wrappers around repeatable workstation operations. They should not contain secrets, keyring passwords, or one-off experiment notes. Put experiment results in the relevant simulation page and keep this page focused on reusable commands.

Quick Reference¶

Remote Desktop: stream-desktop¶

Use stream-desktop when the workstation desktop needs to be viewed through Moonlight. The script starts GDM/X11, waits for DISPLAY=:0, disables screen blanking, then starts Sunshine so the stream uses NVIDIA capture/encoding correctly.

# Start Moonlight desktop access
stream-desktop start

# Stop Sunshine and the X11 desktop
stream-desktop stop

# Restart the whole desktop streaming stack
stream-desktop restart

# Check GDM, X11, display, and Sunshine ports
stream-desktop status

# Recent Sunshine logs
stream-desktop logs

Keep Sunshine autostart disabled. The normal workstation state is no desktop stream until stream-desktop start is run.

More detail: Sunshine Streaming and GPU & Display Config.

Isaac Playback Videos: isaac-clip¶

isaac-clip renders Isaac Lab policy playback videos from named project profiles. It can archive videos locally or send them through a provider. The current providers are:

The real CLI is installed as isaac-clip. Its source repo is currently zeulewan/isaac-runclip; the local checkout is /home/zeul/GIT/telecli. Project/view behavior should live in isaac-clip config, not in local wrapper scripts.

Project presets live in:

~/.config/isaac-clip/projects.toml

The important attached-wheelchair presets are:

# Current fixed-base relaxed attached standing run
isaac-clip send unitree-wheelchair-fixed-relaxed-stand-attached
isaac-clip send unitree-wheelchair-fixed-relaxed-stand-attached --view two_orbit
isaac-clip views unitree-wheelchair-fixed-relaxed-stand-attached
isaac-clip checkpoints unitree-wheelchair-fixed-relaxed-stand-attached

# Older free-chair relaxed attached standing run
isaac-clip send unitree-wheelchair-relaxed-stand-attached
isaac-clip send unitree-wheelchair-relaxed-stand-attached --view two_orbit
isaac-clip views unitree-wheelchair-relaxed-stand-attached
isaac-clip checkpoints unitree-wheelchair-relaxed-stand-attached

# Older attached-push run
isaac-clip send unitree-wheelchair-attached

# Current 1 m/s small yaw-torque refinement run
isaac-clip send unitree-wheelchair-physx-rail-1mps-fast-lean-small-yaw-torque-hard-attach-push-attached
isaac-clip send unitree-wheelchair-physx-rail-1mps-fast-lean-small-yaw-torque-hard-attach-push-attached --training-policy continue

# Inspect available views/checkpoints
isaac-clip views unitree-wheelchair-attached
isaac-clip checkpoints unitree-wheelchair-attached

# Print the render command without running it
isaac-clip send unitree-wheelchair-relaxed-stand-attached --dry-run

The fixed-base relaxed attached wheelchair views include the diagnostic play_args = ["--show-wheelchair-urdf-proxy"], so those renders show the simplified URDF proxy geometry used for collision/handle inspection.

The current fixed-base relaxed attached wheelchair preset uses provider = "site", site_dir = "logs/demos/latest-site", and site_name = "latest.mp4". That means the normal isaac-clip send ... command updates the latest-video page instead of emailing by default:

https://workstation.tailee9084.ts.net:8002/

Site delivery can also compress the stable page copy before publishing it. Configure the project send block with:

compress_site_video = true
site_crf = 38
site_preset = "veryfast"
site_max_width = 960

This keeps the archived render unchanged under logs/demos/<render>/, but transcodes logs/demos/latest-site/latest.mp4 with ffmpeg so the browser page loads faster. The current wheelchair preview target is roughly 500 KB per 12 s clip; tune site_crf and site_max_width per project if a different balance is needed. The site JSON records the original and compressed byte sizes.

To wait for a future checkpoint and update the page automatically:

isaac-clip watch unitree-wheelchair-fixed-relaxed-stand-attached \
  --target-iteration 12000 \
  --view two_orbit \
  --render

To keep updating the page throughout training, use an interval watcher:

isaac-clip watch unitree-wheelchair-relaxed-push-attached \
  --every-iterations 250 \
  --view two_orbit \
  --render \
  --notify \
  --notify-provider telegram \
  --notify-attach-video

That command starts from the next interval after the current latest checkpoint. For example, if the latest checkpoint is model_12300.pt and the interval is 250, the first render target is model_12500.pt, then model_12750.pt, model_13000.pt, and so on. It updates the latest-video page and sends the rendered MP4 to Telegram after each successful render. Drop --notify-attach-video when only a text/path notification is needed.

The notification uses telecli and the local Telegram config at ~/.config/telecli/config.toml. The preferred project config shape is:

[projects.<name>.send]
provider = "telegram"
telegram_attach_video = false

For site-first projects, keep provider = "site" and add notify_provider = "telegram". Omit --notify when only the website should update.

Flag distinction: --notify-attach-video attaches the MP4 to a Telegram notification after a site render. --telegram-attach-video attaches the MP4 when Telegram is the primary delivery provider, as in isaac-clip send <project> --provider telegram --telegram-attach-video.

The default unitree-wheelchair-attached view currently uses:

The CLI writes both an MP4 and JSON metadata file. The metadata records the project, checkpoint, view settings, and exact render command so videos are reproducible.

The command prints the operational status that would otherwise require side checks: selected checkpoint, run directory, view, output directory, GPU memory/utilization before and after render, detected training process, training-guard decision, render duration, and final output paths.

isaac-clip is not a training manager, but it has a render-time training guard. The default --training-policy auto checks for a running Isaac Lab scripts/rsl_rl/train.py process. If GPU telemetry has enough headroom, training keeps running while the video renders. If free GPU memory or GPU utilization crosses the configured thresholds, the CLI pauses the training process for the render and resumes it afterward.

Useful overrides:

# Always render alongside training
isaac-clip send unitree-wheelchair-attached --training-policy continue

# Always pause training during the render, then resume it
isaac-clip send unitree-wheelchair-attached --training-policy pause

# Refuse to render if training is already running
isaac-clip send unitree-wheelchair-attached --training-policy fail

The pause uses process signals, not checkpointing. It avoids compute contention but does not free training VRAM.

For long wheelchair runs where training should not be interrupted, use --training-policy continue. The current 1 m/s hard-attach continuation has enough GPU headroom for 10-env playback renders while the 2048-env training process keeps running.

Views that use follow_best_robot = true can intentionally change the camera target after a configured delay. If follow_best_after_steps = 80, the camera follows the default env briefly and then jumps to the best-moving env after about 1.6 s at 50 Hz. That can look like the video splits or new instances appear. For stable review clips, set the delay to zero:

isaac-clip send unitree-wheelchair-physx-rail-1mps-fast-lean-hard-attach-push-attached \
  --view slow_revolve_best \
  --training-policy continue \
  --follow-best-after-steps 0

The 1 m/s hard-attach project preset now uses follow_best_after_steps = 0, so the latest-video page's New Video button selects the best env at frame zero instead of switching mid-clip.

Do not document or commit Telegram bot tokens, chat IDs, local email credentials, or keyring material. Keep gog only as a fallback if Telegram is unavailable.

Latest Video Website¶

A small local dashboard serves the newest archived Isaac demo video at a stable tailnet URL:

https://workstation.tailee9084.ts.net:8002/

The server runs from unitree_rl_lab in tmux session latest_video_site:

tmux attach -t latest_video_site

It is backed by:

cd /home/zeul/GIT/unitree_rl_lab
./tools/latest_video_site.py --host 127.0.0.1 --port 8002 --title "Latest Isaac Demo Video"

For the active wheelchair push run, start it with the render button wired to the current isaac-clip project:

cd /home/zeul/GIT/unitree_rl_lab
./tools/latest_video_site.py \
  --host 127.0.0.1 \
  --port 8002 \
  --title "Latest Isaac Demo Video" \
  --render-project unitree-wheelchair-physx-rail-1mps-fast-lean-hard-attach-push-attached \
  --render-view slow_revolve_best \
  --render-training-policy auto

Tailscale Serve exposes that local port:

tailscale serve --https=8002 8002
tailscale serve status

The page scans /home/zeul/GIT/unitree_rl_lab/logs/demos/**/*.mp4 at request time and embeds only the newest MP4. It also shows the video creation time, exposes /latest.mp4 for direct playback/download, supports byte-range requests for browser seeking, and auto-refreshes when a newer archived render appears.

When --render-project is set, the page also shows a New Video button. That button runs isaac-clip send <project> --provider site on the workstation, using the latest checkpoint selected by the project preset. For the current 1 m/s hard-attach run, the site button is wired to unitree-wheelchair-physx-rail-1mps-fast-lean-hard-attach-push-attached and the slow_revolve_best view. The local preset's run_glob includes the current continuation run and follow_best_after_steps = 0, so the button renders the latest continuation checkpoint without a mid-video target jump. The status panel shows whether the render is idle/running/succeeded/failed, elapsed time, exit code, and recent isaac-clip output. The Refresh button reloads the page immediately; the page also polls status and reloads itself after a successful render updates latest.mp4.

--render-training-policy auto stops an active Isaac training process when GPU telemetry says there is not enough free memory for the render. Pausing is not enough for the 12,288-env wheelchair run because paused CUDA processes keep their VRAM allocation. After the render completes, restart training from the latest checkpoint if the overnight run should continue.

To update what the page shows, put the desired MP4 under logs/demos with a newer timestamp than the previous videos. The current diagnostic wheelchair-URDF proxy clip was archived as:

logs/demos/unitree-wheelchair-urdf-proxy_model_11100_slow_revolve_20260516_170039/model_11100_urdf_proxy_slow_revolve.mp4

That clip was rendered with scripts/rsl_rl/play.py --show-wheelchair-urdf-proxy, which swaps the playback wheelchair to the diagnostic proxy-visual URDF:

assets/objects/wheelchair/free3d_active_wheelchair/urdf/active_manual_wheelchair_proxy_visual.urdf

When To Use Which View¶

Use isaac-clip for policy playback videos that need to be shared or logged. Use Moonlight through stream-desktop when the goal is interactive inspection: Isaac Sim GUI, RViz2, rtabmap_viz, or any heavy 3D view that should render locally on the workstation and stream as compressed video.

For Foxglove/Lichtblick over the browser, keep the view lightweight. Avoid heavy accumulated point clouds over the network unless explicitly debugging them.