Add NuRec Real2Sim Isaac Lab 3 support - #23
Conversation
e0a5f0b to
2da603b
Compare
Add configurable Isaac Lab Dockerfile support, NuRec asset downloads, Kit PPISP viewport binding/debug captures, and smoke configs for viewport validation.
efb0127 to
fa63f3a
Compare
| stage = omni.usd.get_context().get_stage() | ||
| camera_prim = stage.GetPrimAtPath(str(viewport.camera_path)) if stage else None | ||
| if camera_prim and camera_prim.IsValid(): | ||
| transform = UsdGeom.XformCache().GetLocalToWorldTransform(camera_prim) |
There was a problem hiding this comment.
Are you sure this will return the transform use for rendering the frame in the viewport ? What if it has been updated after viewport rendering ?
There was a problem hiding this comment.
These are agent generated debugging functions. I removed it.
| # render settings | ||
| self.sim.render.enable_dl_denoiser = True | ||
| self.sim.render.antialiasing_mode = 'DLAA' | ||
| self.sim.render_interval = 5 |
There was a problem hiding this comment.
Maybe adding some comment to tell the render_interval is important for rendering quality.
There was a problem hiding this comment.
I moved this arg to train_config_real2sim.gin
|
|
||
| # Defaults shared by all scenes; override per-scene via NurecScene fields. | ||
| DEFAULT_USD_FILE = "stage_particle_spg.usdz" | ||
| DEFAULT_USD_FILE = "particle_spg-runtime.usdz" |
There was a problem hiding this comment.
Do we care about backward compatibility ? (this change will break previous asset)
There was a problem hiding this comment.
Good point .. I think support for particle USD with and without spg-runtime is sufficient. Volume is skipped, and I didn't test with old assets. Are there other assets that we should cover?
To support Particle USD with and without spg-runtime, I changed nurec-usd-file to a CLI arg, and add --spg-runtime / --no-spg-runtime arg.
The RTX render settings will always apply:
renderer_cfg=IsaacRtxRendererCfg(global_settings=IsaacRtxRendererGlobalSettingsCfg(
enable_dl_denoiser=True,
antialiasing_mode="DLSS",
carb_settings={
"/rtx/rtpt/gaussian/skipTonemapping/enabled": False,
"/rtx/rtpt/gaussian/accumulatedDepth/allHits/enabled": True,
"/rtx/rtpt/gaussian/accumulatedAlbedo/enabled": True,
"/rtx/rtpt/gaussian/maxGaussiansToAccumulate": 360,
},
)),
The spg setting depends on --spg-runtime arg:
--nurec-usd-file particle_spg-runtime.usdz --spg-runtime turns on spg setting, and copys render-product for kit viewport:
"--/rtx/spg/enabled=true",
"--/omni/rtx/nre/compositing/disableNuRecPostProcessings=true",
"--/rtx/rtpt/gaussian/skipTonemapping/enabled=false",
"--enable omni.rtx.spg",
--nurec-usd-file particle_sh_optimized.usdz --no-spg-runtime doesn't have spg settings and no render-product setting.
There was a problem hiding this comment.
Is this setting correct?
| data_types=["rgb", "depth"], | ||
| isp_cfg=CameraISPMode.AUTO_ANY, # ← enable PPISP (auto-discover baked shader) | ||
| isp_cfg=CameraISPMode.AUTO_ANY, | ||
| renderer_cfg=IsaacRtxRendererCfg(global_settings=IsaacRtxRendererGlobalSettingsCfg( |
There was a problem hiding this comment.
Some of those settings depends on the assets : are we sure all assets will be the same ? what about backward comp ?
| include 'configs/x_mobility_config.gin' | ||
|
|
||
| NUM_STEPS_PER_ITER = 256 | ||
| DEBUG_IMAGE_STEP_INTERVAL = 20 |
There was a problem hiding this comment.
Changed it to save debug image at step=20, instead of every 20 step, so it's adding too much debug write.
| settings.set("/" + str(key).replace(":", "/"), value) | ||
|
|
||
|
|
||
| def _apply_nurec_identity_exposure_to_camera(stage, camera_path): |
There was a problem hiding this comment.
Note that the identity exposure for camera with PPISP is already handled in IsaacLab
There was a problem hiding this comment.
Please help review nurec_utils.py for the kit viewport config, thank you!
| "omni:rtx:autoExposure:enabled", Sdf.ValueTypeNames.Bool).Set(False) | ||
|
|
||
|
|
||
| def _bind_nurec_ppisp_render_product(stage, viewport, nurec_usd_path, quiet=False): |
There was a problem hiding this comment.
Does compass really need this ? Compass is rendering through IsaacLab which is responsible for the renderproduct management.
69238ce to
be47d76
Compare
35ff403 to
00bd139
Compare
00bd139 to
aabb5a7
Compare
2 participants
Add configurable Isaac Lab Dockerfile support, NuRec asset downloads, Kit PPISP viewport binding/debug captures, and smoke configs for viewport validation.