ICRA 2026 Workshop · Open Challenges for Rigorous Robot Perception
A manipulation-oriented extension of OSMa-Bench for evaluating open-vocabulary semantic maps under controlled synthetic conditions: clutter, small objects, support relations, containment, and lighting variation.
Motivation
Existing open-vocabulary semantic mapping methods can detect objects and build scene-level representations, but downstream manipulation requires more than object names. A robot often needs reliable object counts, support relations, containment, relative placement, and small manipulable objects grounded in 3D space.
OSMa-Bench++ targets this gap by generating controlled indoor scenes from prompts and evaluating whether semantic maps preserve information that is relevant for mobile manipulation.
Pipeline
Generate diverse indoor scene descriptions with an LLM, then filter near-duplicates using cosine similarity to retain varied layouts, small objects, clutter, and spatial relations.
Create 3D scenes in SceneSmith directly from the generated prompts.
Convert SceneSmith assets into a Habitat-compatible format for simulation.
Use HaDaGe to generate RGB-D trajectories under multiple lighting conditions.
Build semantic maps using ConceptGraphs and BBQ.
Evaluate segmentation quality and scene-graph completeness with prompt-grounded object counting and relation queries.
Dataset
The generated dataset extends fixed-scene evaluation with controllable indoor scenarios. Furniture scenes test whether methods preserve global layout and large-object structure, while manipuland scenes stress-test small-object detection, clutter handling, occlusions, and relation understanding required for manipulation.
baseline — static, non-uniform light sources.nominal — uniform illumination from emissive meshes.camera — directional light attached to the robot camera.dynamic_lights — illumination changes along the robot trajectory.Results
We evaluate two scene-graph-based open semantic mapping methods: ConceptGraphs and BBQ. OSMa-Bench++ reveals complementary failure modes: segmentation robustness, object recovery, and scene-graph completeness degrade differently across methods and lighting conditions.
| Method | Baseline | Camera | Dynamic | Nominal | ||||
|---|---|---|---|---|---|---|---|---|
| mAcc | f-mIoU | mAcc | f-mIoU | mAcc | f-mIoU | mAcc | f-mIoU | |
| ConceptGraphs | 57.6 | 27.9 | 53.7 | 14.8 | 56.5 | 28.0 | 55.3 | 26.6 |
| BBQ | 47.0 | 68.4 | 42.0 | 42.8 | 48.2 | 63.5 | 45.3 | 53.5 |
Camera lighting causes the strongest degradation. BBQ keeps higher f-mIoU, while ConceptGraphs achieves higher mAcc, indicating broader object recovery.
| Subset | Category | Method | Baseline | Camera | Dynamic | Nominal |
|---|---|---|---|---|---|---|
| Furniture | Measurements | ConceptGraphs | 12.2 | 9.2 | 15.3 | 14.3 |
| BBQ | 15.3 | 24.5 | 12.2 | 12.2 | ||
| Relations | ConceptGraphs | 12.9 | 6.4 | 20.5 | 17.0 | |
| BBQ | 18.7 | 24.6 | 14.6 | 18.1 | ||
| Manipuland | Measurements | ConceptGraphs | 12.9 | 11.4 | 17.1 | 12.9 |
| BBQ | 15.7 | 24.3 | 18.6 | 22.9 | ||
| Relations | ConceptGraphs | 15.0 | 14.0 | 13.0 | 9.0 | |
| BBQ | 16.0 | 21.0 | 14.0 | 19.0 |
| Subset | Category | Method | Standard | PromptGT |
|---|---|---|---|---|
| Furniture | Measurements | ConceptGraphs | 30.6 | 12.2 |
| BBQ | 18.4 | 15.3 | ||
| Relations | ConceptGraphs | 19.7 | 12.9 | |
| BBQ | 24.6 | 18.7 | ||
| Manipuland | Measurements | ConceptGraphs | 32.2 | 12.9 |
| BBQ | 33.9 | 15.7 | ||
| Relations | ConceptGraphs | 22.2 | 15.0 | |
| BBQ | 26.3 | 16.0 |
PromptGT is stricter because it directly probes prompt-specified counts and relations rather than trajectory-dependent observations.
Citation
@misc{kurkova2026osmabench,
title = {OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes},
author = {Kurkova, Regina and Popov, Maxim and Kolyubin, Sergey},
year = {2026},
eprint = {2605.26831},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2605.26831}
}