Sim2Real-AD:
Closed-loop safety behavior on the vehicle: DriveVLM-RL vs. VLM-RL.
1. Transferable Principle with a Transfer Guarantee. The sim-to-real gap splits into two orthogonal axes, sensing-and-dynamics and task-and-geometry; the former closes without any real-world policy training, and a transfer guarantee bounds the deployment gap by three controllable error terms.
2. Modular, Reward-Agnostic Framework. Sim2Real-AD splits transfer into an observation-space bridge and an action-space bridge, enabling deployment without real-world training or learning-based domain adaptation.
3. Geometric Observation Bridge (GOB). Monocular images become a unified BEV semantic representation via pretrained segmentation and inverse perspective mapping, fully interpretable and training-free.
4. Physics-Aware Action Mapping + Two-Phase Training. PAM and TPT decouple policy learning from platform-specific control, enabling zero-shot transfer with ~30 minutes of calibration and no real-world data, demonstrated on a full-scale Ford E-Transit.
(a) Phase 1 — Action-Space Adaptation: PAM recasts policy outputs as platform-agnostic physical commands via a bicycle model and a PID controller, while observations stay on the privileged GT-BEV. (b) Phase 2 — Observation-Space Adaptation: GOB replaces GT-BEV with a monocular, training-free BEV, with the action interface frozen. (c) Two-Phase Progressive Training (TPT) adapts the action and observation interfaces in sequence rather than simultaneously. (d) Real-time Deployment Pipeline (RDP) integrates the modules into a closed-loop on-vehicle system that meets the 20 Hz control budget.
GOB turns a monocular front-view image into a simulator-compatible multi-channel BEV through Cityscapes-pretrained semantic segmentation and inverse perspective mapping, fully interpretable and training-free. The real-vehicle BEV (bottom) matches the tensor shape and channel layout the policy was trained on, so the same network runs unchanged.
▶ In CARLA (simulation)
GOB in simulation: the Source front view is turned into an IPM BEV that closely matches the simulator's GT BEV, frame by frame.
▶ On the real vehicle
The same bridge on the real vehicle: the front-view cam0 image becomes a GOB-BEV with the tensor shape and channel layout the policy was trained on, so the same network runs unchanged, with no real-world training.
(a) Lab-developed full-scale battery-electric Ford E-Transit van with drive-by-wire control. (b) Forward-facing and elevated (bird's-eye) camera configuration. (c) Three evaluation scenarios: car-following, static obstacle avoidance, and semantic-critical stop-sign interaction.
All curves below are real recorded on-vehicle logs. Across the three scenarios, DriveVLM-RL keeps a stronger safety profile than VLM-RL under the identical transfer pipeline.
Closed-loop safety behavior on the vehicle: DriveVLM-RL vs. VLM-RL.
Liveness / progress behavior during the same real-world runs.
The same Sim2Real-AD pipeline transfers two reward backbones onto the full-scale Ford E-Transit van, zero-shot and with no real-world training data. Press play on the onboard recordings below, paired with the real recorded logs from the paper. The semantically grounded DriveVLM-RL reward keeps its safety edge on the vehicle: it handles the hazards on which VLM-RL needs a human takeover.
▶ DriveVLM-RL
Smooth, stable following
▶ VLM-RL
Completes, but more oscillatory
Both complete the route; DriveVLM-RL tracks a steadier speed and steering profile. (Clips shown at 2× speed.)
Speed: DriveVLM-RL regulates speed more smoothly than VLM-RL.
Steering: DriveVLM-RL makes more gradual adjustments.
Real recorded traces (S1): DriveVLM-RL sustains a smoother velocity and steadier steering while following the lead vehicle.
▶ DriveVLM-RL
✓ Steers around the obstacle
▶ VLM-RL
✗ Human takeover required
DriveVLM-RL avoids the parked vehicle; VLM-RL drives straight at it and is taken over.
Speed: VLM-RL approaches the obstacle without a bypass; DriveVLM-RL preserves forward progress.
Trajectory (colored by speed): DriveVLM-RL bypasses the parked vehicle; VLM-RL stops short and is taken over (×).
Real recorded logs (S2): DriveVLM-RL curves around the parked vehicle and continues, while VLM-RL continues straight until a human takes over.
▶ DriveVLM-RL
✓ Slows and stops
▶ VLM-RL
✗ Fails to stop in time
DriveVLM-RL brakes for the stop sign and crossing pedestrian; VLM-RL keeps going.
Speed: VLM-RL holds speed until intervention; DriveVLM-RL decelerates to a complete stop before the crosswalk.
Distance and speed: DriveVLM-RL increases clearance to the pedestrian while stopping before the crosswalk, whereas VLM-RL fails to yield.
Real recorded logs (S3): DriveVLM-RL decelerates to a stop at the sign, while VLM-RL maintains speed through it.
Per-step latency vs. the 20 Hz budget.
Onboard compute utilization during deployment.
Because all VLM components are confined to offline training, the deployed policy runs at 26.8 ms per step (53.6% of the 20 Hz budget) on the onboard GPU, whereas direct VLM-as-control inference (580 ms) is far beyond the real-time limit.
@article{huang2026sim2real,
title = {Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving},
author = {Huang, Zilin and Wan, Zhengyang and Sheng, Zihao and Wang, Boyue and You, Junwei and Chen, Sikai},
journal = {arXiv preprint arXiv:2604.03497},
year = {2026}
}