End-to-End
Autonomous Vehicle Driving
A research project on learning-based end-to-end driving policies,
from simulation development to real-world deployment on a full-scale autonomous test vehicle
Overview
This project investigates end-to-end autonomous driving, where a single learned policy maps raw sensor observations directly to vehicle control, rather than relying on separately engineered perception, prediction, and planning modules.
While end-to-end methods have shown promising results in simulation, transferring them to real vehicles remains challenging. Policies trained in simulators often face a sim-to-real gap—differences in sensing, dynamics, and scenarios between virtual environments and the physical world—and it is often unclear how well published methods hold up outside their original benchmarks.
To address this, we reproduce and evaluate existing end-to-end driving methods in CARLA, benchmarking their feasibility and identifying performance gaps in the Mcity simulation environment. Building on these findings, we are developing a sim-to-real pipeline that covers driving demonstration collection, policy training, and evaluation, with the goal of deploying an end-to-end policy on Mcity’s full-scale test vehicle.