Research snapshot · August 8, 2026
Autonomous Driving Technology Learning Roadmap
Research snapshot: August 8, 2026. This is a curriculum rather than a leaderboard survey. Resources were selected for conceptual durability, public accessibility, reproducibility, and relevance to real autonomous-driving stacks.
Map legend
Classification key#
- F
Foundational consensus
Classic theory, methods, textbooks, or benchmarks that form shared vocabulary. This does not mean the method remains state of the art.
- M
Mainstream research
An active direction represented by multiple papers, benchmarks, or public implementations.
- E
Exploratory
Evidence is still concentrated in papers, benchmarks, simulation, or limited prototypes.
- P
Public industry practice
A documented public stack, interface, toolchain, or standard originating from an industry-oriented project. This is not evidence of universal deployment, production safety certification, or superiority over proprietary alternatives.
Combined labels such as M/E indicate a mainstream research topic whose real-world maturity remains uncertain.
Reported accuracy, latency, benchmark rank, and deployment claims should always be interpreted as results reported by the authors or project team under their stated datasets, hardware, metrics, and configurations.
The route
Overall Learning Roadmap#
A productive sequence is:
Engineering foundation#
TopicsPython, modern C++, Linux, Git, CMake, Docker, unit testing, profiling, numerical debugging, ROS 2 basics
Anchor resourcesROS 2 tutorials plus small C++/Python sensor-message programs
Move on when you canBuild, test, profile, and containerize a program that reads timestamped sensor data and publishes typed messages.
Mathematics#
TopicsLinear algebra, multivariable calculus, Jacobians, probability, statistics, numerical optimization
Anchor resourcesGilbert Strang’s MIT 18.06 Linear Algebra (MIT OpenCourseWare); MIT 6.041SC Probabilistic Systems Analysis (MIT OpenCourseWare); Boyd and Vandenberghe’s Convex Optimization (Stanford University)
Move on when you canImplement least squares, SVD/PCA, Gaussian conditioning, maximum likelihood, gradient descent, and a constrained quadratic program.
Machine learning and computer vision#
TopicsGeneralization, supervised learning, neural-network optimization, CNNs, transformers, detection, segmentation, calibration and uncertainty
Anchor resourcesStanford CS229 (CS229 Machine Learning); Stanford CS231n (CS231n); Murphy’s Probabilistic Machine Learning books (ProbML)
Move on when you canTrain a model, establish a non-neural baseline, diagnose overfitting, reproduce metrics, and perform error analysis by scenario rather than reporting only an aggregate score.
Robotics geometry and sensors#
TopicsCoordinate frames, SE(2)/SE(3), quaternions, kinematics, dynamics, sensor models, calibration and synchronization
Anchor resourcesLynch and Park’s Modern Robotics (Hades)
Move on when you canTransform measurements correctly across map, odometry, body, camera, LiDAR, and sensor frames, including timestamp and convention checks.
Estimation, optimization and control#
TopicsBayesian filtering, EKF/UKF, smoothing, factor graphs, observability, robust estimation, PID, LQR, MPC
Anchor resourcesBarfoot’s State Estimation for Robotics (剑桥大学出版社); Tedrake’s Underactuated Robotics (MIT Underactuated Robotics)
Move on when you canFuse IMU/GNSS/wheel data, explain unobservable states, linearize a vehicle model, implement LQR, and formulate a constrained MPC problem.
Autonomous-driving fundamentals#
TopicsSensors, calibration, maps, prediction, routing, ODD, open-loop versus closed-loop evaluation, scenario simulation, failure handling
Anchor resourcesUniversity of Toronto Self-Driving Cars Specialization (Coursera); CARLA documentation and simulator (CARLA Simulator)
Move on when you canRun a repeatable simulated route, record all module inputs/outputs, calculate latency and trajectory metrics, and explain why an open-loop improvement may fail in closed loop.
Scene-estimation track#
TopicsPerception → tracking/prediction → localization/mapping
Anchor resourcesFollow the Perception and Localization sections below in parallel
Move on when you canProduce a timestamped world model and ego state with explicit frames, covariance/confidence, validity, and degradation status.
Motion track#
TopicsPlanning → control, initially using simulator ground truth
Anchor resourcesFollow the Planning and Control sections below in parallel
Move on when you canGenerate a dynamically feasible trajectory and track it under noise, latency, actuator constraints, and model error.
Integrated stack work#
TopicsInterface contracts, asynchronous pipelines, QoS, deterministic replay, fallback behavior, resource scheduling
Anchor resourcesAutoware or Apollo plus CARLA or recorded data
Move on when you canReplace one ground-truth input at a time with your own module and identify which closed-loop failures originate from perception, estimation, planning, control, or interfaces.
Research and industry-level capability#
TopicsReproduction, ablation, stress tests, profiling, scenario mining, uncertainty, safety monitors, ODD evidence
Anchor resourcesOne classic baseline, one modern benchmark, and one frontier paper per module
Move on when you canReproduce a baseline, explain discrepancies, run controlled ablations, profile real-time execution, identify failure clusters, and state clearly what the experiment does not establish.
Recommended sequencing rule#
For each module, use this order:
- Textbook or survey
- Simple implementation
- Canonical classical paper
- Modern benchmark and baseline
- Frontier reproduction
- Integration into an open stack
- Closed-loop and failure-oriented evaluation
Learned planners, learned localization, neural map construction, and learned control are much easier to judge after implementing a classical baseline that exposes geometry, uncertainty, constraints, and failure modes.
MODULE 01 — PERCEPTION
Perception#
Beginner#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F CS231n: Deep Learning for Computer Vision | Official course (CS231n) | Covers image classification, CNNs, attention, training practice, debugging, and visual recognition foundations needed before reading driving-perception papers. | Python, linear algebra, calculus, basic probability |
F Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite | Dataset and paper (CVLibs) | Introduces calibrated driving data and classic tasks such as stereo, odometry, optical flow, and object detection; especially useful for learning coordinate systems and evaluation protocols. | Basic computer vision, camera geometry |
F PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation | Paper/project (arXiv) | Establishes a clean way to learn directly from unordered point sets and explains permutation invariance, making later point-cloud architectures easier to understand. | Neural networks, basic 3D geometry |
F/M PointPillars: Fast Encoders for Object Detection from Point Clouds | Paper (arXiv) | Converts point clouds into vertical pillars and a 2D pseudo-image, giving a relatively accessible and efficient driving-oriented 3D-detection baseline. | CNNs, object detection, LiDAR coordinates |
Intermediate / Advanced#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F/M SECOND: Sparsely Embedded Convolutional Detection | Paper (MDPI) | Shows how sparse 3D convolutions make voxel-based LiDAR detection practical and provides the conceptual basis for many subsequent detectors. | Voxelization, CNNs, 3D detection metrics |
M Center-based 3D Object Detection and Tracking — CenterPoint | Paper and code (arXiv) | Models objects through centers rather than only anchor boxes and provides a widely reused detection-and-tracking baseline. | SECOND or PointPillars, heatmap detectors, tracking basics |
F/M nuScenes: A Multimodal Dataset for Autonomous Driving | Dataset/paper and devkit (arXiv) | Provides synchronized camera, LiDAR, radar, map, and vehicle data with 360-degree coverage, making it suitable for learning cross-sensor timing, transforms, and benchmark tooling. | Python, calibration, detection/tracking basics |
M Lift, Splat, Shoot | Paper/project (NVIDIA) | Offers an interpretable pipeline for lifting image features into 3D and aggregating them in bird’s-eye view, which is a good bridge to later BEV methods. | Multi-view geometry, CNNs, camera calibration |
M BEVFormer | Paper/project (GitHub) | Uses spatial and temporal attention to maintain BEV representations from surround cameras and is representative of mainstream camera-centric BEV research. | Transformers, LSS-style BEV models, temporal fusion |
M MMDetection3D | Repository and getting-started guide (GitHub) | Provides standardized datasets, models, configuration, training, inference, and evaluation for LiDAR, camera, and multimodal 3D detection. | PyTorch, Python packaging, dataset formats, GPUs |
Frontier and Industry Practice#
| Resource | Link | Core Content and Why Recommended | Major Limitations | Prerequisites |
|---|---|---|---|---|
M BEVFusion: Multi-Task Multi-Sensor Fusion with Unified BEV Representation | Paper and repository (arXiv) | Unifies camera and LiDAR features in BEV and illustrates a mainstream architecture for geometric-semantic fusion. | Requires careful calibration, timing, sensor-failure handling and compute budgeting; the original public repository was archived in January 2025, so maintenance and dependency risk should be assessed. (GitHub) | CenterPoint, LSS/BEVFormer, multimodal calibration |
M/E UniAD: Planning-Oriented Autonomous Driving | Paper/project (GitHub) | Jointly studies tracking, mapping, motion forecasting, occupancy and planning, making it useful for understanding integrated learning objectives. | Joint training can propagate errors across tasks; benchmark gains do not by themselves establish field robustness, safety, or maintainability. | BEV perception, tracking, prediction, planning metrics |
M/E SurroundOcc: Multi-Camera 3D Occupancy Prediction | CVF paper (开放获取计算机视觉会议) | Represents the move from object boxes toward dense semantic occupancy for describing visible and partially observed space. | Dense supervision and memory are expensive; visibility, occlusion, temporal consistency, calibration and cross-domain behavior remain important evaluation gaps. | BEV models, semantic segmentation, 3D grids |
E GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction | CVPR listing/project (开放获取计算机视觉会议) | Uses Gaussian primitives for streaming occupancy prediction and is useful for studying emerging continuous or sparse alternatives to dense voxel representations. | Very recent; public evidence is primarily benchmark-oriented, with limited independent evidence for long-duration driving, adverse weather, sensor faults, or safety impact. | Occupancy networks, Gaussian representations, temporal models |
P Autoware Perception Documentation | Main documentation and perception examples (Autoware Foundation) | Shows concrete public interfaces for LiDAR/camera/radar perception, fusion, tracking, traffic lights, confidence and downstream use. | Behavior depends on sensor suite, calibration, parameters, maps, hardware and ODD; public documentation is not a safety certificate or evidence of universal production performance. | ROS 2, sensor calibration, one detector/tracker implementation |
P Apollo LiDAR Detection and Perception Components | Official documentation/repository (Apollo 百度) | Provides an industry-origin example of model configuration, component integration and message flow inside a full autonomous-driving stack. | Results are stack-, version-, hardware- and calibration-specific; available models should not be treated as universally validated production solutions. | C++, Cyber RT, LiDAR detection, build-system familiarity |
MODULE 02 — LOCALIZATION / MAPPING
Localization / Mapping#
Beginner#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F Probabilistic Robotics | Publisher/book page (MIT Press) | Develops Bayes filters, localization, mapping, particle filters and probabilistic motion/sensor models that remain foundational vocabulary. | Probability, linear algebra, calculus |
F State Estimation for Robotics, 2nd Edition | Official book resources (剑桥大学出版社) | Gives a rigorous modern treatment of Lie-group estimation, batch methods, filtering and uncertainty with implementation-oriented derivations. | Linear algebra, probability, Jacobians; basic robotics helpful |
F/M Real-Time Loop Closure in 2D LiDAR SLAM — Cartographer | Paper and repository (Google Research) | Combines local scan matching with global loop closure in an inspectable system and is suitable for learning occupancy mapping, submaps and pose graphs. | 2D rigid transforms, scan matching, probability |
F/M ORB-SLAM2 | Paper/repository (GitHub) | Provides a complete monocular, stereo and RGB-D feature-based SLAM pipeline, including tracking, local mapping, loop detection and relocalization. | Camera geometry, feature matching, nonlinear least squares |
Intermediate / Advanced#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F/M LOAM: Lidar Odometry and Mapping in Real-Time | Official RSS paper page (Robotics Proceedings) | Separates high-rate odometry from lower-rate mapping and introduces feature-based point-to-line and point-to-plane registration used by many descendants. | 3D transforms, point-cloud registration, optimization |
M VINS-Mono | Paper/repository (GitHub) | Presents a tightly coupled optimization-based monocular visual-inertial estimator with initialization, loop closure and relocalization. | IMU preintegration, camera geometry, nonlinear optimization |
M LIO-SAM | Official repository/paper resources (GitHub) | Uses factor-graph smoothing to combine LiDAR odometry, IMU, loop closure and optional GPS in a modular implementation. | LOAM, IMU models, factor graphs, ROS |
M FAST-LIO2 | Paper and repository (arXiv) | Uses direct point registration and an iterated Kalman-filter formulation, providing an important contrast with feature-based and smoothing-based LIO. | Error-state filtering, LiDAR timing, IMU calibration |
M Scan Context | Paper/project (GitHub) | Introduces a compact LiDAR descriptor for place recognition and is straightforward to reproduce and analyze under viewpoint and environment changes. | Point-cloud polar coordinates, retrieval metrics |
P/M Lanelet2 | Project and paper (GitHub) | Provides a practical lane-level map model, routing graph and traffic-rule framework widely useful for understanding how maps serve localization and planning. | Road geometry, graph algorithms, C++ |
Frontier and Industry Practice#
| Resource | Link | Core Content and Why Recommended | Major Limitations | Prerequisites |
|---|---|---|---|---|
M/E MapTRv2: End-to-End Online Vectorized HD Map Construction | Paper and repository (arXiv) | Predicts structured vector-map elements directly from sensor input and is representative of mainstream online HD-map construction research. | Map-element average precision does not fully capture topology, geometric consistency, map freshness or planning usefulness; calibration and domain shift remain central concerns. | BEV perception, transformers, vector maps, nuScenes |
M/E FAST-LIVO2: Fast, Direct LiDAR-Inertial-Visual Odometry | Paper and repository (arXiv) | Tightly combines LiDAR, IMU and direct visual alignment in a shared map and is a strong study of high-rate heterogeneous sensor fusion. | Hardware synchronization, exposure, extrinsics, dynamic scenes and photometric assumptions require careful validation; published robotics demonstrations do not establish universal road performance. | FAST-LIO2, direct visual methods, iterated error-state filtering |
E SplaTAM: Splat Track and Map 3D Gaussians for Dense RGB-D SLAM | Paper/project (GitHub) | Demonstrates joint tracking and dense mapping with Gaussian splats and is useful for learning emerging neural scene representations. | Public evaluation is largely RGB-D and indoor-oriented; evidence for driving-scale, long-duration, dynamic and adverse-condition mapping is limited. | RGB-D SLAM, differentiable rendering, Gaussian splatting |
P Autoware NDT Scan Matcher and Localization Monitoring | NDT matcher and monitoring docs (Autoware Foundation) | Documents scan matching, initialization, regularization, covariance handling and localization-error monitoring in a complete ROS 2 stack. | Requires a suitable prior map and initialization; repetitive geometry, map changes, sparse returns and unreliable covariance can produce difficult failure modes. | NDT concepts, ROS 2, point-cloud maps, covariance |
P Apollo Localization Module | Official code documentation (Apollo 百度) | Exposes public configurations and implementations for GNSS/IMU, RTK, NDT and multisensor-fusion localization in a complete driving stack. | Public modules are tightly coupled to specific sensor, map, calibration, timing and platform assumptions; availability is not evidence of universal deployment quality. | C++, Cyber RT, GNSS/IMU, LiDAR localization |
P ASAM OpenDRIVE 1.9.0 | Official specification (GitLab) | Defines an exchange representation for static road networks, geometry, lanes, objects and signals and is important for simulation and map-tool interoperability. | It is not a localization algorithm and does not guarantee map freshness, localization accuracy, dynamic-world modeling or semantic consistency across every toolchain. | Road geometry, XML schemas, map semantics |
MODULE 03 — PLANNING
Planning#
Beginner#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F Planning Algorithms | Official book site (LaValle) | Provides a unified treatment of search, sampling, configuration spaces, dynamic programming and planning under uncertainty. | Algorithms, calculus, linear algebra |
F A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles | Paper (arXiv) | Connects route planning, behavior, trajectory generation and control and gives beginners a vocabulary for the major method families. | Basic robotics and vehicle kinematics |
F Practical Search Techniques in Path Planning for Autonomous Driving | Paper (斯坦福人工智能实验室) | Introduces Hybrid A* and practical heuristics for generating collision-free car-like paths while respecting nonholonomic motion. | A*, heuristics, bicycle-model intuition |
F/M Optimal Trajectory Generation for Dynamic Street Scenarios in a Frenet Frame | Official paper page and PDF view (IEEE Xplore) | Decomposes longitudinal and lateral trajectory generation in Frenet coordinates and remains a useful baseline for candidate sampling and cost design. | Polynomials, road reference lines, kinematics |
F/M PythonRobotics | Repository/project (GitHub) | Offers compact implementations of A*, Hybrid A*, sampling planners, path tracking, localization and control that are easy to inspect and modify. | Python, basic algorithms |
Intermediate / Advanced#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
M/P Baidu Apollo EM Motion Planner | Paper (arXiv) | Describes a practical path-speed decomposition and optimization pipeline and is valuable for understanding an industry-origin planning architecture. | Frenet planning, quadratic programming, road constraints |
M CommonRoad | Project and tooling (CommonRoad) | Provides standardized road scenarios, dynamic obstacles and drivability checks for comparing search-, optimization- and sampling-based planners. | Motion planning, vehicle models, Python |
M nuPlan Closed-Loop Planning Benchmark | Initial paper, benchmark paper and devkit (arXiv) | Provides logged driving data, scenario taxonomy, planner interfaces and closed-loop simulation, making it one of the most useful public environments for studying the open-loop/closed-loop gap. | Planning baselines, Python, substantial storage and compute |
M Rethinking Imitation-Based Planners for Autonomous Driving — planTF | Paper/project (arXiv) | Examines representation, training and evaluation choices for imitation planners and offers a comparatively clear baseline for nuPlan experiments. | PyTorch, imitation learning, nuPlan |
The original 2021 nuPlan proposal and the 2024 benchmark paper report different dataset totals because the benchmark and its documentation evolved; use the version-specific devkit and dataset documentation rather than mixing headline figures across releases. (arXiv)
Frontier and Industry Practice#
| Resource | Link | Core Content and Why Recommended | Major Limitations | Prerequisites |
|---|---|---|---|---|
M/E ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst | Waymo research page (Google Sites) | Demonstrates imitation-based planning augmented with perturbations and synthetic difficult situations, making it a useful foundation for later learning-based planners. | Imitation learning remains vulnerable to distribution shift and causal confusion; authors’ simulation and experimental findings do not constitute general safety evidence. | Imitation learning, rasterized scene inputs, trajectory losses |
M/E VAD: Vectorized Scene Representation for Efficient Autonomous Driving | Paper/project (GitHub) | Represents agents and maps as vectors and links scene understanding to trajectory planning in a compact architecture. | Much public evidence centers on open-loop nuScenes-style evaluation; vectorization errors, metric alignment and closed-loop transfer need separate testing. | BEV perception, transformers, trajectory prediction |
M/E UniAD | Paper/project (GitHub) | Treats perception, prediction and planning as jointly optimized tasks and is valuable for studying cross-task supervision and planning-oriented representations. | Joint models are expensive to train and diagnose, and improvements in component or open-loop metrics do not establish safe closed-loop behavior. | Multi-task learning, BEV, tracking, occupancy, planning |
E DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving | CVPR paper (开放获取计算机视觉会议) | Applies a truncated diffusion process to generate diverse driving trajectories and represents the current generative-planning direction. | Author-reported benchmark performance is not universal evidence; mode selection, rare-event coverage, reproducibility, closed-loop transfer and safety validation remain open engineering questions. | Diffusion models, imitation learning, end-to-end planning |
M/P infrastructure Waymax | Official project (GitHub) | Provides a vectorized, data-driven simulator suitable for large-scale planning, prediction and policy experiments. | It is not a raw-sensor simulator or a complete high-fidelity vehicle plant; conclusions depend on logged-data coverage, agent models and simulator fidelity. | JAX, trajectory prediction, closed-loop evaluation |
P Autoware Behavior Path Planner | Planning documentation and dynamic-obstacle example (Autoware Foundation) | Shows how behavior modules, lane changes, avoidance, road rules and trajectory optimization are decomposed in a public ROS 2 driving stack. | Rule coverage, module interactions, map semantics and parameters are ODD-specific; every new scenario can expose previously hidden interactions. | ROS 2, Lanelet2, Frenet/path planning, testing |
P Apollo Planning Framework: Scenario, Stage and Task Architecture | Scenario pipeline and public-road planner docs (Apollo 百度) | Provides a concrete example of scenario decomposition and reusable planning tasks in an industry-origin full stack. | Architecture and behavior depend on Apollo version, maps, configuration and intended ODD; public code does not establish universal production deployment. | C++, Cyber RT, finite-state machines, trajectory optimization |
MODULE 04 — CONTROL
Control#
Beginner#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F Vehicle Dynamics and Control, 2nd Edition | Publisher page (Springer) | Covers longitudinal and lateral dynamics, tire behavior, estimation and automotive controllers with equations directly relevant to road vehicles. | Differential equations, linear algebra, basic control |
F Underactuated Robotics | Official notes (MIT Underactuated Robotics) | Connects nonlinear dynamics, linearization, LQR, trajectory optimization, robust control and MPC in a computationally oriented treatment. | Calculus, differential equations, linear algebra |
F Implementation of the Pure Pursuit Path Tracking Algorithm | Report (卡内基梅隆大学机器人研究所出版物) | Gives a simple geometric path-tracking controller whose assumptions and tuning behavior are easy to visualize and test. | Coordinate geometry, bicycle-model intuition |
F Stanley: The Robot That Won the DARPA Grand Challenge | Paper (Stanford Robots) | Describes the complete Stanley vehicle and its geometric steering approach, providing historical context for modular autonomous-driving control. | Pure Pursuit, vehicle coordinates, basic feedback |
Intermediate / Advanced#
| Resource | Link | Core Content and Why Recommended | Prerequisites |
|---|---|---|---|
F/M Automatic Steering Methods for Autonomous Automobile Path Tracking | Report (卡内基梅隆大学机器人研究所出版物) | Compares geometric and model-based steering methods and is particularly useful for understanding gain, speed and path-curvature effects. | Pure Pursuit/Stanley, basic feedback control |
F/M Kinematic and Dynamic Vehicle Models for Autonomous Driving Control Design | Paper (Research Collection) | Derives and compares common vehicle models, clarifying when kinematic approximations become inadequate for controller design. | Dynamics, tire forces, state-space models |
M Predictive Active Steering Control for Autonomous Vehicle Systems | Paper (IEEE Xplore) | Demonstrates how model-predictive steering can incorporate vehicle dynamics and constraints and is a canonical bridge from LQR to MPC. | State-space control, optimization, vehicle dynamics |
M Optimization-Based Autonomous Racing of 1:43 Scale RC Cars | Paper and MPCC code (arXiv) | Introduces model predictive contouring control, jointly optimizing progress and path error while enforcing constraints. | MPC, nonlinear optimization, bicycle dynamics |
P Autoware MPC Lateral and PID Longitudinal Controllers | MPC and PID documentation (Autoware Foundation) | Shows concrete trajectory messages, timeout handling, vehicle-model selection, QP formulation, feedforward and feedback logic in a ROS 2 stack. | PID, linear MPC, ROS 2, actuator interfaces |
Frontier and Industry Practice#
| Resource | Link | Core Content and Why Recommended | Major Limitations | Prerequisites |
|---|---|---|---|---|
M Control Barrier Function Based Quadratic Programs for Safety-Critical Systems | Paper (arXiv) | Shows how control Lyapunov and barrier constraints can be combined in a QP, forming the basis of many runtime safety-filter designs. | Guarantees depend on model validity, correct safe-set construction, feasibility, state knowledge and continuous-versus-sampled implementation; a CBF is not a safety certificate by itself. | Nonlinear control, Lyapunov theory, constrained optimization |
E Differentiable MPC for End-to-End Planning and Control | Paper/project (arXiv) | Makes an MPC solution differentiable so model and cost parameters can be trained through the optimization layer. | The original experiments are mainly small control systems rather than road-vehicle validation; numerical stability and constraint activity can complicate gradients. | MPC, implicit differentiation, PyTorch |
M/E Learning-Based Model Predictive Control for Autonomous Racing | Paper/project (Research Collection) | Combines model learning with MPC in a physically demanding racing setting and clearly exposes the relationship between residual model error and controller improvement. | Racing and repeated-track assumptions do not generalize automatically to urban driving; learned dynamics need boundedness, coverage and fallback analysis. | MPCC, Gaussian processes or residual models, vehicle dynamics |
P/M toolchain acados | Repository and features (GitHub) | Provides C-based fast solvers and Python/MATLAB interfaces for nonlinear MPC, moving-horizon estimation and embedded optimal control. | It solves the problem formulated by the engineer; model quality, cost design, constraint correctness, scaling, warm starts, deadline behavior and fallback remain application responsibilities. | Nonlinear optimization, MPC, C/Python, numerical conditioning |
M/E TinyMPC | Paper/project (GitHub) | Demonstrates a compact MPC solver aimed at resource-constrained systems and is useful for learning embedded optimization tradeoffs. | Its strengths are most direct for structured linear or convex MPC; complex nonlinear vehicle and tire behavior may require approximations or a different solver. | Linear MPC, QP, embedded programming |
P/E Autoware Smart MPC Trajectory Follower | Official documentation (Autoware Foundation) | Combines path-following MPC with mechanisms for model learning and evaluation inside a public autonomous-driving stack. | Treat project-team demonstrations as implementation evidence rather than independent validation; model/data mismatch, dependencies, ODD coverage and failure fallback need dedicated testing. | Standard MPC, Gaussian processes or learned models, Autoware |
System map
Relationships Between Modules#
Sensors and vehicle buses
│
├── calibration, synchronization, diagnostics
│
├──────────────► Perception
│ objects, tracks, lanes, traffic lights,
│ free space / occupancy, uncertainty
│
└──────────────► Localization / Mapping
ego pose, velocity, acceleration,
covariance, map transforms, map match status
│
▼
Prediction / scene evolution
─────────────────────────────
predicted trajectories, probabilities,
occupancy flow, interaction hypotheses
│
▼
Route + map + ego state + perceived/predicted scene
│
▼
Planning
behavior + feasible trajectory
│
▼
Control
steering, throttle, braking commands
│
▼
Vehicle and actuator response
│
└────────► new sensor observations
Prediction is not one of the four requested modules, but it is an essential bridge. Public stacks explicitly distinguish prediction of other road users from ego-vehicle planning, while map-based prediction supplies future paths and probabilities to planning. (Apollo 百度)
Important interfaces#
| Producer → Consumer | Recommended Interface Content | Common Failure Modes |
|---|---|---|
| Sensors → Perception / Localization | Sensor timestamp, acquisition interval, frame ID, calibration version, raw measurement, health status and synchronization quality | Timestamp interpreted as arrival time; wrong axis convention; stale extrinsics; rolling-shutter or LiDAR-motion distortion; silent packet loss |
| Perception → Planning / Prediction | Object ID, class distribution, position, polygon/box, velocity, acceleration, covariance, existence probability, timestamp, free-space or occupancy representation, traffic-light state | Flickering tracks, duplicated objects, overconfident covariance, class-dependent bias, late data, map-frame mismatch |
| Localization / Mapping → All modules | Ego pose, twist, acceleration, covariance, map-to-odometry and odometry-to-body transforms, map version, localization state and degradation flags | Pose jump after relocalization, stale transform, covariance not reflecting degeneracy, map change, GNSS multipath |
| Planning → Control | Time-parameterized trajectory containing position, yaw, curvature, velocity, acceleration, jerk or their bounds, gear, stop intent, validity horizon and emergency status | Geometric path without timing; curvature discontinuities; infeasible acceleration; trajectory older than controller latency budget |
| Control → Vehicle / System monitor | Steering, throttle and brake requests; actuator mode; request timestamp; saturation status; commanded-versus-measured actuator state | Delay, dead zones, saturation, rate limits, steering offset, command arbitration, controller windup |
| Vehicle feedback → Localization / Control | Wheel speeds, steering angle, acceleration, yaw rate, gear, actuator state and diagnostic status | Scale-factor errors, CAN delay, filtering phase lag, steering-ratio mismatch |
Autoware’s trajectory follower, for example, checks input freshness and shares a trajectory reference between lateral and longitudinal controllers; Apollo’s Cyber RT documentation likewise illustrates decoupled components communicating over typed channels. (Autoware Foundation)
Concepts to study across modules in parallel#
- Coordinate frames and conventions: SE(2)/SE(3), handedness, quaternion conventions, map versus odometry frames, camera projection, LiDAR and radar frames.
- Time: sensor timestamp semantics, clock synchronization, interpolation, motion compensation, latency measurement and end-to-end age of information.
- Uncertainty: covariance, calibration, confidence, existence probability, multimodality, correlation and uncertainty-aware gating.
- Optimization: least squares, robust losses, sparse linear algebra, QP/NLP solvers, conditioning, warm starts and infeasibility handling.
- Road and vehicle geometry: Frenet coordinates, lane topology, bicycle models, curvature, steering and actuator constraints.
- Evaluation: dataset leakage, scenario stratification, confidence intervals, open-loop versus closed-loop metrics, regression tests and rare-event mining.
- Real-time engineering: scheduling, memory, GPU/CPU transfer, QoS, deterministic replay, monitoring, timeout behavior and graceful degradation.
- ODD and fallback: define where a module is expected to work, how it detects degradation, what information it exposes, and which fallback consumes that information.
Hands-on
Recommended Hands-On Projects#
Perception Projects#
| Project and Link | Features or Functions to Implement | Dataset / Simulation | Expected Learning Outcomes | Difficulty and Major Pitfalls |
|---|---|---|---|---|
| PointPillars → CenterPoint baseline ladder in MMDetection3D (arXiv) | Parse calibration; visualize points and boxes; run pretrained inference; train a small baseline; calculate class/range/occlusion error buckets; compare PointPillars and CenterPoint | KITTI first, then nuScenes mini/full (CVLibs) | Understand voxel/pillar encoding, 3D box conventions, training configuration and detection metrics | 3/5 Frequent problems: wrong yaw convention, coordinate transforms, class mapping, dataset version, GPU memory and incompatible framework dependencies |
| Camera-to-BEV with Lift-Splat-Shoot (NVIDIA) | Implement image encoder, depth-bin distribution, lift, geometric projection, splat and BEV segmentation; perturb extrinsics and measure degradation | nuScenes mini; optionally CARLA for controlled calibration perturbations (arXiv) | Understand view transformation, calibration sensitivity and why BEV representations simplify downstream geometry | 4/5 Pitfalls: inconsistent augmentation across camera and calibration, depth discretization, memory use, asynchronous cameras and weak debugging visualizations |
| Open-stack perception replay | Replay a CARLA or recorded ROS bag; inspect detected-object and traffic-light messages; measure per-node and end-to-end latency; replace one detector or classifier | CARLA plus Autoware perception documentation (CARLA Simulator) | Learn real module contracts, timestamps, frame IDs, QoS, diagnostics and integration testing | 4/5 Pitfalls: build/version conflicts, unavailable acceleration libraries, frame mismatch, stale messages and assuming model inference time equals end-to-end latency |
Localization / Mapping Projects#
| Project and Link | Features or Functions to Implement | Dataset / Simulation | Expected Learning Outcomes | Difficulty and Major Pitfalls |
|---|---|---|---|---|
| EKF and batch-estimation sensor-fusion notebook | Simulate or load IMU, wheel odometry and GNSS; estimate pose, velocity and sensor bias; compare EKF with batch least squares; inspect innovation and covariance consistency | Synthetic trajectory, CARLA, or KITTI vehicle data (剑桥大学出版社) | Understand process/measurement models, linearization, bias, observability and covariance | 2–3/5 Pitfalls: degrees versus radians, gravity/sign conventions, timestamp alignment, unrealistic white-noise assumptions and tuning covariance solely to improve trajectory plots |
| Visual/LiDAR SLAM replay and evaluation | Run ORB-SLAM2, Cartographer or LOAM; generate maps; calculate absolute and relative trajectory error; disable loop closure; perturb calibration | Datasets supported by the selected project, including KITTI where applicable (GitHub) | Learn front end versus back end, drift, loop closure, map consistency and evaluation alignment | 3/5 Pitfalls: monocular scale, trajectory frame alignment, dataset-specific settings, loop-closure false positives and comparing trajectories with different timestamp sampling |
| LIO-SAM or FAST-LIO2 integration (GitHub) | Implement or inspect point-cloud deskew, IMU initialization/preintegration, extrinsics, map update, loop/GNSS factors and relocalization behavior | Official example bags or compatible LiDAR-IMU data; KITTI requires careful format/timing adaptation | Understand high-rate sensor fusion and the engineering difference between factor-graph and iterated-filter systems | 4–5/5 Pitfalls: missing per-point timestamps, incorrect IMU axis/sign, gravity initialization, poor hardware synchronization, LiDAR-specific scan patterns and dynamic objects |
Planning Projects#
| Project and Link | Features or Functions to Implement | Dataset / Simulation | Expected Learning Outcomes | Difficulty and Major Pitfalls |
|---|---|---|---|---|
| Planner ladder: A_ → Hybrid A_ → Frenet trajectories | Implement grid A*, nonholonomic Hybrid A*, collision checking with vehicle footprint, Frenet polynomial candidates and cost terms for jerk, curvature, clearance and progress | PythonRobotics examples and small custom maps (斯坦福人工智能实验室) | Understand graph search, motion primitives, heuristic design, reference-line coordinates and trajectory costs | 2–3/5 Pitfalls: inadmissible heuristics, inconsistent units, point-only collision checking, discontinuous curvature and tuning one scenario rather than a scenario set |
| CommonRoad feasible-planner benchmark | Implement a lattice, sampling or optimization planner; validate collision, road-boundary and kinodynamic constraints; create failure categories | CommonRoad scenarios and drivability checker (CommonRoad) | Learn standardized scenario I/O, feasibility checking and comparison across planner families | 3–4/5 Pitfalls: scenario-coordinate semantics, prediction assumptions, numerical collision tolerances, incorrect vehicle dimensions and confusing goal-region satisfaction with safe completion |
| nuPlan closed-loop baseline and planTF experiment | Run a rule-based/PDM-style baseline; run planTF; change one input feature, loss or augmentation; evaluate both open-loop and closed-loop; inspect scenario-level regressions | nuPlan devkit and dataset (GitHub) | Learn simulator integration, scenario mining, closed-loop metrics and the difference between imitation error and driving quality | 5/5 Pitfalls: large storage and preprocessing cost, software-version friction, metric gaming, reactive-agent assumptions and drawing conclusions from an aggregate score without scenario inspection |
Control Projects#
| Project and Link | Features or Functions to Implement | Dataset / Simulation | Expected Learning Outcomes | Difficulty and Major Pitfalls |
|---|---|---|---|---|
| Pure Pursuit → Stanley → LQR comparison | Implement all three around a common bicycle model; add speed-dependent gains, steering saturation, sensor noise and delay; compare lateral error, heading error and steering smoothness | CARLA or F1TENTH/RoboRacer simulation (卡内基梅隆大学机器人研究所出版物) | Understand geometric versus model-based feedback and how speed, path curvature and latency change behavior | 2–3/5 Pitfalls: sign and frame conventions, low-speed singularities, lookahead tuning, derivative noise, ignoring actuator delay and comparing controllers with unequal saturation logic |
| Constrained linear MPC trajectory tracker | Derive/discretize a bicycle model; construct QP cost and constraints; add steering-rate and acceleration limits, delay compensation, warm start and infeasibility fallback | CARLA or F1TENTH; compare with Autoware’s documented MPC structure (Research Collection) | Learn prediction models, receding horizon, cost scaling, constraints and solver timing | 3–4/5 Pitfalls: poor scaling, wrong linearization point, overly short horizon, infeasibility, omitted actuator dynamics and tuning only at one speed |
| NMPC/MPCC with acados and optional CBF filter | Implement nonlinear bicycle dynamics, contouring cost, track/road constraints and actuator-rate limits; profile worst-case solve time; add an independently testable CBF safety filter | F1TENTH/RoboRacer or a controlled CARLA track (arXiv) | Learn real-time nonlinear optimization, solver diagnostics, model mismatch and layered safety-control design | 5/5 Pitfalls: bad initial guesses, ill-conditioned costs, missed deadlines, infeasible constraints, tire-model mismatch and claiming formal safety when perception, state estimation or sampled implementation violates assumptions |
Caution
Evidence-Limited Directions#
The following are worth monitoring but should not be presented as established industrial consensus:
- End-to-end integrated driving: UniAD, VAD and DiffusionDrive demonstrate important research directions, but public results remain strongly dependent on benchmark design, input representation and open-loop or simulated closed-loop metrics. (GitHub)
- Generative planners and driving world models: Diffusion and latent-world-model approaches are increasingly visible, but publicly reproducible evidence for rare-event coverage, calibrated uncertainty, real-time fallback and broad ODD field robustness remains limited.
- Gaussian occupancy and neural dense mapping: GaussianWorld and SplaTAM are promising representational studies, but the available public evidence does not yet justify treating them as mature replacements for conventional road-scale localization or mapping pipelines. (开放获取计算机视觉会议)
- Learning-based low-level control: Published demonstrations often use racing, small vehicles or simplified plants. They are valuable for model adaptation research but do not automatically establish robustness for general urban vehicles. (Research Collection)
- Public open-stack components: Autoware and Apollo are valuable sources for architecture and integration practice, but their public documentation should not be interpreted as a complete record of proprietary deployment configurations, validation evidence or certification.
The short shelf
Minimum Essential Reading List#
This is the compact path with the highest expected return on learning effort. Datasets and stack documentation should be used alongside the readings as lab infrastructure.
| Module | Three Essential Beginner Resources | Three Essential Intermediate / Advanced Resources | Representative Frontier Resources |
|---|---|---|---|
| Perception | 1. CS231n (CS231n) 2. PointNet (arXiv) 3. PointPillars (arXiv) | 1. SECOND (MDPI) 2. CenterPoint (arXiv) 3. Lift-Splat-Shoot (NVIDIA) | BEVFusion (arXiv); UniAD (GitHub); SurroundOcc (开放获取计算机视觉会议) |
| Localization / Mapping | 1. Probabilistic Robotics (MIT Press) 2. State Estimation for Robotics (剑桥大学出版社) 3. ORB-SLAM2 (GitHub) | 1. LOAM (Robotics Proceedings) 2. VINS-Mono (GitHub) 3. FAST-LIO2 (arXiv) | MapTRv2 (arXiv); FAST-LIVO2 (arXiv); Autoware NDT localization (Autoware Foundation) |
| Planning | 1. Planning Algorithms (LaValle) 2. Paden et al. survey (arXiv) 3. Hybrid A* (斯坦福人工智能实验室) | 1. Frenet trajectory generation (IEEE Xplore) 2. Apollo EM Planner (arXiv) 3. nuPlan (arXiv) | VAD (GitHub); UniAD (GitHub); DiffusionDrive (开放获取计算机视觉会议) |
| Control | 1. Vehicle Dynamics and Control (Springer) 2. Underactuated Robotics (MIT Underactuated Robotics) 3. Pure Pursuit report (卡内基梅隆大学机器人研究所出版物) | 1. Kinematic and Dynamic Vehicle Models (Research Collection) 2. Falcone et al. predictive steering MPC (IEEE Xplore) 3. Model Predictive Contouring Control (arXiv) | CBF-QP (arXiv); Learning-Based MPC (Research Collection); acados (GitHub) |
A learner reaches credible module-level capability not by accumulating papers, but by being able to:
- derive and implement a simple baseline;
- reproduce a modern baseline under a pinned environment;
- explain frames, time, uncertainty and constraints at every interface;
- measure runtime and data age rather than only neural inference time;
- identify scenario-specific failure clusters;
- distinguish open-loop, simulated closed-loop and real-world evidence;
- integrate the module with explicit validity, degradation and fallback behavior.