Integrated perception and prediction for autonomous driving: A survey and perspectives

Safe autonomous driving requires not only accurate perception of the current traffic scene, but also reliable prediction of how the surrounding environment will evolve. Conventional modular stacks decouple perception and prediction, which simplifies component development but can amplify upstream errors and restrict joint reasoning over geometry, semantics, interaction, and uncertainty. Integrated Perception and Prediction for Autonomous Driving (PnP4AD, or PnP for short) addresses this issue by jointly learning scene understanding and future forecasting from raw sensor observations while preserving explicit future predictions for downstream planning. We classify existing PnP methods according to the dominant internal scene representation. Sparse entity-centric methods organize scene understanding and future reasoning around agents, vectorized entities, or query-level states. Dense scene-centric methods model scene evolution through spatial fields such as Bird’s-Eye-View (BEV) semantics, occupancy grids, future instance masks, and motion flows. Hybrid methods couple dense spatial representations with sparse instance-level states, aiming to combine scene-level coverage with identity- and interaction-aware reasoning. For each paradigm, we analyze its assumptions, representative architectures, and trade-offs. We further summarize common sensor inputs, output modalities, datasets, benchmarks, and evaluation metrics. Finally, we discuss challenges that arise specifically from integrating perception and prediction. This survey aims to clarify the design space of PnP4AD and identify research directions toward planner-aware and deployable future-scene prediction.