Toward Reliable and Adaptive Perception in Autonomous Mobile Systems

Autonomous vehicles and robots rely on perception systems for safety-critical decision-making, yet their reliability can be compromised by adversarial attacks and natural sensing degradations.

Attackers may place physical objects, such as colored patches, cardboard, or metal foil, to deceive camera-, LiDAR-, radar-, and sensor-fusion-based detectors. Existing defenses are often limited to specific attacks or sensing modalities. Our first work develops a generalized runtime defense and adaptation framework that combines real-time attack detection with deep reinforcement learning-based motion planning. The system adapts its trajectory toward viewpoints from which the attack is less effective, while uncertainty-aware filtering suppresses corrupted perception outputs. Experiments in both simulated and real-world environments achieve an attack mitigation success rate of over 90% while maintaining safe and smooth motion.

Beyond adversarial attacks, perception failures may also arise from natural sensing factors and inherent model limitations, including variations in object range, size, orientation, weather, occlusion, and environmental context. To address these broader challenges, we introduce a physically aware reliability atlas that captures historical detector behavior under diverse conditions and retrieves condition-matched references at runtime. It determines whether a detection is statistically typical, weakly supported, or atypical. The framework achieves high anomaly detection rates across sensor modalities and degradation conditions.

As future work, we will integrate the reliability monitor with large vision-language models to validate atypical detections, diagnose failures, and guide runtime adaptation through sensor reconfiguration and model selection, while balancing reliability and computational overhead for closed-loop perception under uncertain conditions.

Committee: Chenglin Miao (major professor), Meisam Mohammady, Wensheng Zhang, Ying Cai and Yang Li