Learning | ACP Lab

VLM-Guided Semantic Safety for Onboard Aerial Autonomy

Fri, 02 Oct 2026 12:30:00 +0000

Geometric collision avoidance alone is not enough for aerial robots that operate around people and valuable assets. A drone should also understand what it is flying over and near, and adjust its behavior based on the semantics of the scene. Vision-language models open a path to this kind of semantic safety reasoning directly onboard the robot.

Our first work in this direction, “VLM-Guided Semantic Safety for Onboard Aerial Autonomy” by Tarunkumar Palanivelan, Anirudh Nallawar, and Prof. Guanrui Li, was accepted at two IROS 2026 workshops, the 3rd AI Meets Autonomy workshop and the Workshop on Embodied Neuro-Symbolic AI for Reliable and Safe Robotics (ReS AI). This student project builds on that work. Students will help extend the framework, improve its onboard performance, and validate it in real flight experiments.

Desired Skills

Students are not expected to have experience in every area, but familiarity with one or more of the following is desirable.

• Deep learning with PyTorch, ideally with vision-language or other foundation models

• ROS 2 or other robotics middleware

• An interest in onboard computing, perception, and safe autonomy

Learning agile flights in clustered environments for quadrotors from pixel information using differentiable simulators and novel sensors

Thu, 01 May 2025 20:07:33 +0000

This project focuses on enabling high-speed, agile autonomous aerial robot navigation in visually degraded and environmentally challenging conditions. These include scenarios such as post-earthquake zones, underground exploration, anti-poaching missions, and night-time search and rescue operations where conventional sensors like LiDARs and cameras fail. To address these challenges, the project introduces a novel visuo-sonic sensing suite that combines low-power event cameras and ultrasound sensors, which excel in conditions like darkness, fog, smoke, and transparent environments. Leveraging differentiable physics-based learning, the system adapts control policies in real-time to unconventional sensor inputs, ensuring reliable operation in harsh and dynamically changing environments.

Learning Contraction Metrics for Robust Control of Hybrid Aerial Robots with Suspended Payloads

Thu, 05 Dec 2024 20:07:33 +0000