I spent this summer writing autonomous vehicle software at the RANCS lab at USF — Resilient, Autonomous, Networked Control Systems — where the research spans control and communication for connected and autonomous vehicles, vehicle security, and platooning. The part I lived in was localization: getting a vehicle to answer, continuously and reliably, the question of where it is.
There is a version of that problem that looks solved. You take a LiDAR-inertial odometry package off the shelf, run it on a public dataset, watch the trajectory come out clean, and conclude that localization is a library call. Then you drive a real car through a real parking lot and the illusion evaporates.
Odometry is a graph, not a sensor
The mental model that made everything else click: modern LiDAR-inertial systems are not producing a position, they are maintaining a pose graph. Each scan match adds a constraint. The IMU adds constraints between scans. GPS, when it is trustworthy, adds constraints tying the whole graph to the world frame. Loop closures add constraints saying "you have been here before." The estimate is whatever pose configuration best satisfies all of it at once.
Once you see it that way, the tuning knobs stop feeling arbitrary. A threshold that decides which GPS fixes are allowed into the graph is not a magic number — it is a statement about how much you trust one sensor relative to the others. Adding GPS factors too aggressively drags a locally-consistent trajectory toward noisy global fixes. Adding them too rarely lets drift accumulate unchecked. There is no correct setting, only a tradeoff you are choosing on purpose or by accident.
Real data lies in specific ways
Public datasets are polite. The sensors are calibrated, the timestamps line up, the sky is open. Real logs are not polite:
GPS degrades exactly where you need it. Next to a building, under cover, in the shadow of anything tall, fixes drift by meters while still reporting confidence. A filter that trusts the reported accuracy will happily wreck a good trajectory.
Loop closure is a proof-of-concept until it is not. Recognizing that you have returned to a place is easy in a small log and genuinely hard when the candidate set grows and near-identical geometry appears in two different places.
Playback speed is a real constraint. Replaying a log faster than the optimizer can keep up produces results that look like an algorithm failure but are actually a compute failure. Learning to tell those two apart saved me more time than any single fix.
The gap between simulation and the parking lot
RANCS runs a Vehicle-in-the-Loop setup, where a real vehicle interacts with a virtual environment — a digital twin. It is a genuinely good way to test dangerous scenarios without danger, and it makes the gap it cannot close very legible.
Simulated sensors are too clean, and every unrealistically clean input is a bug you will not find until later. Simulation tells you whether your logic is right. It does not tell you whether your assumptions are. Only the parking lot does that, which is why so much of the summer was spent between the two: reproduce in simulation, verify on the vehicle, then explain the difference.
What I am taking with me
The habit of distrusting a working demo. A trajectory that looks right on one log is a hypothesis, not a result — and the interesting engineering starts when you go looking for the conditions that break it.
Also, more practically: research code and production code fail differently, and both are worth being fluent in. The version that gets a paper figure and the version that runs on a car at speed are not the same program, and pretending otherwise is how people get hurt.
Thanks to everyone at RANCS for a summer of good problems.