I spent this summer working on autonomous vehicle software at the RANCS Lab at USF, short for Resilient, Autonomous, Networked Control Systems. The lab works on areas like connected and autonomous vehicles, vehicle security, platooning, and control systems.
Most of my work was focused on localization, testing, and validation. I worked with the software that helps the vehicle figure out where it is, then tested that software using recorded sensor data, simulation, and the actual vehicle. A big part of the work was not just getting something to run, but figuring out whether the result was reliable and repeatable.
That meant validating sensor inputs, comparing outputs across different runs, testing under different conditions, and tracking down why something failed. Sometimes the issue was the algorithm. Sometimes it was bad sensor data, timing, configuration, hardware, or simply the computer not keeping up with everything in real time.
At first, localization can seem like a mostly solved problem. You can take a LiDAR-inertial odometry package, run it on a public dataset, watch it generate a clean trajectory, and assume the hard part is done.
Then you put it on a real car and drive through a parking lot.
That assumption disappears pretty quickly.
Odometry is a graph, not a sensor
The biggest thing that helped me understand modern localization systems was realizing that they are not simply producing a position from a sensor. They are maintaining and optimizing a pose graph.
Each LiDAR scan match adds a constraint between poses. The IMU adds constraints describing motion between scans. GPS can add global position constraints when the measurements are reliable. Loop closures add another type of constraint by telling the system that the vehicle has returned to a place it has already seen.
The final trajectory is the configuration of poses that best satisfies all of those constraints together.
Once I started thinking about the system this way, a lot of the tuning parameters made more sense.
For example, deciding whether to accept a GPS measurement is really a decision about how much you trust GPS compared to the rest of the system. If you add GPS factors too aggressively, noisy measurements can pull an otherwise consistent trajectory away from where it should be. If you barely use GPS at all, the system can slowly drift over time.
There is not always one correct threshold. You are choosing a tradeoff, whether you realize it or not.
Real data fails in very specific ways
Public datasets are usually well behaved. The sensors are calibrated, timestamps are synchronized, and the data collection environment is often fairly controlled.
Real vehicle data is different.
GPS becomes unreliable exactly where you wish it would not. Driving next to buildings, under cover, or around other large structures can cause GPS measurements to shift by several meters. The frustrating part is that the receiver may still report the measurement with reasonable confidence. If the localization system trusts that value blindly, one bad fix can damage an otherwise good trajectory.
Loop closure works well until the environment gets complicated. Detecting that the vehicle has returned to a previous location is straightforward in a small dataset. It gets much harder as the log gets larger and different areas begin to contain similar-looking geometry. A false loop closure can be much worse than missing one entirely.
Playback speed matters more than I expected. If you replay recorded sensor data faster than the optimizer can process it, the resulting trajectory can look completely wrong. At first, it can seem like a SLAM or tuning problem when the actual issue is simply that the system cannot keep up with the incoming data. Learning to separate algorithm failures from compute failures saved me a lot of debugging time.
The gap between simulation and the parking lot
RANCS also uses a Vehicle-in-the-Loop setup where a real vehicle interacts with a simulated environment and digital twin.
It is extremely useful for testing scenarios that would be difficult or unsafe to reproduce with a real vehicle. It also makes the limitations of simulation very obvious.
Simulated sensors are usually cleaner than real sensors. Timing is more predictable. Noise is controlled. Objects behave the way the simulator expects them to behave.
Real sensors do not care about any of that.
Simulation can tell you whether your logic works under the assumptions you gave it. It cannot always tell you whether those assumptions are realistic.
A lot of the work this summer involved moving back and forth between both environments. Reproduce an issue in simulation, test it on the vehicle, then figure out why the results are different.
That difference was usually where the interesting engineering started.
What I am taking with me
A trajectory that looks perfect on one log does not prove that the system works. It means the system worked under that particular set of conditions. The real test is figuring out what happens when those conditions change.
I also gained a much better appreciation for the difference between research software and software that needs to run reliably on an actual vehicle.
Code that generates a good result for an experiment and code that has to process sensor data continuously on a moving car have very different requirements. Both are valuable skills to learn, but treating them as the same thing can become a serious problem when software leaves the lab.
Overall, it was a summer of debugging sensors, staring at point clouds, tuning localization systems, running tests, validating results, and trying to understand why something that worked perfectly yesterday suddenly stopped working today.
Thanks to everyone at the RANCS Lab for a summer full of interesting problems.