Pollinator
How can we replace bees?
The summer after grade 11, I was a research assistant at the University of Windsor's AI Robotics Centre. I was busing 40 minutes to the lab every morning. Completely worth it.
The prof overseeing me basically said: we have a robot arm and a depth camera. Combine the two and make it act like it's pollinating these flowers. So the camera scans for flowers, the robot finds a path through the ones it saw, and then it executes.
I fine-tuned YOLO and Faster R-CNN models to detect blossoms, tested them, and compared them. The training images were mostly sunlit and showed flowers at a different stage from the ones in the lab. I used data augmentation to vary the lighting.
The arm was from Universal Robots, with an Intel RealSense D435i depth camera mounted on it. I controlled both from Python, using UR's RTDE interface and Intel's SDK. I also 3D modeled and printed the tool mount. That's the yellow thing at the end of the arm: it holds the camera and a motor-driven brush that vibrates against the flowers. The brush motor was controlled separately from the arm routine.
The depth camera can estimate where a flower is in 3D. But that's relative to the camera. Where is the camera relative to the robot? Every time the arm moves, the camera moves with it. I used an SE(3) transform to account for the camera's offset from the tool and the arm's position and rotation. That rotation and translation turn "the camera sees a flower here" into a point the robot can actually move to.
When the arm went to the wrong place, the final movement didn't tell me where the problem was. I split the pipeline into smaller sections I could test separately, instead of trying to diagnose everything by watching the arm.
Getting closer didn't necessarily help. The depth camera became less reliable up close, and some readings were missing entirely. I took readings from several frames and viewpoints. Looking from another angle also gave the detector a chance to see flowers hidden by leaves. I grouped repeated detections with DBSCAN, so that seeing the same flower from different angles didn't turn it into several targets.
Once I had those targets in the robot's coordinates, I could plan the visit order. A nearest-neighbor pass gave an initial route, then 2-opt shortened it by swapping sections. The arm could work through the flowers it had found.
The whole thing ran on a Jetson Orin Nano. I'd been developing on my Windows laptop, so moving it to Linux meant reworking a bunch of stuff. I also wrote guides for controlling the arm in Python, using the RealSense SDK, and getting started with YOLO.
I later refactored the robot-control code for an xArm. We tested and iterated on the physical arm and got the port working near the end of my time on the project.
This was my first big taste of robotics, and it made me want to pursue it. I wonder if a vision-language-action model could make the same job much smoother today. I'd like to try that.