The Agricultural Bottleneck
Manual fruit sorting in New Zealand packhouses is labor-intensive, error-prone, and fatigue-limited. Standard off-the-shelf vision models fail completely: standard COCO-pretrained weights classified a cluster of kiwifruit as “teddy bear” at 88% confidence and cannot distinguish ripe skin from fungal rot.
Detection, Measurement & Grading
- Detection: YOLOv8s detects fruit from a webcam feed.
- Measurement: OpenCV GrabCut contours measure width, height and area in centimetres, calibrated at 37 pixels/cm. For boxes without depth, frames are processed in monochrome to find exact edge points; their mean draws each line.
- Grading: the Claude Vision API grades each fruit across 10 fields, including quality, defects and days remaining.
- Backend: Flask REST API, PostgreSQL and a live dashboard (MJPEG and SSE streaming), deployed in Docker on a self-managed Ubuntu server; 1,129 scans logged.
Custom Segmentation Models
- Off-the-shelf COCO weights labelled kiwifruit a “teddy bear” at 88%, so two Mask R-CNN models were fine-tuned separately in PyTorch/Detectron2 from COCO weights (training loss 2.36 → 0.33).
- Results: 3-class fruit instance segmentation at 98–99% confidence, and a kiwifruit detector at 85–95%.
- When the two models disagreed, a bridging step re-runs the model and checks the base image and edge detection before confirming the fruit type.
Hands-Free AR Inspection (XReal One Glasses)
- Developed with Unity 2022.3 and XREAL C# SDK.
- Unity/C# Android client anchors bounding boxes onto real fruit, solving head parallax with world-space projection and finding the server by UDP discovery.
- Hands-free grading allows workers to handle and pack produce without checking external computer screens.
Tech Stack & Repositories
- Vision Pipeline: PyTorch, YOLOv8, Detectron2 (Colab T4 fine-tuned), OpenCV GrabCut.
- Backend & Database: Flask REST API, PostgreSQL 15, Gunicorn, Docker.
- Augmented Reality: Unity 2022.3, C#, XREAL NRSDK.
- GitHub Repository: ai-fruit-quality-inspection-ar