Proof of Delivery Vision

Vision for proof of delivery

Every drop off photo runs through a detection model that finds the packages and backs a clean proof of delivery. It is tuned for recall, so we would rather double check a real package than miss one.

The pivot off YOLO

The original model was built on YOLO. When the license changed, keeping it would have meant open sourcing our model weights. That was a non starter, so we rebuilt the core on a stack we fully own.

A recall-first Mask R-CNN

The new model is a Mask R-CNN with a ResNet-50-FPN backbone.

• Trained the head first with a frozen backbone, then fine tuned the top layers.
• Heavy augmentation (mosaic, safe bounding box crops, blur) for generalization.
• Returns boxes, confidence, and segmentation masks in encoded RLE.
• Beat the old model on both recall and precision, and recovered a large share of packages the old one missed entirely.

Labeling at scale: Label Studio + SAM2

I stood up Label Studio as our labeling platform. It outgrew this project fast and is now used by multiple teams across multiple labeling efforts.

• Wired SAM2 into Label Studio for assisted annotation. Drop a point or a box and SAM2 fills in the mask.
• The same SAM2 hookup doubles as a way to review and evaluate model predictions during labeling.

Evaluation, rethought

Stock AP and IoU punish you when packages are stacked or overlapping, which is exactly our hard case. So we built metrics that fit it.

• Modified mAP and recall that do not penalize sensible predictions around naturally clustered items.
• Mean Overlap Percentage to score how well predictions cover real packages.
• Detection Rate and True Negative Rate to track misses and false alarms head on.

Why it matters

Better, owned, and easier to label for. The model catches more real packages, the stack carries no licensing strings, and the labeling pipeline makes the next model faster to build.