This called for an AI-native approach built around visual language models. The bigger idea borrowed from lean manufacturing's principle of gemba — go to where the work is actually performed. In software, that means getting close to the metal: instead of Nomad Go doing the data collection and model training centrally, give the customer, who is already standing in front of the product, the tools to solve it themselves.

I chose this direction because it was a chance to learn firsthand what visual language models were actually capable of. My first attempt tried to get a VLM to both locate and identify the object in a single pass — that failed: inference was too slow, and the model was only mediocre at picking out the product's region of interest in the image.

Good software hands the end user the tools to solve their own problem — that's the belief that got me into this field in the first place.

Barcode information was pulled from up to 4 different sources. Images were downloaded and returned data was pruned. Then a prompt was engineered to call the OpenAI API with the images as context. Structured data with all properties including dimensions was returned.

Success was measured against a set of "ground truth" examples for known products. That evaluation data was then used to fine-tune a vision language model, increasing accuracy by 30%.

Images were vectorized using Apple's Vision framework and cosine similarity used to compare in real time.

Evaluation was done using unit tests and real world analysis. Techniques to normalize the images were used to eliminate false positives.

I built the end-to-end pipeline — barcode and image ingestion, the OpenAI-powered structured data extraction, and the Visual RAG matching system. I also brought in help where it mattered: peers code-reviewed the asynchronous code, and a teammate built the generic ROI models for form factors that weren't off-the-shelf.

The result: customer onboarding dropped from weeks to immediate.

The biggest lesson was around team dynamics. I built this in an isolated codebase instead of inside the main product's sprint, and by the time it was ready, it wasn't in a state that could merge into the main codebase in time.

I also should have worked more closely with sales, showing customers the upcoming solution earlier — both to keep them engaged and to validate the workflows against real use cases.