Shared Vector Infrastructure
The process that keeps up to date the product information more than ten AI systems rely on: if it goes stale, they stop finding what is new.
The problem
Several portal apps depend on one shared store of visual product information to find similar products and group them. If it is not refreshed, agents stop finding new products: visual search ages silently, without ever failing.
What I did
A batch job that reads recently-sold products from SQL Server, generates each visual embedding with CLIP and upserts it incrementally into the vector store. It maintains two collections: the 512-dimension one every similarity app consumes, and a 256-dimension PCA-reduced version for the case that needs lighter vectors. Orchestration runs on EasyMorph, the same ETL engine behind the dashboard pipeline: classic data engineering feeding the AI layer.
The outcome
In production as shared infrastructure: more than ten agents read what this job writes. It is the piece that lets the portal visual search know about new products.
How it keeps the store fresh
- Recently sold productsInput: from SQL Server
- Visual embedding with CLIPOne vector per photo
- Incremental upsertOnly what changed
- PCA reductionA lighter version
- Orchestration with EasyMorphThe area's ETL
- Store read by 10+ agentsOutput
This is invisible infrastructure: when it works nobody notices, and when it fails nothing breaks — visual search simply stops knowing about new products. Hence a batch, incremental job on the same ETL orchestrator the rest of the area uses.
Stack
- Python
- CLIP
- PyTorch
- Qdrant
- scikit-learn (PCA)
- SQL Server
- pyodbc
- EasyMorph
Blue Star Group internal project — the code is not public.
Next project: Product Photo Standardizer →