WORLDTECHProperty Finder found 96.8% of its vision-AI labels came from its incumbent model
Property Finder, a digital real estate application, found that 96.8 percent of its evaluation labels had been carried over from a single model's predictions instead of being checked by human experts. The engineering team uncovered this while moving its vision AI workload to a more cost-effective option on Amazon Bedrock.
Labels mirrored the incumbent model
The labels had been produced by the model already in use, so its ground truth tilted toward that model and made every alternative look weaker than it might be. The labels also contained errors, but the skew was the real obstacle to an objective comparison.
The problem lay in the evaluation method rather than in the search for a better model. Under that method, each alternative was judged against labels that favoured the model already deployed.
Scale of the image workload
The company processes tens of thousands of property images each day across the Middle East and North Africa. Its system scores 11 quality attributes for every image, and the load amounts to thousands of new listings with an average of 15 images each.
Property Finder needed a cheaper option for this workload, but it could not judge alternatives fairly while the labels favoured the incumbent.
Rebuilding the evaluation loop
The Property Finder team and the AWS Generative AI Innovation Center ran a prompt optimization loop on Amazon Bedrock to raise classification accuracy. The model returns predicted attributes as JSON for each property image, and those predictions are scored with Micro F1 against a development set of 256 expert-annotated images.
Error analysis of failed images shapes each new prompt version, and the loop ran from v0 to v27. AI assistants helped refine the prompts, and later versions relied on automated optimization. According to the organisation, the approach yielded a more accurate and cost-effective model, cutting cost by 86 percent across the 11 image-quality attributes.
Open questions
- How large was the accuracy gap between the incumbent and the selected model?
- Will the rebuilt ground truth be applied to other Property Finder workloads?