.png){kind=logo}
Validating an AI-driven workflow processing 13,000 cases a year
Introduction
We designed and delivered the end-to-end systems integration testing strategy for a major AI-driven document intelligence platform at a leading insurer. We ensured the new solution could process high volumes accurately and reliably before go-live.
The challenge
The organisation was transitioning from a legacy document processing workflow to a new AI-driven solution. The legacy process handled around 13,000 submissions annually, with multiple manual handoffs and an average processing time of 45 to 70 minutes per case. The new solution covered ingestion, classification, OCR extraction, evaluation, deduplication, and decision-making. It involved multiple interconnected components across teams and technologies, but lacked end-to-end visibility and a defined QA approach. Limited ability to customise test data meant testing had to rely on real-world datasets, adding complexity to scenario design. Resolving this was critical to ensure data accuracy, reduce manual processing effort, and support faster underwriting decisions.
The solution
We took ownership of designing a structured end-to-end SIT testing strategy covering the full workflow. A comprehensive test suite was created in Excel and Azure Test Plan. Test cases were organised by component and by cross-component scenarios to reflect real business flows. We leveraged business-provided datasets of 50-plus emails to simulate real-world conditions. We collaborated closely with developers, business analysts, and the test lead to align expected outputs with business rules and refine the evaluation testing approach for LLM-based validation. We proactively identified gaps, ambiguities, and edge cases. Complex AI workflows were translated into clear, testable scenarios, improving shared understanding across QA, engineering, and business teams.
The benefit delivered
The structured test suite covered 50-plus real-world documents across multiple document classes, substantially improving test coverage across the full pipeline. Integration between all components was validated rather than tested in isolation, increasing confidence in end-to-end processing. Key risks were identified early, cross-team alignment was improved, and potential downstream rework was reduced. Clear test scenarios, expected results, and traceability enabled smoother SIT readiness. The testing strategy supported validation of a workflow handling around 13,000 submissions annually. It contributed to a validated transition from 45 to 70 minutes per case to an AI-assisted workflow of 10 to 15 minutes.