AI Powered ETL: Data Quality Management using ETL and GenAI in AWS

Imagine the following scenario: You are part of a young, up-and-coming company that receives product data from various sources. This data is not only constantly growing in volume, but its formats and quality also present an ever-widening spectrum.

You will soon recognise the problem: product data is often an unstructured mess. Descriptions like “Screw M5 galv. 100pcs,” “T-Shirt Bl. Sz. L,” or “Smartphone 128GB black” must be converted into uniform, clean categories. The modern reflex? “Let’s just run it through an AI!” But is that really the most efficient solution? And how can you use data quality management tools in AWS to solve the problem?

Our recent practical experience clearly shows that a pure AI approach is often the digital sledgehammer method—powerful, but imprecise and expensive. In the worst-case scenario, it can even cause damage through incorrect classifications: wrong data is processed incorrectly and delivers invalid insights, which in turn serve as the basis for flawed business decisions. The better approach is the intelligent combination of classic data processing (ETL) and modern Generative AI (GenAI) into a solution that not only works but is also cost-effective, trustworthy, and strategically superior. You can also read our guide: New data platforms = The End of ETLS?

The Challenge: From Data Chaos to Strategic Advantage

The data chaos described in the opening scenario is far more than just a cosmetic issue. If you really want to tackle the problem, read the complete guide to data strategy. For a growing company, it is the central hurdle on the path to data-driven decision-making. If this problem remains unsolved, it develops into tangible disadvantages that endanger the entire success of the business:

• Flawed Business Decisions Based on False Insights: If, for example, screws are mistakenly categorised as electronics accessories, sales reports will be distorted. Management could then draw the wrong conclusions, invest in the wrong product lines, misjudge margins, or implement ineffective marketing budgets. Bad data inevitably leads to poor strategic direction.
• Operational Chaos in Logistics and Warehousing: If the system cannot distinguish between a “Tshirt Sz. L” and a “T-Shirt Size M” because the data is unclean, it leads to concrete problems in order a fulfillment. Products end up in the wrong storage location, deliveries are delayed, and process costs increase. The efficiency of the entire supply chain suffers.
• Poor Customer Experience and Lost Revenue: The problem is immediately visible to the customer. If a search for “smartphone” only displays half of the available devices because the rest are hidden under “gadgets” or “electronics,” the customer becomes frustrated and may buy from a competitor. Flawed filter and search functions are a direct consequence of poor data categorisation and cost real money.
• Manual Processes as a Brake on Growth: Initially, it might be possible for an employee to manually correct data quality. But as data volume explodes, this process quickly becomes an insurmountable bottleneck. The company cannot scale quickly because its data processes can’t keep up. Manual error correction ties up valuable resources that are urgently needed elsewhere.

The goal, therefore, is not just to “clean up,” but to create a scalable and automated process for data quality management that reliably and quickly classifies thousands of product items into predefined categories, thereby laying the foundation for a real strategic advantage.

Our Solution Strategy: A Hybrid Approach Prevails

We consciously decided against a rigid, off-the-shelf market solution and also against a pure “all-or-nothing” approach with AI. Why use an expensive, resource-intensive AI for a task that a simple rule can perform in milliseconds? Our strategy is based on a two-stage, hybrid model.

Key Learnings from Practice with GenAI

Success with GenAI is not a matter of chance, but of precise control:

• Precise Prompting is Key: The be-all and end-all for reliable results is the prompt—the instruction given to the AI. We had to design it to act like a contract: it defines the exact desired output format, the permitted categories, and the behavior in case of uncertainty. This is the only way to make the AI’s results repeatable and deterministic.
• Double Chunking for Stability: To avoid overwhelming the AI and prevent errors, we divided the data twice. First into large “chunks” for process control, and then again into “micro-chunks” for the actual AI request. This ensures the AI remains focused and the results are optimal.
• Homogeneous Data Chunks Yield Better Results: We found that the AI performs significantly better when processing a list of similar products at once (e.g., only T-shirts). This allows the model to better recognise patterns and apply a consistent logic, instead of having to switch context for every single product.

How AWS Provides the Building Blocks for a Lean Solution to Replace Manual Processes

A key advantage of our solution lies in its lean and modern architecture, which has completely replaced a previously manual, error-prone reconciliation of Excel spreadsheets. Instead of a complex and expensive infrastructure, we relied on cost-effective, serverless AWS services:

• AWS Step Functions acts as the conductor of our process, controlling the entire workflow from start to finish and ensuring each step is executed at the right time.
• AWS Lambda represents the workhorses. These small, serverless functions perform the individual tasks—both the simple keyword searches and the targeted calls to the AI.
• Amazon Bedrock is our creative AI expert, providing access to powerful language models that handle the complex categorisation work.
• Amazon S3 is more than just our central warehouse. We use it as our cost-effective data foundation. Instead of setting up an expensive relational database system (RDBMS), simple S3 tables were completely sufficient for storing and processing our data.

Of course, one could alternatively turn to an existing quality management tool from the market. But for a dynamic, growing company, this would have significant disadvantages: high license fees, rigid processes, and slow adaptation to new technologies. Our custom-built approach, however, grows with the company. We can start small and expand the solution modularly as needed. Above all, this path gives us the flexibility to react to the constant innovations in the GenAI space. We can test and integrate new models or adjust the workflow without having to wait for a vendor’s update cycle. This not only makes the solution future-proof but turns it into a real competitive advantage.

Outlook and Next Evolutionary Steps

The current solution is a strong foundation, but good quality management is never “finished.” The modern cloud and AI landscape offers exciting opportunities for further development, which we are already evaluating.

1. Introducing an AI Confidence Score A great idea is to have the AI return not just a category, but also a confidence score (e.g., from 1=very certain to 10=a guess). This allows us to intelligently steer the process: all results with a score below a defined threshold are automatically flagged for targeted human review. This focuses the valuable time of experts on the cases where it is truly needed.

2. The Reviewer Approach: Four AI Eyes See More Than Two For particularly critical data, we could implement a second, independent AI instance as a reviewer. The first AI categorises, the second validates the result. If there is a discrepancy, the case is also routed for manual clarification. This principle, often referred to in research as the “Reviewer-Critic Model” or “AI Consensus,” is a proven method for further increasing the reliability of automated processes, as studies in AI safety have repeatedly shown.

3. The Learning Loop through Active Learning The corrections made by human experts are pure gold. The next step is to establish a feedback loop that uses these corrections to continuously improve the system. This approach, known as Active Learning, transforms the process from a static pipeline into a dynamic, self-learning system. In the long term, this could even lead to automated fine-tuning of the AI model, as supported by leading cloud providers.

4. The Next Dimension: Multimodal AI for Image and Text Recognition So far, we have only analysed text data. The future, however, lies in multimodal models that can process different data types simultaneously. Imagine the AI not only reading “T-Shirt Bl. Sz. L,” but also analysing the corresponding product image. This would dramatically increase accuracy for ambiguous terms. Platforms like Amazon Bedrock are rapidly evolving in this direction, making such approaches increasingly tangible.

Conclusion

The hype around Generative AI is justified, but true business value is created through pragmatic and strategic application. Instead of discarding proven data processing methods, we should use them as a strong foundation.

The future does not lie in trying to solve everything with a single universal solution—be it a rigid software suite from the market or an undifferentiated AI approach. It lies in building intelligent systems where each component performs the task for which it is best suited. That is the essence of smart, value-creating, and future-oriented quality management.