Building a large-scale RFM data pipeline on Amazon EKS Spot with Config | Amazon Web Services

Config rebuilt its large-scale robot data preprocessing pipeline on Amazon EKS, replacing SQS and Lambda with RabbitMQ, KEDA, Karpenter, and EC2 Spot Instances.

The new architecture adds job-specific queues, autoscaling, and separate CPU/GPU Spot worker pools to handle action data more efficiently.

Config says the redesign removed key bottlenecks and improved visibility while avoiding Lambda limits and high on-demand compute costs.

According to the article, processing costs fell by 70% to 90%, and end-to-end turnaround dropped from days to hours.