How I bypassed Vercel Serverless timeouts to build a decoupled document ingestion pipeline

The article explains how document ingestion and embedding jobs were moved off Vercel serverless routes into a queued worker pipeline to avoid timeout limits.

Next.js routes handle validation and job enqueueing, while BullMQ and Redis coordinate background processing through a persistent Railway worker.

The worker streams files from Cloudflare R2, chunks text, generates embeddings, and uses Postgres locking to safely update quotas and ensure idempotency.

A stateless pass-through mode can also return embeddings via webhook without storing data, making the system useful for RAG and other async AI workflows.