For medium and large e-commerce businesses, integrating supplier data is essential. However, as soon as your store starts working with dozens of wholesale suppliers, each providing an XML or JSON feed containing tens of thousands of products, variants, prices, and stock levels, you hit a hard technical wall.
A traditional approach, such as a simple cron job that downloads XML files nightly, loads them into memory, iterates through products, and runs thousands of SQL queries against the database, is a ticking time bomb. With large file sizes, servers overload, RAM allocation exceeds limits, CPU utilization spikes sharply, and the database connection pool is exhausted. This leads to row-level or table-level database locking, leaving real customers on your site staring at a 504 Gateway Timeout error instead of a fast page load.
At the technology studio nolimeo, we help companies replace these fragile, unstable integrations with a robust, asynchronous architecture. In this engineering analysis, we will look at why traditional synchronization systems fail, how to work with Node.js streams without exhausting system memory, and how to build a high-throughput processing pipeline using Redis-backed BullMQ queues and PostgreSQL (via Drizzle ORM).
1. Decoupled and Asynchronous Processing Architecture
The fundamental rule for processing large volumes of data is complete decoupling from the main application thread. The primary e-commerce server handling customer requests on the frontend (Next.js, Medusa.js) must never be bogged down by computationally intensive XML parsing and synchronous database writes.
Our solution shifts to a distributed and asynchronous pipeline, where file parsing and database insertion run in separate background processes (workers) governed by a robust queue manager.
[ XML Feed URL (Supplier) ]
│
▼ (HTTP Stream - Fetch API)
[ Node.js Readable Stream ]
│
▼ (Pipeline / Pipe)
[ XML Stream Parser (Limited RAM) ] ───► (Backpressure Mechanism)
│
▼ (Batching by 100 items)
[ Redis / BullMQ Queue ]
│
┌────────┴────────┐
▼ (Worker 1) ▼ (Worker 2) ───► (Controlled Concurrency)
[ PostgreSQL DB (Drizzle UPSERT) ]
Why This Architecture Works
- Prevents Memory Outages: The XML file is never loaded into memory as a whole. It is streamed in small chunks directly from the HTTP connection to the parser. The memory footprint remains stable at a few dozen megabytes, regardless of whether the feed is 10 MB or 10 GB.
- Eliminates Database Spikes: Downloading and database writing are not synchronously linked. If the database cannot write fast enough, the queue (BullMQ/Redis) acts as a buffer. Products are pushed into the queue quickly, and workers commit them to the database at a controlled pace.
- Type Safety and Validation: Before insertion, every product from the feed is passed through a strict schema validation (via Zod). This filters out invalid pricing, missing EAN codes, or corrupted data without crashing the entire import process.
2. Why fs.readFile() Fails: Memory Physics and Backpressure
If you use Node.js's fs.readFile() or download the entire feed into memory as a single giant string, the V8 engine allocates a massive amount of RAM. An XML file of 500 MB can consume 3 GB to 4 GB of RAM once parsed into a memory object (DOM tree). This happens because every XML element, attribute, and text node is represented in JavaScript as a complex object with its own metadata and internal methods.
The solution is streaming and parsing on the fly (event-driven / SAX parsing). In this mode, we read the file in small fragments and trigger events only when navigating through specific XML nodes.
The Role of Backpressure
When streaming data, we run into a speed mismatch: reading data from the network or disk is generally much faster than writing data into a Redis queue or database. If we read data without limits, the memory buffer overflows, causing the application to crash due to OOM (Out of Memory) errors.
Node.js pipelines implement automatic backpressure handling. If the write stream signals that its internal buffer is full (by returning false on a write call), the read stream is temporarily paused until the write buffer drains.
3. Technical Implementation in TypeScript: Stream, Queue & Worker
Below is a production-ready example of our TypeScript code for Node.js. This script streams the XML feed directly from the HTTP connection, filters products asynchronously using a stream parser, validates their structure via Zod to protect database integrity, and forwards them in batches to a BullMQ queue for worker processing.
3.1. Stream Parser and Populating the BullMQ Queue
To process the XML stream efficiently, we will use the fast-xml-parser package combined with native Node.js streams.
// src/services/feed-processor.service.ts
import { Readable, pipeline } from "stream";
import { promisify } from "util";
import { XMLParser } from "fast-xml-parser";
import { Queue } from "bullmq";
import IORedis from "ioredis";
import { z } from "zod";
const streamPipeline = promisify(pipeline);
// Redis connection for BullMQ queue
const redisConnection = new IORedis(process.env.REDIS_URL || "redis://127.0.0.1:6379");
const productQueue = new Queue("product-import-queue", { connection: redisConnection });
// Strict schema for validation of products from feed
export const ProductFeedSchema = z.object({
sku: z.string().min(3),
name: z.string().min(1),
price: z.preprocess((val) => Number(val), z.number().positive()),
stock: z.preprocess((val) => Number(val), z.number().int().nonnegative()),
ean: z.string().optional(),
});
export type ValidatedProduct = z.infer<typeof ProductFeedSchema>;
/**
* Main service to download and memory-efficiently parse the XML feed
*/
export async function downloadAndParseFeed(feedUrl: string): Promise<void> {
const response = await fetch(feedUrl);
if (!response.ok || !response.body) {
throw new Error(`Failed to download feed from supplier server: ${response.statusText}`);
}
// Convert Fetch ReadableStream to Node.js Readable stream
const reader = Readable.from(response.body as any);
// Configure fast-xml-parser for async stream parsing
const parser = new XMLParser({
ignoreAttributes: false,
attributeNamePrefix: "",
});
let productBuffer: ValidatedProduct[] = [];
const BATCH_SIZE = 100;
// Create custom transform stream for product extraction and validation
const filterTransform = new Readable({
objectMode: true,
read() {} // Required by the stream interface
});
// Event-driven data processing directly from flow
let leftoverData = "";
reader.on("data", async (chunk: Buffer) => {
const data = leftoverData + chunk.toString("utf-8");
const matches = data.matchAll(/<product>([\s\S]*?)<\/product>/g);
let lastIndex = 0;
for (const match of matches) {
if (match.index !== undefined) {
try {
const parsed = parser.parse(`<product>${match[1]}</product>`);
const rawProduct = parsed.product;
// Validate type using Zod to protect database from supplier errors
const validated = ProductFeedSchema.safeParse({
sku: rawProduct.code || rawProduct.sku,
name: rawProduct.name || rawProduct.title,
price: rawProduct.price,
stock: rawProduct.stock_quantity || rawProduct.stock,
ean: rawProduct.ean,
});
if (validated.success) {
productBuffer.push(validated.data);
// Filling queue in optimized batches
if (productBuffer.length >= BATCH_SIZE) {
const batchToQueue = [...productBuffer];
productBuffer = [];
await productQueue.add("import-products-batch", { products: batchToQueue }, {
removeOnComplete: true,
attempts: 3,
backoff: { type: "exponential", delay: 2000 }
});
}
}
} catch (parseError) {
console.warn("XML block parsing error:", parseError);
}
lastIndex = match.index + match[0].length;
}
}
leftoverData = data.substring(lastIndex);
});
reader.on("end", async () => {
// Send remaining products from buffer
if (productBuffer.length > 0) {
await productQueue.add("import-products-batch", { products: productBuffer }, {
removeOnComplete: true,
attempts: 3,
});
}
console.log("XML stream successfully read and partitioned into queue.");
});
await streamPipeline(reader, new Readable({ read() {} }));
}
3.2. Asynchronous BullMQ Worker and PostgreSQL Database Writes (via Drizzle ORM)
The worker runs as a standalone background microservice, fetching batches from the queue and executing an optimized, atomic bulk UPSERT in PostgreSQL. This minimizes individual write overheads and prevents database starvation.
// src/workers/import.worker.ts
import { Worker, Job } from "bullmq";
import IORedis from "ioredis";
import { db } from "@/db/drizzle"; // Drizzle DB client
import { products } from "@/db/schema"; // Drizzle schema
import { sql } from "drizzle-orm";
import { ValidatedProduct } from "../services/feed-processor.service";
const redisConnection = new IORedis(process.env.REDIS_URL || "redis://127.0.0.1:6379");
/**
* Bulk insert products into database using Drizzle ORM UPSERT syntax
*/
async function processBatchToDatabase(productsBatch: ValidatedProduct[]) {
// Optimized batch write for minimized I/O operations
await db.insert(products).values(
productsBatch.map((p) => ({
sku: p.sku,
name: p.name,
price: sql`${p.price}::numeric`,
stock: p.stock,
ean: p.ean || null,
updatedAt: new Date(),
}))
).onConflictDoUpdate({
target: products.sku,
set: {
name: sql`EXCLUDED.name`,
price: sql`EXCLUDED.price`,
stock: sql`EXCLUDED.stock`,
ean: sql`EXCLUDED.ean`,
updatedAt: new Date(),
}
});
}
// Initialize asynchronous worker with strict concurrency control
const importWorker = new Worker(
"product-import-queue",
async (job: Job<{ products: ValidatedProduct[] }>) => {
const { products } = job.data;
if (!products || products.length === 0) {
return;
}
try {
await processBatchToDatabase(products);
console.log(`Successfully processed product batch. Job ID: ${job.id}, Batch size: ${products.length}`);
} catch (dbError: any) {
console.error(`Database error during batch write (Job: ${job.id}):`, dbError.message);
// Throw the error so BullMQ marks the job as failed and triggers retry logic
throw dbError;
}
},
{
connection: redisConnection,
concurrency: 5, // Max 5 parallel database writes per worker
limiter: {
max: 100, // Rate limiting to prevent database overload
duration: 1000,
}
}
);
importWorker.on("failed", (job, err) => {
console.error(`Job failed after all attempts. ID: ${job?.id}. Reason: ${err.message}`);
// Implement logging to monitoring tools or send alerts here
});
4. Handling Edge Cases and Failures in Network Imports
Experienced developers know that third-party integrations fail regularly. If your pipeline doesn't define exactly what to do during these failures, your system will be unstable.
Here is how we asynchronously address three common critical scenarios at nolimeo:
4.1. Supplier Server Drops Connection or Returns 504 Timeout Mid-Stream
XML feeds are often dynamically generated on the supplier's side. If the supplier's server is overloaded, the connection may drop mid-stream, leaving the parser with incomplete, corrupted XML code.
- Solution: We never overwrite production data directly during a live network download. Instead, we download the XML feed into a temporary storage file on the server (
/tmp/import_feed.xml). Only after the download completes successfully without network errors do we initiate the parsing process from the local disk. This helps ensure that an incomplete download does not corrupt the database or accidentally delete missing items.
4.2. Invalid XML Data and Corrupted Types (Broken Feed)
A supplier might change their price formats (e.g., using commas instead of decimal points) or send malformed XML syntax that crashes standard parsers.
- Solution: Dead Letter Queues (DLQ). BullMQ has built-in automatic retries with exponential backoff. If a batch fails because of a temporarily locked database, the worker retries in 2, 4, and 8 seconds. However, if a batch fails because of invalid data schema (Zod validation throws an exception), the job is not repeated indefinitely. It is automatically moved to a Dead Letter Queue (DLQ) to alert developers, while most valid products continue importing without interruption.
4.3. Database Row/Table Locking
If you run too many parallel workers trying to execute UPSERT commands on the same products table, PostgreSQL will begin throwing Deadlock detected exceptions. Transactions block each other while waiting for row locks.
- Solution: Controlled Concurrency. In the BullMQ worker configuration, we strictly set
concurrency: 5(or lower, depending on the database hardware). Even though we have the CPU capacity to run hundreds of processes, this asynchronous choke valve ensures the database receives only as many writes as it can handle safely without locking tables or degrading the e-shop's customer-facing performance.
5. Why You Should Avoid Unstable No-Code Platforms and Templates
Many companies try to automate these heavy processes using click-and-configure tools like Zapier or Make.com. While this might suffice for small shops, it is a dead end for medium and large B2B platforms.
- Prohibitive Scaling Costs: Downloading tens of thousands of products daily consumes millions of operations. Subscription fees for no-code platforms quickly scale to thousands of dollars per month, far exceeding the hosting cost of a dedicated Redis instance.
- No Backpressure Support: These platforms lack backpressure control. Once you feed them a large file, they either crash due to memory exhaustion or overwhelm your target API with thousands of parallel requests, taking down your e-shop.
- Lack of Transaction Control: When an error occurs mid-transfer, you cannot roll back the database transaction or isolate only the broken rows, leaving the database in an inconsistent state where some prices are updated and others are not.
At nolimeo, we build exclusively on type-safe solutions with clean, custom code. Your company owns the code, pays no licensing fees, and keeps direct control over every data pipeline and network socket.
Conclusion: Gain Control Over Your Data Flows
XML feed processing is not about how fast you can read a file, but about how safely and intelligently you transfer data from the supplier's server to your database without impacting customer search and browse experiences.
If your e-shop suffers nightly slowdowns during catalog synchronization, if your developers are constantly patching legacy cron scripts, or if you are paying massive monthly bills for unstable no-code integrations, it is time to transition to a modern, asynchronous architecture.
At nolimeo, we are a specialized boutique team of senior software engineers. Our clients deal directly with the engineers working on the project, and we never outsource complex tasks to inexperienced juniors. We focus on clean, high-performance code, fast response times, and stable ERP integrations that reduce technical debt and support your business as it scales.
Want to rebuild your supplier integrations, stabilize your accounting syncs, and keep the e-shop responsive during peak traffic? Contact us and we’ll review your feed pipeline, sync risks, and the safest technical direction.