One person opened five tabs.

Each tab had the same transaction report open. They clicked Export in all of them, one after another, trying to download the files in parallel.

Then the service started returning 503.

Not long after that, one pod died.

The strange part was the size of the trigger. It was not a huge campaign, not a sudden flood of traffic, and not every endpoint getting hit at once.

It was one user, five tabs, and five export downloads started close together.

The export path came from an older part of the product. It still worked, so it had stayed around. A user clicked Export, the request stayed open, the service collected the matching rows, built a CSV, and streamed the file back to the browser.

Everyone knew it was heavy. It showed up on CPU graphs, and users had to wait for the browser download to start. But for a while, it sat in that familiar place where an uncomfortable system keeps surviving because there is always something more urgent to fix.

The culprit

The problem was not the Export button. It was where the work happened.

The service was doing the expensive part inside the HTTP request.

rows := loadAllMatchingRows(filters)
file := buildCsv(rows)
stream(response, file)

That shape was fragile. One request could load a large result set, hold memory longer than needed, spend CPU building the file, and keep the connection open until the whole thing finished.

One export was already unpleasant. Five exports from those tabs made the same pod repeat that work at the same time. CPU climbed, memory pressure followed, and the pod ran out of room while those requests were still alive.

At first it was tempting to talk about query tuning, but the failure was bigger than one slow query. A faster query would only delay the next crash. The request path itself had become the place where heavy report generation happened, so every click turned into direct pressure on the web service.

Challenges

We could not remove the export or make it stale.

The ops team used this report every day. They needed fresh transaction data for reporting, reconciliation, and daily follow-up. When something had to be checked in the morning, waiting for a manual database pull was not a real option.

That made the fix more constrained. The report still had to include current data. It still had to be reliable enough for people outside engineering to depend on it. And from the user's side, Export still needed to feel like a normal product feature, not a special operation that required asking someone to run a script.

There was also a production concern. Exports tend to happen in clusters because teams work on the same reporting rhythm. If one person exports at the end of the day, a few others may do the same. The system had to survive that pattern without pretending users would naturally spread their clicks apart.

The design

The design came from a simple algorithm idea I learned in college, the divide and conquer.

Do not process one big export as one big request. Split it into smaller pieces, process those pieces with a clear limit, then merge the results at the end.

The API only creates an export job and returns a jobId. After that, workers handle the CSV generation in the background. Users can see the job progress and download the file when it is ready.

Kafka helped move the work out of the request path, but the queue was not the main fix by itself. If we had put one giant export into one Kafka message, one worker would still carry the whole report. The important part was splitting the export into bounded batches.

Each batch owned one window of data. A worker read that window, wrote one CSV part, and uploaded it. When all parts were done, the final step merged them into one file in storage.

The merge needed one guard. Several batches could finish close together, so more than one worker might notice that the job looked complete. We used the job row to make finalization happen once.

Only the worker that updated the row could trigger the merge. The others stopped because finalization had already been claimed.

That was enough detail for the system to behave well without making every part clever. The request stayed short. The workers handled bounded work. The database kept job progress visible. Storage handled the final file.

Takeaways

The visible product change was small. Users clicked Export, waited for the job to finish, and downloaded the same report.

The operational change was much larger. The web service stopped building full CSV files inside open HTTP requests. Memory became tied to batch size instead of the user's selected date range. Concurrent exports became queued background work instead of five heavy request handlers running side by side.

The useful part of the fix was not that we added Kafka, or GCS, or a new job table. Those were only pieces of the design. The real change was moving the expensive work to the right boundary and making the large task small enough to control.

After that, five people clicking Export at the same time stopped being a production incident.