Build Log #4: One Bad Statement Can't Take Down the Batch

Someone importing years of statements across five banks is easily over a hundred files at once. None of them should be able to take the others down, and the app shouldn't freeze while they process. This entry is the background job queue that makes both true.

Simpler than the original plan, on purpose

The design called for a process pool. I shipped a single background thread with a sequential poll loop instead. A process pool on Windows means picklable work functions and a separate database connection per worker process — real complexity, in exchange for throughput headroom the app doesn't need yet. The queue and retry logic underneath are identical either way, so upgrading later is contained to one file.

Claiming a job is one conditional database update — set status to PROCESSING only where it's currently QUEUED or RETRYING — checked by how many rows it actually changed. Two workers racing for the same job can't both win it; the database itself is the lock.

1. 01Retryable failures — a worker crash, a transient read error, an OCR timeout — get up to two retries, three attempts total.
2. 02Deterministic failures — a corrupted file, a password-protected one — never retry. Trying again won't change the outcome.
3. 03A batch stays "Processing" until every one of its jobs reaches a state that can't change anymore, then becomes Completed, or Completed With Warnings if anything was excluded along the way.

One test caught a real inconsistency: a job that failed permanently on its first attempt was recorded as attempt count 0, because only the retry path had ever incremented it. Fixed so the count always reflects how many times a job actually ran, no matter which path ended it.

A batch isn't done because most of it finished. It's done when every single job in it has reached a state that can't change anymore.

Next

Bank detection and the first real parser — the point where this pipeline finally reads an actual bank statement.