System design: a scraping jobs service

Step 4 of 36

Functional requirements

Requirements are kept as a file in the service repository, like the glossary in the DDD chapter. A wiki drifts away from what the service does over time. A file next to the code is edited in the same commit, and review shows which requirement changed along with the code.

The table on the right is that file. The step's rows are under the text; move them into the table. There is one document for the whole write-up: numbers and new properties will be added in the SLO and estimate steps.

A functional requirement describes an action of the system. Every action has a caller, so the table has a column for it. If there is no caller, it is not a requirement but an early decision about the design.

Rows are numbered so they can be referenced: "FR-5 requires a worker that can stop midway" is shorter than a retelling.

The queue, the pool, and retries are not in the table. They are ways of implementing the system, even though people often try to list them as features first.

Step 5 of 36

Out of scope

What the system does not do should be written down as explicitly as what it does. In six months someone will ask why there are no schedules, and the answer should be in the document.

• Solving captchas and bypassing anti-bot protection. Changes both the technical and the legal side of the task. If it is needed, it is a separate product with its own risks.
• Billing for completed jobs. A separate context with its own vocabulary. Including it here means designing two services at once.
• Schedules. Jobs are submitted from outside for now. Running on a timer needs a separate scheduler with its own state, missed runs, and a rule for when the previous run is still going.
• Parsing responses into structured data. We return what the target returned. Turning HTML into a table is a separate task with its own release cycle, and collection should not depend on it.

On the right are ten statements from the same conversation. Sort them by kind. Some properties sound like features, and these are the ones people usually argue about at review.

Step 6 of 36

Non-functional requirements

The second section is properties. A non-functional requirement adds no actions to the system. It sets a guarantee for the actions that already exist.

These requirements are easiest to check through consequences, so under each property we write what breaks without it. If there is nothing to write, it is a wish, and it is better to remove it right away.

The "Target" column is empty for now. "Intake answers fast" is a property; "within 200 ms, 99 times out of 100" is a promise. Promises come in the next step and go into this column: NFR-2 gets a number, and its wording stays the same.

Step 7 of 36

The numbers we commit to

A property without a number cannot be checked: whether intake is fast can be argued forever. An SLO sets the share of requests, the threshold, and the window they are counted over.

• Intake availability, 99.9% over 30 days. That is 43 minutes a month when jobs are not accepted. A hundred per cent would cost far more than this task is worth.
• Intake answers within 200 ms, 99 times out of 100. We measure the upper bound, not the average: the average hides the slow answers that make clients retry.
• 95% of jobs start within the first minute. This is a promise about the queue. It breaks when the flow is larger than the pool can handle.
• 99% of accepted jobs reach a final state without a human. Final states are "done" and "failed for good". A job stuck in "running" forever is worse than a failed one.

Decisions are made based on these numbers. If the first one is broken, we scale intake. If the third one is broken, we grow the pool or split the queue.

Step 8 of 36

Estimate: jobs per second

Users and their habits set the job flow. The flow and the job duration set concurrency, and concurrency sets the size of the pool. The calculator is on the right: try your own values.

50,000 users with 8 jobs a day give 400,000 jobs a day, under five a second on average. The load is uneven: more people come in the morning and evening, and a sixfold peak gives 28 jobs a second.

The number of jobs running at once follows Little's law: the flow multiplied by the time one job takes. 28 jobs a second at 20 seconds each is 556 jobs at once. The pool size depends on this number, not on the number of users.

Writes are not the whole flow. People ask about a job, and they start asking the minute it is accepted. Five seconds in the queue plus twenty of running, polled every five seconds, is five status requests and one more for the result. Six reads per write: that is 167 reads a second at peak against 28 writes. The pool is sized by writes, but the database lives under reads, and the second number gets lost more often than the first. What to do about it is covered in the step on caching.

Waiting is a slider here rather than a result: Little's law does not say what it will be. What the law does is turn the wait into jobs — 139 of them queued at peak — and let NFR-8 be checked. While the average wait is five seconds, the promise that "95% of jobs start within the first minute" holds; move the slider towards a minute and it breaks, and the read flow grows with it.

The last row of the table answers what a queue is for at all. It computes a different case: a pool sized for the average flow, where five minutes of peak pile up nearly seven thousand jobs. The gap between average and peak does not go anywhere, it accumulates. A pool sized for the peak does not accumulate it, but costs more and idles most of the day.

Try changing the duration. At two seconds a pool of fifty jobs and one machine are enough. At thirty, over eight hundred jobs run at once: that needs a cluster, autoscaling, and a separate budget. The task is the same; only the speed of the target API changed. That is why the first question to product was what exactly we scrape.

Step 9 of 36

Estimate: how many workers

Concurrency is calculated, but one worker handles more than one job. A scraper spends most of its time waiting on the network, and one process holds dozens of open requests. Counting one job per worker would mean twenty times more machines.

At 20 jobs per worker, 556 jobs at once need 28 workers. How many jobs one worker actually holds has to be measured. The limit can be memory (the target's whole response is kept in it), file descriptors, outbound connections, or limits in the client library.

Two architecture requirements follow from this.

The pool scales on queue depth and the age of the oldest job. A scraper's CPU load is low both in normal operation and during an incident, so autoscaling on CPU notices nothing.

A worker can crash at any moment. A machine with 20 jobs in progress leaves 20 jobs that need to go back to the queue. How to do that is covered in the lifecycle step.

Step 10 of 36

Estimate: how much data

A result does not fit in a database row. 200 KB per job at 400,000 jobs a day is 76 GB a day and over 2 TB for a month of retention. That volume cannot live in the jobs table.

This estimate leads to three decisions.

Results go to object storage. It is cheap, practically unlimited, and does not care about response size. The database keeps metadata: id, state, link, size, timestamps. Listing jobs does not read the data itself.

Retention is a requirement. "Kept for a month" means something deletes old data. Otherwise the volume grows linearly, and the storage bill becomes the first limit.

Result size is an input. In the calculator one step up, change it from 200 KB to 20 MB: monthly volume grows from 2 TB to 200 TB, and choosing storage becomes a budget question.

Step 11 of 36

C1: the system and everything that talks to it

Now we can draw. The first frame is C1: the system as one block.

On the left are the users: a human in a UI and a script. On the right are the target APIs. We do not control them: they can be slow or down, and they rate-limit us. At the bottom is the webhook for those who asked to be notified.

The frame shows the system boundary. The target APIs are outside it, even though the task is about them: their availability is not part of our SLO, they set the limits, and the response format is theirs. We can only treat them as an unreliable external environment.

This splits our promises. "The job is accepted" we guarantee ourselves. "There will be data" only holds if the target answers.

Step 12 of 36

The way in: gateway

The first element inside the boundary is the entry point, api-gateway. All external requests come here: both "submit a job" and "what happened to it".

The gateway checks who is calling, drops excess traffic, and passes the request on. It is separate because authentication and per-client limits are needed by both the write side and the read side. If they were implemented twice, the rules would drift apart over time.

• New on the diagramapi-gatewayThe way in: checks who is calling, holds the shared limits and hands the request inwards. Knows nothing about jobs.

Step 13 of 36

The write side

Behind the gateway is job-command, the only service that creates jobs, and the database jobs-pg where they are stored.

Writes and reads are split following CQRS: the command side creates and changes jobs, the read side only answers questions about them. The two sides have different loads. A job is created once and asked about dozens of times: a human refreshes a list, a script polls its thousand jobs. Scaling both sides as one unit means sizing everything for reads, and writes get far more capacity than they need. More on the split in the CQRS chapter.

The command service validates parameters, checks the idempotency key, and writes the job.

There is no queue in the frame yet: first the write, then how the rest of the system learns about it.

• New on the diagramjob-commandThe only thing that creates a job. Never calls a target API.
• New on the diagramjobs-pgpostgresStores jobs: whose, in what state, how many attempts.

Step 14 of 36

How the queue learns about a job

The simple option — write the job to the database and send a message to the queue right away — is unreliable. If the write succeeds and the publish fails, the job exists but nobody will run it. If the message is sent and the transaction rolls back, the queue points to a job that does not exist. One transaction cannot cover both the database and the broker.

So the command service writes only to the database, and a separate publisher, job-publisher, sends the event to the bus. A separate table is not needed for this: the job is written with an empty published_at column. Two elements appear on the diagram: the publisher and the bus. How it works step by step comes next.

• New on the diagramjob-publisherFinds rows in jobs with an empty publish mark, sends the event to the bus, and sets the timestamp. May send again — delivery is at-least-once.
• New on the diagramMQWork nobody has taken yet and events about finished work. It is inside the boundary because we run the bus ourselves.

Step 15 of 36

Inside intake and publishing

Intake is one transaction with one write: INSERT INTO jobs with an empty published_at. After COMMIT, the row either exists together with its "to publish" mark, or it does not exist. A second table is not needed: the job state and the publish mark live in the same row.

The publisher runs as a separate loop: it selects rows with an empty published_at, sends the event to the bus, and sets the timestamp. The bus is not part of the transaction, so there is a window between sending and marking. If the publisher crashes in that window, after a restart it finds the same row and sends the event again. At-least-once delivery means a worker can receive the same job twice, so taking a job goes through a status in the database.

The completion event goes the same way: the worker sets the status to "done" together with an empty finish_published_at, and the publisher sends the event. The worker itself does not write to the bus.

On the right are three scenarios: everything succeeds, the service crashes before COMMIT, the publisher crashes after sending. Compare what is left in the table and in the queue.

Step 16 of 36

The worker pool

The worker pool, scrape-worker, does the work: it reads the queue, takes a job, and calls the third-party API.

The pool scales separately from intake, on queue depth and the age of the oldest job. By our estimate that is 28 processes with 20 jobs each, and the number depends on the speed of the target API.

The boundary in the frame separates what we control from what we do not. Our pool is on the left, third-party APIs on the right. On our side we can set a timeout, retries, and a per-domain limit.

• New on the diagramscrape-workerTakes a job from the queue and calls the third-party API. Nothing else in the system calls external APIs.

Step 17 of 36

Claiming a job, and the stuck ones

Receiving a message from the queue is not enough to take a job. The worker sets its status to "running" with the time it was taken, and only then calls the target. If a duplicate message arrives, the worker sees that the job is already taken and skips it.

This approach has a known problem: a worker can crash before it finishes. The job then stays "running" forever, and a dashboard will not show it, because formally it is in progress.

The fix uses the same timestamp. The scheduler service, job-scheduler, has a cron module: once a minute it finds jobs that have been in progress for too long, sets their status to queued, and clears published_at. The publisher sends them to the queue again. The C2 diagram shows job-scheduler itself: the cron is a module inside it, and its modules are shown at the C3 level, after the full C2 diagram.

• New on the diagramjob-schedulerThe scheduler: picks the next job, decides when attempts are exhausted, and its cron module returns stuck jobs to the queue once a minute.

Step 18 of 36

The read side: status and history

job-query covers the brief's requirement to see what is happening with jobs.

The read side answers three questions: what state a job is in, which jobs a user has, and what happened to a specific job. The third is the most useful: an attempt history with the target's answers shows, for example, "three timeouts, then 401", and from that it is clear what to fix.

The gateway sends these requests here, not to the command service. Their requirements differ: intake must answer within 200 ms and keep working when targets are down, while a job list can take a hundred milliseconds longer.

• New on the diagramjob-queryAnswers what state a job is in and what happened to it. Scales separately from writes: there are more reads.

Step 19 of 36

A cache on the read side

167 reads a second at peak against 28 writes, and all 167 land on jobs-pg — the same database intake writes to. The availability of intake is NFR-1, so a polling loop in someone's script can take down job intake, which it has nothing to do with.

jobs-cache goes between job-query and the database — an intermediate buffer holding the answer to GET /jobs/{id}: state, link to the result, timestamps. The key is job:{id}.

The scheme is cache-aside. The read side goes to the cache first: on a hit the answer is served without touching the database, on a miss job-query reads the row from jobs-pg and puts it into the cache itself. The write side knows nothing about the cache — neither job-command nor the worker writes to it. Write-through, where the value enters the cache at the moment it is written to the database, would need the opposite: both of them would have to know about it.

The job list is not cached. GET /jobs?status= has its own answer for every combination of filter, sort order and page, and any other job of the same user invalidates it. Many keys, a low hit rate, frequent eviction. What goes into the cache is what gets asked most often and answered identically: one job by id.

Then comes consistency. The job reached a final state while the cache holds "running"; the client reads a stale value and keeps polling. In that case the cache removed no load and added latency. So an entry has two lifetimes.

While the job is not in a final state, the TTL is two seconds. Such a TTL does nothing for a lone poller: at a five-second polling interval it misses every time. The bet is that a job usually has more than one reader — an open tab, a script and a page refresh produce several requests within one second, and one of them reaches the database.

Once it reaches a final state, the entry lives until retention ends. The state will not change again, and such a job is read for a long time: people come for the result link an hour later. Those reads do not reach the database at all.

What switches between the modes is an event. The fact that a job finished already travels through the bus, published by job-publisher for the webhook. The read side subscribes to the same queue and deletes the key; the next request goes to the database once and puts the final state into the cache.

The short TTL stays, and not as a duplicate of invalidation. Cache-aside has a race that event invalidation does not close: job-query misses and is reading the row from the database, meanwhile the job finishes, the event arrives and deletes a key that is not there yet — and then job-query puts a stale "running" into the cache. The window is narrow, but nothing bounds it except the TTL.

The other known hazard is a cache stampede, where a hot key expires and every request waiting on it goes to the database at once. There is no room for it here: at a two-second TTL and a dozen readers per job that is a handful of requests. On a longer TTL and hotter keys, one reader would have to be let through and the rest made to wait for its answer.

What the hit rate turns out to be depends on how many readers there are per job. That number gets measured — like the number of jobs one worker holds.

• New on the diagramjobs-cacheredisAn intermediate buffer in front of jobs-pg holding the answer to GET /jobs/{id}. On a hit, status polling never reaches the database.

Step 20 of 36

Getting the result

For the client a job has two states: still running, or done. Finding out which is an ordinary database query through job-query. The answer contains the status and, if the job is done, a link to the result.

The data itself lives in results-s3: its volume is in terabytes, and it does not belong in the state database. job-query does not pass it through; it returns a time-limited presigned URL, and the client downloads the result directly from storage.

The worker sets the status to "done" only after the file is already in storage, so the link of a finished job always leads to the result. How storage and retention work is covered in the step "Where the results live".

• New on the diagramresults-s3s3Job results. Their volume is in terabytes, so they are stored apart from the state database.

Step 24 of 36

C3: inside job-scheduler

One level down. C3 shows what a single service is made of. Take job-scheduler: on C2 it was one box that writes to the database, and inside it has three modules.

• stuck-sweeper — the cron module from the lease step. Once a minute it finds jobs with an expired lease, returns them to queued, and clears published_at.
• retry-planner — decides what happens after a failed attempt. A temporary failure gets the next attempt scheduled with backoff; a permanent failure or exhausted attempts move the job to failed. Details are in the retries step.
• jobs-repository — the only module that talks to jobs-pg. The other two go through it.

No module writes to the bus. They change the job row and clear the publish mark, and job-publisher sends the event. So the scheduler has the same guarantee as intake: the state change and the "to publish" mark are one write.

• New on the diagramstuck-sweeperCron module: once a minute returns jobs with an expired lease to the queue.
• New on the diagramretry-plannerSchedules the next attempt with backoff, or moves the job to failed when attempts run out.
• New on the diagramjobs-repositoryThe only module that talks to jobs-pg.

Step 25 of 36

The contract: a receipt instead of a result

POST /jobs answers 202 Accepted, not 200 with data: there is no data yet, and an open connection would be cut by the load balancer's timeout. The answer is a receipt: the id and where to track the job.

Idempotency key: a client that did not get the answer retries and gets the same job, not a second one.

The result is served not by the API but by storage, through a signed link: storage serves large volumes more cheaply. The webhook complements polling but does not replace it: it may not arrive.

• POST/jobs202 Accepted

  Submit a job. The answer is immediate and carries no data: just a receipt and where to track it.

  request

  Idempotency-Keyurlparamspriority

  response

  idstatus: queuedlinks.self

• GET/jobs/{id}200 OK

  Job state, number of attempts and the last target error.

  response

  statusattemptslast_error

• GET/jobs?status=200 OK

  Your jobs filtered by state: a script with a thousand jobs needs one request.

  response

  items[]next_cursor

• GET/jobs/{id}/result200 OK

  Not the data but a signed storage link with an expiry.

  response

  urlexpires_at

• POST{callback_url}200 OK

  outboundWe call the client when the job is finished. It may not arrive, so polling stays.

  request

  idstatusfinished_at

Step 26 of 36

The life of a job

A job has few states, and those are what the user sees.

queued → running → succeeded or failed. In addition, cancelled: while the job has not started, or while the worker can still stop it. From running there is a transition back to queued: the attempt failed, and the job waits for the next one.

The worker moves a job to running when it takes it. It also moves it to succeeded, but only after the result is saved in storage; otherwise there would be "done" with no data. A job moves to failed when its attempts run out. The scheduler decides that, not the worker.

A job that was taken and lost needs separate handling: the worker crashed, and the job stayed in running. This is solved with a lease: the worker extends its ownership of the job, and if the lease expires, the job returns to the queue. Without a lease, the jobs of a crashed machine never come back, and a dashboard will not show it, because formally they are running.

Step 27 of 36

Timeouts, retries, and what counts as failure

The brief asks that jobs do not fail for no reason. For that we need to tell two kinds of failure apart. The line is not the status code but whether another attempt can give a different result.

Temporary failures. A timeout, a dropped connection, 429, 503. Retrying makes sense, but not immediately and not with the whole pool at once. We need exponential backoff with random jitter. Without jitter, a thousand jobs that failed in the same second retry in the same second and overload the target again just as it starts to recover.

Permanent failures. 404, 401, wrong parameters, an invalid response. A retry gives the same result and costs another request to the target. Such a job goes straight to failed.

The number of attempts is limited. Jobs that run out of attempts go to a dead-letter queue, where a human looks at them. For example, a hundred jobs to one domain answering 401 usually mean one expired key.

A retry is safe only if the target does not treat it as a new action. For reads this is almost always true, but it is worth checking before retries are turned on.

Step 28 of 36

Being polite to somebody else's API

We sized our own pool, but the target has its own limits. 28 workers with 20 jobs each give up to 560 parallel requests. If they all go to one domain, we get blocked before we run out of our own capacity.

So the limit is set per target: so many requests a second and so many concurrent connections per domain. This is a token bucket per host, shared by all workers, so it is stored outside process memory, where all workers can see it.

It follows that the queue is split by target, not by user. In a shared queue, a job for a free domain waits behind jobs for a busy one, and the pool sits idle. With a queue per domain, a busy domain only delays its own jobs.

The Retry-After header in the target's response must be respected. Ignoring it can turn a temporary limit into a permanent block.

Step 29 of 36

The noisy neighbour

Tenant isolation is a single line in the requirements. The simulator on the right shows what it means in practice.

Three users: a shop and an analyst with a small flow, and a crawler that submitted 400 jobs at once. A pool of eight workers at four seconds a job handles two jobs a second. Turn on the shared queue: in the first seconds the crawler completes dozens of jobs, while its neighbours have none done and their queue grows. The system is working correctly: it runs jobs in the order they arrived.

Switch the order to equal share. Same workers, same flow, same job duration; only the rule for picking the next job changed. The neighbours get their results, the crawler gets a third of the pool, and its queue takes longer to drain.

A fair order makes the scheduler more complex: it needs a queue per tenant, tracking of allocated slots, and a rule for a tenant that was quiet for a long time and suddenly arrives with work. The next step is weights: paying customers can get more than free ones. Product decides the weights.

Step 31 of 36

What to measure

A system like this has many metrics, but four are enough for decisions.

• Age of the oldest job in the queue. Matters more than queue length: a thousand jobs processed within a minute is normal work, ten jobs waiting for an hour is an incident. It is also a good metric to scale the pool on.
• Time to first attempt, p95. Maps directly to the SLO. Other metrics help explain why it was missed.
• Failure rate by class and by domain. The overall failure rate mixes one customer's expired key with problems at one target. A hundred 401 responses from one domain are one incident.
• Queue depth per tenant. Shows a noisy neighbour before anyone complains.

Jobs taken by a crashed worker need separate tracking. Formally they are in running, and normal metrics treat them as fine. They can only be found by time in state: a job running longer than its lease should appear in a report.

Step 32 of 36

A new requirement: priorities

Six months later product comes back with a new ask: some jobs are urgent. The shop needs a competitor's price before the sale ends, while the analyst's nightly export can wait.

POST /jobs gets a priority field: high, normal or low, normal by default. It is written to the jobs row with the rest of the job.

One queue cannot give priority: the broker hands out messages in order. So there are now three work queues: jobs.high, jobs.normal and jobs.low. The publisher reads the priority from the row and puts the event into the right one. The outbox does not change, only the route does. A retry after a failure goes back into the same queue.

The key part is how the worker picks a queue. If it always takes high first, a steady stream of urgent jobs means low never starts. So the worker takes by weight: out of ten jobs, six from high, three from normal and one from low. An empty queue gives its share to the rest.

Tenant fairness stays inside each queue: priority decides which queue to take from, fairness decides whose job to take from it.

• New on the diagramjobs.highUrgent jobs. The worker takes six out of ten from here.
• New on the diagramjobs.normalThe default priority. Three out of ten.
• New on the diagramjobs.lowWhat can wait. One out of ten, but never zero.

Step 33 of 36

Priorities in the simulator

The same pool of eight workers and the same crawler with a burst of 400 jobs. Now each tenant has a priority: the shop is high, the crawler is normal, the analyst is low.

Run it with strict priority. The shop gets a slot right away, the crawler works through its burst, and the analyst never starts while the crawler has work. Nobody made a mistake: low is simply always last.

Switch to weights. The crawler and the shop are still ahead, but the analyst gets its share, one job out of ten, and its queue moves.

Change the priorities above the diagram: give the crawler high. The burst still does not take the whole pool: high gets six picks out of ten, the rest goes to the other queues.