System design: panda clips

Step 4 of 33

Functional requirements

Requirements live as a file in the service repository. The table on the right is that file; the step's rows are under the text, move them into the table. There is one document for the whole write-up: numbers and new properties will be added in the SLO and estimate steps.

A functional requirement describes an action of the system, and every action has a caller. So the table has a separate column: who presses the button.

The neural network is not in the table. It is a way to fulfil FR-2, not a requirement: from the uploader's point of view the system "finds the panda", and they do not see with what. Likewise there is no queue, no GPU and no CDN: those are decisions, and they appear on the diagram once the numbers justify them. And a "clip" in the table is what a viewer watches and tags; whether it is a file or timecodes is decided in the estimate.

FR-8 stands apart: delete a clip with no panda in it. It looks minor, but it is the only way to fix a detector mistake without a human on our side, and it comes back in the detection step.

Step 5 of 33

Out of scope

What the system does not do is written down as explicitly as what it does. In six months someone will ask why there are no recommendations, and the answer should be in the document.

• Moderation. Someone will upload a video with no panda and with something that should not be in the gallery. That is a separate loop with people, rules and reports.
• Recommendations. The first version lists by time and by tag. "Pick by taste" is a separate product with its own data.
• Other animals. There is one detector and it looks for a panda. A model per animal is a different pipeline and a different budget.
• Manual editing. Moving a clip boundary by a second. If a clip is timecodes, that is cheap: two numbers in a row. We still do not do it in the first version, but the door stays open, unlike with cutting.
• Live streams. Processing a stream rather than a file is a different system.

On the right are ten statements from the same conversation. Sort them by kind: some properties sound like functions, and those are exactly what reviews argue about.

Step 6 of 33

Non-functional requirements

The second section is properties. A non-functional requirement adds no actions to the system; it sets a guarantee for the ones that exist.

They are checked through consequences, so under each one it says what breaks without it. This task has three properties the scraper did not have, and all three are about bytes.

A video uploads without passing through our API. An hour-long clip is gigabytes. A request that size hits the body limit, the load balancer timeout and the memory of the service receiving it. The bytes must go straight to storage.

A clip starts playing fast from any country. The viewer does not wait, and the clip's bytes must not pass through our service either.

One uploader does not take the whole pool. A zoo will upload ten years of archive, and a tourist with one video will wait a day behind it.

The Target column is empty for now. Numbers come next.

Step 7 of 33

The numbers we commit to

A property without a number cannot be checked. An SLO sets the share, the limit and the window.

• Upload intake: 99.9% over 30 days. Start and complete answer even when the GPU pool is down. That is 43 minutes a month when uploading is impossible.
• 99% of videos reach a final state within a day with no human. The final states are ready, no pandas, and failed. A video stuck in processing forever is worse than a failed one.
• 95% of videos up to ten minutes get their clips within 15 minutes. This is a promise about the queue and the pool. It breaks when uploads outpace what the GPUs work off.
• The first frame within two seconds in 95 cases out of 100, from any country. We count the upper bound, not the average, and not only from our own city.
• Search within 300 ms in 99 cases out of 100, and a fresh tag visible within five seconds. The second number matters more: a search that cannot find a tag added a minute ago looks broken.

The targets go into the column next to the rows already in the table. Decisions are made by these numbers later: the third one broken, we grow the pool; the fourth one broken, we look at the CDN and at where the index sits in the file.

Step 8 of 33

Estimate: how many GPUs

The calculator is on the right. Five thousand videos a day is 0.06 videos a second on average and 0.23 at peak, if the peak is four times the average. For comparison, the scraping jobs service had 28 jobs a second. The flow here is a hundred times smaller, and at this flow the queue is not a question. The question is what happens to each video.

Each video costs GPU time. Not every frame: the model looks at two frames a second, and a five-minute video takes about a minute, 0.2 seconds of GPU per second of video. That is 83 GPU hours a day. Processing does nothing else: there is nothing to cut or transcode, a clip is timecodes.

How many GPUs to keep is given by Little's law: the flow at peak multiplied by the time of one processing run. 0.23 videos a second at 60 seconds each is 14 videos in flight at once. A GPU processes one video at a time, so that number is also the pool size at peak. Four are enough on average: the pool grows and shrinks with the queue depth.

The queue wait is a slider, not a result: Little's law does not say what it will be, but it translates it into videos. Five minutes of waiting is 69 videos in the queue at peak and six minutes from upload to clips; NFR-4 holds with room to spare. Push the wait to half an hour, and the promise is broken.

Push the GPU factor to one, so the model looks at every frame, and the pool grows to seventy. One detector slider costs more than everything else in this design, and that is the first argument for sampling frames rather than running them all.

Step 9 of 33

Estimate: how many bytes on disk

Now size, the reason the task is what it is.

A five-minute phone video at 10 Mbit/s weighs 358 MB. Five thousand of those a day is 1.7 TB of uploads per day. That much does not pass through our API; that is already requirement NFR-2, and on the diagram it becomes the first decision.

A clip is timecodes, so the original stays: it is what gets watched. 1.7 TB a day is 622 TB a year, and storage grows at that rate for as long as the videos live. The first thirty days, while a video is being watched, it stays hot, in a tier for frequent reads; after that it moves to an infrequent-access tier by a storage rule. In numbers that is 51 TB hot and 571 TB cold after a year.

The calculator also counts the alternative. If clips were cut into separate files and only those were kept, with a panda in frame for a fifth of the video and a clip encoded at 3 Mbit/s, that would be 37 TB a year, 17 times less. That is the price of "timecodes instead of cutting", and it has to be said out loud.

What it buys. No cutting and no transcoding: no CPU per video, no files that can drift away from the rows in the database. A clip boundary moves by changing two numbers. A missed panda can be found by a rerun, and a new model can be applied to the archive. With cutting, none of that is possible: the original is gone.

Push the video length to an hour or the bitrate to 50 Mbit/s, and the storage bill becomes the biggest in the system. Then cutting comes back to the table, and that is the calculator's honest answer. The storage budget row goes into the table; its number comes in the retention step.

Step 10 of 33

Estimate: how much is watched

The third calculator is about viewers. If every uploaded video collects a hundred clip views, that is 500 thousand views a day and 23 a second at peak. Each view is 20 seconds of the original at 10 Mbit/s, 25 megabytes: the clip is not transcoded, it is watched at the bitrate the phone recorded. In total 11 TB a day and 341 TB a month.

That is traffic that cannot pass through our service: a CDN serves it, and our API serves only the address and two timecodes. Row NFR-12 goes into the table; its number comes on the playback frame.

The bitrate while watching is a slider. Push it to three megabits: the traffic drops threefold, to 102 TB a month. That is what a one-time transcode on upload would give, one file per video at the delivery bitrate, still no cutting. The first version does not do it: it is CPU per video and a second copy. When the traffic bill outgrows the storage bill, the slider shows how much it would save.

The second half of the calculator is metadata. Each view is one read of the clip page from the database, and every fifth viewer also searches. At peak that is 28 reads a second against 0.7 clip writes: forty reads per write. The system is read-heavy, as the YouTube analogy promised, but 28 reads a second is nothing for one database with an index. A cache and a separate read side are not needed here; they will be needed when views grow a hundredfold, and the road is known: the chapter on CQRS.

Push views to a thousand per video: bytes grow to petabytes a month, while database reads are still under three hundred a second. Bytes grow faster than requests, and that is the main proportion of the task.

Step 11 of 33

C1: the system and everything that talks to it

Now we can draw. The first frame is C1: the system as one block.

On the left is the user: they upload, watch and tag. On the right is the CDN, somebody else's network of caches through which the viewer gets the clip's bytes. We do not operate it, only configure it, and its availability is not in our SLO, but NFR-5 cannot be met without it.

The two lines from the user are the colour legend for every following frame: writes go into the service, reads come from the CDN. That both byte paths bypass our services becomes visible on C2.

Step 12 of 33

The way in: bytes past the API

The first elements inside the boundary are api-gateway, video-api, the clips-pg database and the videos-s3 store.

Upload takes three moves. POST /videos creates a video row in the uploading state and returns presigned URLs for the parts. The phone sends the parts straight to videos-s3: in parts, re-uploading a dropped part, with no request body limit on our side. POST /videos/{id}/complete says everything has arrived and moves the video to uploaded.

Two small requests pass through video-api, and of the 358 MB of video, not a single byte. That is NFR-2 on the diagram. A service that proxied terabytes a day would have to scale by traffic rather than by requests, and it would do nothing with those bytes except pass them along.

The file that landed here is the only copy of the video for its whole life. The detector will take frames from it, and the viewer will watch clips from it.

• New on the diagramapi-gatewayThe entry point: checks who came, holds the shared limits and routes the request inside. Knows nothing about videos.
• New on the diagramvideo-apiThe only service with endpoints: starts an upload, accepts complete, writes tags, serves the gallery and search. Video bytes do not pass through it.
• New on the diagramclips-pgpostgresVideos, clips, tags. The video row is also the processing task; the clip row is two timecodes.
• New on the diagramvideos-s3s3The video as uploaded, the only copy. The user sends the bytes here directly, the detector reads them from here, the CDN pulls them from here.

Step 13 of 33

The queue is a table

After complete the video has to be processed, and the usual move is a broker: video-api publishes "video uploaded", a worker reads it. There is no broker here, and that is a decision, not an omission.

The videos row is the task. It has the uploaded status, an attempt number, a lease deadline and a next-run time. The clip-pipeline service asks the database once a second: SELECT … WHERE status = 'uploaded' AND next_run_at <= now() ORDER BY … FOR UPDATE SKIP LOCKED LIMIT 1, sets processing and a lease, and goes to work. Several workers take tasks from one table without getting in each other's way: locked rows are skipped.

Why this beats a broker here specifically. The flow is 0.23 videos a second at peak; fourteen workers at one query a second is fourteen index lookups. The video state and the task are one row, so there is no dual write of database plus broker, the one that gave the scraper its outbox. And the queue is visible with one SQL query: how many wait, how many are in flight, which task is the oldest.

A broker is for thousands of messages a second, many different consumers, or events that must reach several services. None of that is in the task: there is one consumer, and it is heavy. The table itself and the queries against it are on the next step.

• New on the diagramclip-pipelineA scheduler with the processing inside: takes the next video from the table under a lease, downloads the original, finds the panda and writes the clip timecodes. Cuts nothing.

Step 16 of 33

From the original to the timecodes

The pipeline took the task. Three actions follow, and none of them creates a new video.

The original is downloaded from videos-s3 once, to the worker's local disk. The detector samples frames from it, runs them through the model and merges the hits into ranges: start and end in milliseconds, and a confidence. From the frame with the most confident panda it makes a thumbnail and puts it into videos-s3, next to the video.

Then one transaction in clips-pg: the clip rows with their timecodes and the video status ready. Until it commits, the video is still processing for the whole system, and it has no clips.

The clip's bytes are the bytes of the original, which already sits where the user put it. The pipeline cuts and transcodes nothing: ffmpeg here only extracts frames.

Step 17 of 33

A worker died holding a video

Processing takes a minute, sometimes ten. In that time a worker can die: the model ran out of memory on a 4K frame, the cloud took back a spot machine, someone deployed a release. The video stays processing, and the dashboard does not show it, because formally it is being processed.

So the task has a lease. Taking a video, the worker sets lease_until ten minutes ahead and extends it every minute while it works. The same clip-pipeline looks once a minute for processing rows with an expired lease and returns them to uploaded: attempt plus one, next_run_at after a pause that grows with every attempt. After the third, failed with the text of the last error.

A retry must be safe. Clips are written with the key (video_id, start_ms), thumbnails with deterministic names: half the clips from the previous attempt get overwritten, not doubled. That is the same idempotency as at intake, only inside the pipeline. The original is not touched by a retry at all; it is only read.

Permanent errors, a corrupt file, not a video at all, are not worth retrying: those go straight to failed. The line runs not by error code but by whether a retry can give a different result.

Step 18 of 33

Reading: gallery, page, search

The read side is the same video-api. The scraper split reads and writes into separate services; here 28 reads a second at peak do not justify a second service, and the split is postponed until the numbers change.

Three requests. The gallery: clips by time, paged by cursor. The video page: the clips of one video_id, plus stage and attempts if the video is still processing; that is FR-2 and half of NFR-14. Search: clips by tags, with a step of its own.

What the API returns in each of them: metadata, the address of the original and two timecodes. Not a single byte of video is in the responses. Where those bytes are is the next frame.

Step 19 of 33

Playback: a range of the original from the CDN

GET /clips/{id} returns a video_url, start_ms and end_ms. The address points not at our service but at the CDN, with videos-s3 behind it. The player opens the original at the right second: with a request carrying the Range header it asks the CDN only for the bytes of the clip, not the whole file. On a miss the CDN node pulls that range from storage and keeps it.

One detail makes this possible. A phone often writes the frame index, the moov atom, at the end of the MP4, and a player cannot start in the middle without reading the file to the end. Moving the index to the front once, on upload, solves this without transcoding: the bytes are copied as they are.

Why a CDN, if there is already S3? Storage can serve ranges on its own, by link, and at a small traffic that is the right answer; the delivery calculator says so. A CDN in front of it is there for three things.

• Distance. The bucket lives in one region, and a viewer from the other hemisphere would cross an ocean for every chunk. A CDN node stands next to the viewer.
• Repeats. A good clip is watched a hundred thousand times. Without a CDN that is a hundred thousand reads from the bucket; with one, the bucket serves a range once per node, and the rest comes from the edge cache.
• Money. Egress from storage to the internet is billed higher than egress from a CDN, and every request to the bucket is billed too.

Two requirements close at once. NFR-12: 11 TB a day pass through neither video-api nor api-gateway. NFR-5: the first frame comes from the nearest edge, not from one region. The clip has a thumbnail shown while the first chunk loads.

Clips are public, as on YouTube, so the links are plain. If product wants private videos, signed CDN URLs appear in this same place, and it is the only place that changes.

Step 23 of 33

C3: inside clip-pipeline

We go one level down. C3 shows the modules of a single service. clip-pipeline was one box on C2; inside there are three.

• task-poller takes the next task under a lease, extends the lease and returns expired ones. Everything that belongs to the queue lives here.
• panda-detector decodes the original at two frames a second, runs the frames through the detector in batches and merges the hits into ranges: with a confidence threshold, a minimum length and padding at the edges. From the best frame it makes a thumbnail and puts it next to the video.
• videos-repository is the only module that goes to the database: timecodes and status in one transaction.

If a one-time transcode of the original to the delivery bitrate is ever needed, it becomes a separate module on a CPU, not a GPU, and it makes one file per video, not per clip.

• New on the diagramtask-pollerThe whole queue: the next task under a lease, extension, returning expired ones.
• New on the diagrampanda-detectorFrames through the network, hits into ranges with a threshold, a minimum length and padding at the edges; a thumbnail from the best frame.
• New on the diagramvideos-repositoryThe only module that goes to clips-pg: timecodes and status in one transaction.

Step 25 of 33

The contract: a receipt and addresses

The contract as cards. Three of them are about upload, and the video bytes are in only one, which is not ours.

POST /videos answers 201 with an id and presigned URLs for the parts. A PUT to such a URL is S3, not us: the signature limits the period and the object, and the client re-uploads a dropped part itself. complete is 202 Accepted: there are no clips yet, there is a receipt and an address to watch.

The move to uploaded is a conditional UPDATE … WHERE status = 'uploading'. A repeated complete after a lost answer changes nothing and gets the same 202: processing starts once. That is NFR-8, and the idempotency key here is the video id itself.

The remaining cards are reads and tags. GET /clips/{id} returns a video_url on the CDN and the pair start_ms, end_ms: the player asks for the range itself. DELETE /clips/{id} closes FR-8: the row and the thumbnail go, the original only together with the video.

• POST/videos201 Created

  Create a video. The answer carries the id and presigned part URLs: the bytes go there, past the API.

  request

  Idempotency-Keysizecontent_type

  response

  idstatus: uploadingupload.part_urls[]upload.expires_at

• PUT{upload.part_url}200 OK

  A part of the video, straight to storage. Not our endpoint: this is S3, we only signed it.

  request

  part bytes

  response

  ETag

• POST/videos/{id}/complete202 Accepted

  The upload is finished. A repeat with the same id gets the same answer: processing starts once.

  request

  parts[].etag

  response

  status: uploadedlinks.self

• GET/videos/{id}200 OK

  What is happening to the video: stage, attempts, finished clips.

  response

  statusstageattemptsclips[]

• GET/clips?tag=&cursor=200 OK

  Clips by tag, paged by cursor.

  response

  items[]next_cursor

• GET/clips/{id}200 OK

  The clip page: description, tags, the original address on the CDN and two timecodes. The player asks for the range itself.

  response

  titledescriptiontags[]video_urlstart_msend_msthumbnail_url

• POST/clips/{id}/tags201 Created

  Add a tag. The word is normalised; the same tag is not added twice.

  request

  tag

  response

  tags[]

• DELETE/clips/{id}204 No Content

  Remove a clip: a false positive, or your own video. The row and the thumbnail go; the original goes only with the video.

Step 26 of 33

The life of a video

A video has six states, and three of them are final.

uploading → uploaded → processing → ready. Next to it two other ends: empty, when the detector found nothing, and failed, when the attempts ran out or the upload was never completed within a day.

Who moves it is shown by colour. Into uploaded: video-api on complete. Into processing and on to ready or empty: the pipeline, and only after the timecodes commit. Back to uploaded and into failed: the poller by lease and attempts, not the detector itself. A crashed worker moves nothing; whoever notices the expired lease does it for it.

empty is an honest state, not an error. Half the videos from a zoo will be of an enclosure with no panda, and the user is told so in words, not with an empty page. And the original stays: if the detector was wrong, the video can be queued again, with the same move to uploaded.

Step 27 of 33

Detection: threshold, length, errors

The neural network is not a black box with a yes or no answer but a source of confidence per frame. The rules that turn it into clips are set by product, and they go into the requirements table.

Threshold. A frame counts as a panda frame at confidence above 0.7. Lower means more false clips; higher means more misses. The threshold is tuned on a labelled sample and changed without a deploy.

Length and merging. Hits are merged into a range if the pause between them is under two seconds; a range shorter than five seconds is dropped; a second is added at each edge so the panda does not walk into the frame on the clip's first frame. All of this is arithmetic over timecodes, and the rules can change without touching a single file.

Two kinds of errors. A false positive, a clip with a raccoon, is fixed by the uploader through FR-8, and that is cheap: a row goes. A miss, a panda with no clip, is fixed by a rerun: the original is in place, and "run it again" is an ordinary task in the same queue, with a new threshold or a new model. The cutting design has no such path.

A new model. Applies to new uploads immediately, and to the archive through the same queue, one video at a time, in the background and without hurry. That is the second thing bought by keeping the originals.

Step 28 of 33

A noisy neighbour on the GPUs

Uploader isolation is written down as one row. The simulator on the right shows what it means.

Three users: a tourist and a school trip with a video every two seconds, and a zoo that uploaded its archive, four hundred videos at once. A pool of eight GPUs at four seconds per video works off two videos a second. Switch on the shared queue: the zoo occupies the pool for three minutes, and the tourist has not a single clip ready in that time. The system is right all along: it takes tasks in order of arrival.

Switch to "equal share". The pool is the same; only the rule for the next task changed: whoever has the fewest in flight goes next. The tourist gets clips in seconds, the zoo gets a third of the pool.

In a table this is one ORDER BY in the poller's query: owners with fewer in flight first, then by upload time. What a broker would need a per-owner queue and a scheduler of your own for is one line of SQL here. Weights, more for a paying zoo, are the same line with a coefficient.

Step 30 of 33

What to keep, and for how long

Three classes of data with different fates.

Originals. They live as long as the video does: they are the clips. 622 TB a year at the current flow is the main cost line, and it is managed by a storage rule rather than by code. For the first thirty days, while a video is being watched, it sits in a tier for frequent reads; then it moves to an infrequent-access tier, where storage is cheaper and each read costs more. Repeated reads are covered by the CDN: a cold original comes up from storage once per node. Deleting a video takes the original, the thumbnails and the clip rows, and purges the CDN by address. Abandoned uploads, a row without complete for a day, are picked up by a scheduled cleanup.

Thumbnails. Kilobytes per clip; their volume does not show on the bill.

Rows in the database. Millions of clips a year is a table Postgres barely notices. videos needs attention of its own: the status changes several times in a row's life, and dead versions pile up. Autovacuum copes, but it has to be watched, because the queue runs on this same table.

The number for NFR-10 goes into the table: the original lives as long as the video, about 622 TB a year, an infrequent-access tier after thirty days.

Step 31 of 33

What to measure

There are many metrics; five are enough for decisions.

• The age of the oldest task in uploaded. More important than the queue length: the pool grows by it, and it shows the pool is not keeping up before NFR-4 breaks.
• Time from complete to ready, p95. Maps directly to the SLO.
• Failures by error class. "Out of memory", "cannot decode", "model did not answer" are three different incidents, and they cannot be mixed into one failure rate.
• The share of empty. Doubled in a day means the threshold drifted or the model degraded, and users have not complained yet.
• CDN hit ratio and egress from videos-s3. If storage egress grows with views, the CDN is not caching ranges, and the storage bill will say so later than the metric.

And stage with attempts on the video page: the on-call engineer should not need the database to see "attempt 2 of 3, crashed on frame 1,840".