What we measured building on Deadline Cloud.

AWS's own scene reader finds 46 files or 1, depending where you run it

# same shot, same code, library present:
blend_paths(absolute=True) -> 46
libraries -> ['//../../../../lib/sets/monastery/monastery.blend']
images -> 45, of which linked: 45

# library absent:
blend_paths(absolute=True) -> 1

A shot that links a set library names exactly one thing: the library. The 45 textures are inside it, and Blender can only report them once it has opened it. AWS’s Blender submitter asks Blender a single question — bpy.utils.blend_paths() — and there is no second pass anywhere in their repo.

That is not a bug in their code. Their submitter is an addon: it runs inside the artist’s own Blender, where every linked library is already open, so the recursion is Blender’s and one call returns the whole tree. From where they stand the problem does not exist.

It exists for anyone reading the scene anywhere else. Run their code on a render worker holding a copy of the scene and nothing else, and it truthfully reports one dependency — and a scan that stops there has just described a shot as needing one file.

Production ran 11 migrations while the repo had 30

column farm_connection.watermark_previews does not exist
SQLSTATE 42703  →  500 on every route that reads a connection

Our runbook said migrations were “run by hand from the CLI”. Deploys are triggered by a git push, so there was no CLI step in a release at all — the hand-run was a thing to remember, and after four days nobody did.

The gap stayed invisible while the schema happened to be enough. Then a release shipped code selecting a column production had never been given, and every route that reads a farm connection answered 500. The event-ingest endpoint was among them, so a connected farm’s events would have been retried ten times an hour against a wall.

500 passing tests over a pipeline that could not render a frame

461 web tests passed
 39 bundler tests passed
  0 frames submittable

A worker writes files; the server reads them. Over one day we changed four things about that contract on one side only. Every check stayed green while the scan failed on every scene and the submit path sent a debug command to the farm as its render template.

The cause was not missing tests. It was that every test asserted one side of the boundary against a fixture typed out in the same file, so both halves of a broken contract agreed with themselves — forever, and confidently.

The gate written afterwards then found the same shape one level up: it needed the worker’s Python to run, CI had none, so it skipped. Twenty-six tests disappeared into a green run, because a skipped suite and a passing suite are the same colour.

The per-application render adaptors are already on the farm's own channel

CondaPackages: blender-openjd
downloaded 444.9 MB/444.9 MB (2/2 packages complete), 426.2 MB cached
blender          5.1.2   deadline-cloud
blender-openjd   0.6.1   deadline-cloud

AWS publishes a render adaptor for ten applications — Blender, Maya, Houdini, Nuke, Cinema 4D, 3ds Max, Unreal, After Effects, KeyShot, VRED — under Apache 2.0. We had assumed adopting one meant installing it onto workers ourselves, and costed a slower cold start for it.

It resolves straight from the deadline-cloud conda channel the queue already defaults to, and pulls the application itself as a dependency — so it replaces blender in the package list rather than adding to it. The expensive half of that plan did not exist.

This does not generalise across the channel, which is why it was measured rather than reasoned. The bundling libraries deadline-cloud-job-attachments are not there — we hit that wall in June. Render-side adaptors are; asset-side libraries are not.

We sent a scene to the farm without the textures it asks for

scene staged alone, read by Blender's own parser on the farm:
  //tex/grid_diffuse.png   referenced, not sent
  //tex/panel_rough.png    referenced, not sent

A submission sent the file the artist picked. A scene is not one file — it points at textures, caches, references — and nothing read it, so nothing knew. The render either fails on the farm or comes back quietly wrong, and the person who submitted it finds out in the morning.

Reading the scene is the fix, but the scene has to be held to be read, and customer assets never enter our account. So the reading happens on the studio’s own machines, using the application’s own parser — never a format reader of ours, which would be a per-application maintenance tax whose failure mode is a silently incomplete render.

The same read answers questions we had been asking the artist to type. That scene declares frames 1–250, EEVEE, saved with Blender 5.1. Our form had a frame range defaulting to 1–1, which meant the obvious path rendered a single frame of a 250‑frame scene and said nothing about it.

The farm forgets which machines ran your renders

our record of the same window : 37 sessions, 18 workers, 1 fleet
aws deadline search-workers   : 0

Both numbers were taken within a minute of each other. Every one of those eighteen machines had already been forgotten by the platform: the fleet scaled to zero and took the worker records with it.

The session records survive a while longer, which is the only reason the history was recoverable on the day we went looking. After that, “which machine ran this shot, and for how long” is not answerable at any price. There is no archive to page back through — either something wrote it down while the fleet was warm or the answer is gone.

The build that shipped this page failed, because a local build is not the same build

local   pnpm build  ->  green   (.env.local auto-loaded by Next)
vercel  pnpm build  ->  Error: Invalid server environment
                          DATABASE_URL: received undefined

Releasing this page failed on the first attempt. The auth module is built lazily on purpose, so that importing it never reads the environment — but one line called it at module scope, and Next evaluates route modules at build time. So the build demanded a database password it had no use for.

Every gate had passed locally. They passed because Next auto-loads .env.local, so the local build was quietly holding the exact secrets production does not have. It was never evidence about production — it was the same test run with a privilege production lacks.

The cost API rounds to whole cents, so most shots cost $0.00

instance         n   cost.min   cost.max   cost.avg   cost.sum
c5.4xlarge      20    0.00043    0.02678    0.00000       0.04
c5.4xlarge       4    0.00069    0.00159    0.00000       0.00
c5ad.4xlarge     7    0.00071    0.00242    0.00000       0.01
c5d.4xlarge      2    0.00097    0.00199    0.00000       0.00
c6i.4xlarge      2    0.00098    0.00308    0.00000       0.00

Deadline Cloud will aggregate session statistics for you, and the response carries a costInUsd field. Run it over real work — 37 sessions, 1,231 seconds of compute across four instance types — and the precision is not uniform. min and max carry five decimals; sum and avg carry two.

Four of seven rows report sum: 0.00 while carrying real per-session cost. avg is 0.00000 in six of seven. A naive rollup of the field that looks most like a total returns six cents for the entire window and silently discards every sub-cent row.

Five different mistakes return a byte-identical error

wrong external id   ->  AccessDenied
missing role        ->  AccessDenied
wrong account       ->  AccessDenied
malformed role arn  ->  AccessDenied
our own token dead  ->  AccessDenied

Connecting a studio’s farm means assuming a read-only role in their AWS account. When that fails, STS returns the same AccessDenied text for at least five distinct causes — four of them theirs, one of them ours.

So any error message that names a single cause is guessing. Ours guessed. It told a studio their IAM role was misconfigured when the real cause was our own expired session token — we sent them to go fix something that was working.

Our own onboarding link was broken, and only the customer's path showed it

Connecting a farm hands the studio a CloudFormation quick-create URL with the parameters pre-filled, including a per-connection external ID that guards against the confused-deputy problem. That parameter was marked NoEcho: true, which is correct — it is a secret.

Quick-create links silently drop NoEcho parameters. The external ID never arrived, so the stack could not be completed and the connection could never verify. We had not noticed because we deployed our own stack from the CLI, where the parameter passes normally.

A job's own status field says READY while one of its tasks is running

taskRunStatus:       "READY"
taskRunStatusCounts: { "RUNNING": 1, "SUCCEEDED": 5, "READY": 0 }

Both of those lines came out of the same get-job payload. The job-level taskRunStatus is not an aggregate of the job’s tasks — it appears to describe the next task to schedule.

Trust it and a dashboard says queued over work that is visibly running. Ours did, and we only found it because we submitted two real renders to prove a colour was reachable.

The farm re-ran a corrupt scene twelve times

corrupt scene   -> 12 sessions  (File format is not supported)
missing binary  ->  6 sessions  (exited with code: 127)
canceled midrun ->  1 session   (no error text at all)

We broke renders on purpose to see what the farm does with a failure it cannot win. A scene Blender cannot open burned twelve sessions. A missing renderer binary burned six.

Both failures are deterministic. Attempt two was never going to work, and neither was attempt twelve. Each one leases a worker and bills for it.

A canceled task produces a completely clean log

Cancel a render mid-run and the session log shows normal setup, normal cleanup, and no error text of any kind. Nothing in the log distinguishes it from a healthy run.

Anything that classifies failures by reading logs will call that task fine. The truth is only in the run-status transitions on the event stream.

There is an undocumented cap of two pending quota requests per account

default on-demand vCPU   50    (raised: automatic)
default spot vCPU        50    (raised: support case)
default GPU               0    (blocked)

A new Deadline Cloud account starts at 50 on-demand vCPU, 50 spot, and zero GPU. Fifty vCPU is three of the instances the setup wizard itself recommends, so every studio doing real work has to raise these before the farm is usable.

Raising GPU from 0 was refused — not on merit, but because two other quota requests were already pending. The cap is not in the docs and the error only appears at submit time, after the form is filled.

The guided setup dead-ends by telling the buyer to contact their IT admin

"No groups in your management instance.
 Contact your IT admin."

That is step 2 of the guided farm setup — a hard stop, no link, no inline fix. At a 30-person studio the person reading it is the IT admin.

The groups it wants live in a different service (IAM Identity Center) inside a different account (the organisation’s management account). Nothing on the screen says so. You need to already know the shape of AWS Organizations to understand what you are being asked for.

Fixing that dead-end signed us out and reset the wizard to step one

Resolving the previous finding means opening a second AWS account’s console. Multi-session is off by default, so opening the management account silently ended the session in the first one.

No warning, no prompt. Wizard state gone, sign in again, start from screen one — while holding in your head the thing you left to go fix.

The dashboard felt loose because the design it copies renders in quirks mode

prototype  job row: 16.0px / 25.5px   <- no doctype, quirks mode
ours       job row: 11.5px / 22.67px  <- inherited the body font

The verdict on the first cut of the dashboard was that it “doesn’t have the tightness” of the design it is meant to copy. Reading both stylesheets side by side explains nothing: they agree.

Dumping computed styles from both pages at one viewport found it in a minute. The ratified design file has no <!doctype>, so it renders in quirks mode — where table cells do not inherit the body font and fall back to 16px. That one-step size difference was the entire gap.