Documentation

Operations and limits

Operational limits, failure checks, and benchmark context.

Fixed request and render limits

Limit	Value
HTTP request body	1 MiB
Render data	64 KiB
One request asset	256 KiB
All request assets	512 KiB
One published object	2 MiB
One downloaded control-plane object	4 MiB
Concurrent cold catalog loads	32
Cold catalog load deadline	30 seconds
Worker-pool target	2
One render per worker	1
Render deadline	10 seconds

Operator configuration

The registry is a local bounded LRU cache, not a catalog mirror. A cold render reads the version from PostgreSQL and S3, SHA-256-verifies its objects, and then caches it. Startup does no catalog-wide load and eviction never prevents publication. Local publish, load, retire, and unretire work is coordinated per version, so slow I/O for one version does not block warm reads of another. Cross-replica retirement PostgreSQL LISTEN/NOTIFY invalidates peers, while freshness-bounded reconciliation repairs missed notifications under ADR 0004.

Variable	Default	Meaning
`REGISTRY_MAX_VERSIONS`	`128`	Versions in one renderer cache
`REGISTRY_MAX_BYTES`	`67108864`	Source, font, and static bytes in that cache
`MAX_ACTIVE_VERSIONS`	`10000`	Deployment-wide published-version limit
`DEFAULT_TEMPLATE_ACTIVE_VERSIONS`	`100`	Default template published-version limit
`MAX_STORAGE_BYTES`	`1073741824`	Catalog-accounted retained-version byte cap
`WORKER_BINARY`	sibling `katagami-worker`	Dedicated worker executable; production Compose uses `/usr/local/bin/katagami-worker`
`WORKER_POOL_SIZE`	`2`	Target supervised workers; each admits one render
`RENDER_DEADLINE_MS`	`10000`	Per-worker render deadline
`WORKER_MEMORY_BYTES`	`536870912`	Per-worker address-space limit on Unix except macOS; unsupported native hosts fail startup
`WORKER_MAX_REQUEST_BYTES`	`16777216`	Maximum worker IPC request
`WORKER_MAX_RESPONSE_BYTES`	`33554432`	Maximum worker IPC response
`WORKER_RESTART_BUDGET`	`5`	Deaths allowed in the restart window before the pool degrades
`WORKER_RESTART_WINDOW_MS`	`60000`	Restart-budget window
`WORKER_RESTART_BACKOFF_MS`	`250`	Replacement-spawn delay
`WORKER_RECOVERY_COOLDOWN_MS`	`30000`	Delay before a degraded pool retries recovery
`WORKER_LIVENESS_INTERVAL_MS`	`5000`	Worker liveness-probe cadence
`WORKER_LIVENESS_TIMEOUT_MS`	`1000`	Liveness deadline; cannot exceed its interval

All values are positive integers. PostgreSQL applies active-version and storage limits under its transaction-scoped advisory lock. Per-template overrides are SQL-resident operator policy, never publish input:

Compose applies RENDERER_CPUS (default 2.0) and RENDERER_MEMORY_LIMIT (default 2g) to the HTTP server and its complete worker pool. Size the pool within those shared cgroup limits. Native startup fails closed on macOS and non-Unix hosts because WORKER_MEMORY_BYTES cannot be enforced. Linux containers enforce both the per-worker address-space limit and the container cgroup limit.

UPDATE templates SET active_version_limit = 3 WHERE id = 'invoice';
UPDATE templates SET active_version_limit = NULL WHERE id = 'invoice';

Retirement frees an active slot but retains objects. Unretirement rechecks active capacity and re-verifies the stored pack. It does not free retained-byte capacity.

Legacy rows with object_bytes = NULL are repaired only when a cold render SHA-256-verifies that one version and records exact bytes. Startup never reloads the whole catalog for this. New publications require exact bytes. Until retained rows are accounted, publication fails closed with 503 catalog retained-version byte capacity exceeded or incomplete.

Distinct physical-object accounting is a separate ADR 0003 concern.

Metrics, traces, and privacy

Every renderer has a process-local Prometheus endpoint at GET /metrics on its dedicated METRICS_ADDR listener, defaulting to 0.0.0.0:9090. The public API listener returns 404 for /metrics. The supplied Compose files deliberately do not publish port 9090: scrape http://renderer:9090/metrics and http://renderer2:9090/metrics from the internal monitoring network instead of a public proxy or ingress.

Signal	Stable metric
Request rate and errors	`katagami_http_requests_total`, `katagami_http_errors_total`
HTTP latency	`katagami_http_request_duration_seconds`
Render rate, latency, and admission	`katagami_render_requests_total`, `katagami_render_duration_seconds`, `katagami_render_active`, `katagami_render_capacity`
Worker pool health	`katagami_worker_pool_state`, `katagami_worker_pool_workers`, `katagami_worker_pool_failures_total`
Catalog sync	`katagami_catalog_sync_age_seconds`, `katagami_catalog_sync_failures_total`
PostgreSQL and S3 failures	`katagami_dependency_failures_total`

Treat these as views of failures, not independent incident counters. A PostgreSQL catalog-listener transport error can increment both katagami_dependency_failures_total{dependency="postgres",operation="receive"} and katagami_catalog_sync_failures_total{class="receive"}. Count that as one incident: the first metric identifies the transport cause and the second records the catalog-sync effect. Do not add the two values together.

Labels are closed operational classifications. Template IDs, versions, request IDs, asset names, raw URLs, request JSON, and other unbounded customer values are never metric labels.

Metrics work without a collector. Set OTEL_EXPORTER_OTLP_ENDPOINT only for an operator-managed OTLP HTTP/protobuf collector; Katagami posts to its /v1/traces endpoint. The base URL can use HTTP or HTTPS; HTTPS uses the container’s trusted CA certificates. Sampling, storage, and retention are operator policy, and exporter failure is best effort: it cannot change requests, liveness, or readiness. Logs and traces exclude render JSON, request assets, admin tokens, credentials, source files, fonts, and PDF bytes.

Prometheus scrape failure and OTLP exporter or collector failure do not stop request handling. An unexpected exit of Katagami’s internal metrics listener is different: Katagami starts graceful shutdown and exits. Run Katagami under production supervision that restarts the process.

First checks

Symptom	First check
Renderer exits at startup	Check PostgreSQL, object storage, and required environment variables
`502 control plane unavailable`	Check object hashes and PostgreSQL and storage connectivity
`409 template version already exists or template is unknown`	Use a new immutable version name or create the template ID
`409 template version lifecycle state conflicts with requested transition`	The version is already in the requested lifecycle state
`410` on render	The version is retired; unretire only if active capacity permits
`413`	Reduce request data, assets, objects, or worker IPC frames
`422`	Check the request JSON shape, validate it against the published schema, then inspect Typst source and declared assets
`502 render worker died while rendering`	Inspect the worker exit in renderer logs; clients may retry a read-only render
`503 renderer cannot currently admit work`	Retry after saturation, degradation, or shutdown clears
`503` from `/health/ready`	Keep the renderer out of traffic until catalog freshness and worker admission recover
`503 active template version capacity exceeded`	Raise an operator active-version limit or retire an active version
`503 catalog retained-version byte capacity exceeded or incomplete`	Raise retained-byte capacity or cold-verify a legacy row; retirement does not help
`503 template catalog load capacity exceeded or timed out`	Retry after current cold catalog loads finish; warm cache hits remain available

The startup log has a closed failure_class. config identifies invalid environment configuration. public_api_listener and metrics_listener identify bind failures. postgres_connect, postgres_migrate, storage, and worker_config identify the failed startup stage without exposing an endpoint, credential, or raw error.

Readiness and recovery

Use GET /health/live only to prove the HTTP process can answer requests. Route traffic with GET /health/ready: it returns 204 only while catalog reconciliation is within its freshness bound and the worker pool can admit a render. It returns 503 while the pool is saturated, degraded after its restart budget, stopping, or while catalog freshness is stale. A worker death is reaped and replaced after the configured backoff; budget exhaustion enters the configured recovery cooldown before replacement is retried.

Benchmark context

On an ARM64 Docker host with both renderers capped at 2 CPU and 2 GiB, a small, steady-state invoice workload measured Katagami at 1,440–2,151 requests per second. The equivalent Gotenberg HTML conversion measured 10–14 requests per second. This is a stored-template API comparison, not a general renderer-only claim.

Reproduce it with:

REQUESTS=100 CONCURRENCY=4 CPUS=2 MEMORY=2g \
  bash scripts/benchmark-gotenberg.sh

Edit this page ↗ Report an issue ↗