# NoETL DSL knowledge base > Machine-readable, version-accurate NoETL playbook grammar. Generated from > noetl-orchestrate-core; never hand-edited. - apiVersion: noetl.io/v2 (document kind: Playbook) - schema_digest: sha256:7c841140c76037ec82f64e6a0055743d037831b4d704b8e21492aa66f4eb58ef - source: noetl/server@62137e65756e ## Artifacts - [Playbook JSON Schema (draft 2020-12)](/schema/v2/playbook.schema.json) - [Tool-kind manifest](/dsl/manifest.json) - [Provenance](/dsl/source.json) - [Resource and performance model](/dsl/resource-model.json): slots, limits, measured latency, planning heuristics ## Tool kinds (default gates) - agent: disabled (gated by NOETL_TOOL_KIND_AGENT); worker also needs NOETL_WORKER_AGENT_TOOL=true - runs where: Needs the server gate AND a worker started with NOETL_WORKER_AGENT_TOOL=true (noetl/worker#398, noetl-tools 4.5.0); otherwise the worker fails the command as an unknown tool. The shared pool refuses `framework: claude-code`. A playbook-bound `tools:` entry (`playbook: `) is refused before the first model call unless the worker also sets NOETL_WORKER_AGENT_PLAYBOOK_TOOLS=true (noetl/worker v6.14.0); see the manifest's worker_runtime.agent. - artifact: enabled - container: enabled - duckdb: enabled - ducklake: enabled - gateway: enabled - gcs: enabled - http: enabled - iterator: enabled - mcp: disabled (gated by NOETL_TOOL_KIND_MCP) - runs where: Runs on any noetl/worker: noetl-tools McpTool, wrapped by the worker's WorkerMcpTool for auth headers. Only the server gate is needed. - nats: enabled - noop: enabled - playbook: enabled - runs where: Runs on any noetl/worker (noetl-tools PlaybookTool). Fields: `path` or `catalog_id`, `version` (integer), `payload`/`args`, `return_step`, `return_result`. With NOETL_CATALOG_PINNING on, `path` may be a catalog reference (`a/b@3`, `a/b@latest`, `a/b@sha256:...`, `sha256:...`, a source URI) and the child resolves inside the root's pin; see the manifest's catalog_features. - playbooks: enabled - postgres: enabled - python: enabled - rhai: enabled - script: enabled - secrets: enabled - shell: enabled - snowflake: enabled - snowflake_transfer: enabled - subscription: enabled - task_sequence: enabled - transfer: enabled - wasm: enabled - runs where: Runs on a noetl/worker built with its `wasm-plugin` cargo feature (on by default and in the shipped image), which routes it to the plug-in host instead of the tool registry. Step: `plugin: { path: , version: , entry: }` plus optional `input` (sent to the plug-in as `args`). The module is loaded from the catalog. Since noetl/worker v6.13.0 every call runs under fuel, wall-clock and memory limits that are ON by default; see the manifest's worker_runtime.wasm_limits. - workbook: enabled ## Playbook metadata a server acts on (flag-armed) - metadata.expose.mcp: only when the server sets NOETL_MCP_SERVER=true; serves POST /api/mcp (one MCP tool per exposed playbook) - description: required - input_schema: required; written by hand, never derived from workload; also the tools/call argument allowlist - mode: sync (default) | async - result_step: optional; the step whose result is returned - timeout: sync wait in seconds (default 30, max 120) - tool: required; MCP tool name, [A-Za-z0-9_.-]{1,64} - metadata.expose.agent: only when the server sets NOETL_AGENT_ENDPOINT=true; serves GET /api/agents, GET /api/agents/cards/{path}, POST /api/agents/send, GET /api/agents/tasks/{id} - card: required mapping; card.name and card.description required, card.version and card.skills optional - input_schema: required unless expose.mcp.input_schema is set; written by hand; the argument allowlist - result_step: optional - metadata.catalog.hot_reload: only when the server sets NOETL_CATALOG_HOT_RELOAD=true and NOETL_CATALOG_PINNING=true and NOETL_CATALOG_CAS=true; serves an explicit `@latest` sub-playbook reference inside this root's pinned execution tree - (list): globs over sub-playbook paths, read from the ROOT playbook only; at most 64, each 1-512 chars; malformed entries are ignored, so a typo only narrows the policy ## Catalog references and features (flag-armed) - ` the pinned version inside a pinned tree, else latest` - `@ an exact version` - `@latest latest; inside a pinned tree only with hot reload allowed` - `@sha256:<64hex> this name, and it must have this body` - `sha256:<64hex> this body, under any name` - `://[#sha256:<64hex>] a source URI (fetched by locate)` - source schemes: noetl, gs, s3, az, minio+http, minio+https, https, http, db, ehdb, file - content_addressing: needs NOETL_CATALOG_CAS=true; digest on register, dedup, /api/catalog/{resolve,blob,sources,pin} - execution_pinning: needs NOETL_CATALOG_PINNING=true and NOETL_CATALOG_CAS; POST /api/execute `version` (a positive integer or "latest") and `digest` (sha256:, wins over version) are honoured, and `path` may be a reference form. A `kind: playbook` step sends `version` as an integer only, so it asks for @latest or a digest through its `path`. A child of a pinned execution with no version resolves to the root's pinned version. With the flag off, version/digest are ignored and the path resolves to its latest version - hot_reload: needs NOETL_CATALOG_HOT_RELOAD=true and NOETL_CATALOG_PINNING and NOETL_CATALOG_CAS; `@latest` (e.g. a `kind: playbook` path `a/b@latest`) inside a pinned tree, only for paths the root lists in metadata.catalog.hot_reload; otherwise refused, never silently downgraded - discovery: needs NOETL_CATALOG_DISCOVERY=true and NOETL_CATALOG_CAS and NOETL_INSTANCE_REGISTRY and NOETL_CATALOG_PEER_KEY_FILE; POST /api/catalog/peer/resolve (signed, trusted peers only), POST /api/catalog/locate (internal token); imports under peer// - uri_fetch: needs NOETL_CATALOG_URI_FETCH=true and NOETL_CATALOG_DISCOVERY and NOETL_CATALOG_URI_ALLOW; locate fetches a reference's own source URI; fail closed, nothing is fetched without an allowlist - remote_execution: needs NOETL_CATALOG_REMOTE_CALLS=true and NOETL_CATALOG_PINNING and NOETL_CATALOG_CAS; inside a pinned tree a remote reference resolves to what the ROOT first got for it, in both modes, even if the owner registers a new version mid-run; a child of an imported playbook resolves bare names on its home instance (peer//). With the flag off `remote: delegate` is refused by name and nothing else changes - `db:///.?path=[&version=]` (server 3.160.0+): one row from an operator-owned Postgres table with columns path (text), version (integer), content (text); without version the highest version for the path - rules: the alias names a credential in this server's credential store (host/port/user/password/database, or connection_string), never user:pass in the URI; schema and table are identifiers [A-Za-z_][A-Za-z0-9_]* of at most 63 bytes; path is required; version is a positive integer - needs: NOETL_CATALOG_URI_FETCH, NOETL_CATALOG_URI_ALLOW - `ehdb://catalog/` (server 3.160.0+): a registration record in the cluster's EHDB catalog tier, via the worker relay; the record's content_sha256 is checked before the body is returned; found by a paged scan of the tier (a keyed lookup is TARGET) - rules: only the `catalog` namespace resolves - needs: NOETL_CATALOG_URI_FETCH, NOETL_CATALOG_URI_ALLOW, NOETL_EHDB_WORKER_QUERY_URL, NOETL_CATALOG_LOG - on a cluster: the catalog tier holds records only when the server records registrations (NOETL_CATALOG_LOG=shadow) and the worker relay takes the server as the tier's producer (NOETL_EHDB_CATALOG_MIRROR_SOURCE=server, unset by default); otherwise the read finds nothing or the relay reports the tier disabled ## Worker runtime (read by noetl/worker, not the server) - Read by noetl/worker, not by this server; the values here are the worker's code defaults. Which worker version a cluster runs is the cluster's fact, not this file's. - WASM limits (worker 6.13.0+): ON: every limit applies by default; set a variable to 0 / off / none / unlimited to remove that one limit; an unparseable value keeps the default; the module is stopped and the step fails with an error naming the variable - NOETL_WASM_FUEL: fuel units, default 10000000000 - NOETL_WASM_TIMEOUT_MS: milliseconds (wall clock), default 30000 - NOETL_WASM_MAX_MEMORY_MB: MiB (linear memory), default 1024 - Agent playbook-bound tools (worker 6.14.0+): needs NOETL_WORKER_AGENT_PLAYBOOK_TOOLS=true and NOETL_TOOL_KIND_AGENT=true and NOETL_WORKER_AGENT_TOOL=true; OFF (strict "true"); without it a playbook-bound tools: entry is refused before the first model call - tools: entry: { name, description, input_schema, playbook: , result_step: (optional; without it the child's last step result) } - the worker hosts the agent loop; each playbook-bound call starts a child execution (kind: playbook, return_result: true, parent = this execution) that runs on whatever pool its playbook targets; its result is the next turn's tool_result; a failed child is an is_error tool result - not durable: the loop lives in one command on one worker: if that worker dies mid-loop the step retries from the first turn and children that already ran run again (orchestrator-driven resume is TARGET) - NOETL_AGENT_MAX_TOOL_CALLS: tool calls per agent step, default 32 - NOETL_AGENT_TOOL_TIMEOUT_SECS: seconds per playbook-bound child, default 180 - Agent event types (emitted on the agent's own execution by the hosted loop only (NOETL_WORKER_AGENT_PLAYBOOK_TOOLS=true), through the normal emit path): - agent.turn: turn, outcome (tool_calls | final), tool_calls [{id, name}] - agent.tool_call: turn, tool_use_id, tool, binding, playbook, child_execution_id, is_error, duration_ms, input and result previews (2 KiB) - agent.final: turns, tool_calls, child_execution_ids, duration_ms; or refused: tool_call_budget ## Resource and performance model (plan before you run) - Full model: /dsl/resource-model.json (model_version 1, figures as of 2026-11-11) - Every figure is labelled: DEFAULT = a code default (env var + value); a deployment may override it; DEPLOYED = read off the production cluster on as_of; configuration, not code; GUIDANCE = a planning rule derived from the figures; not a measurement; MEASURED = observed on a cluster or harness; read environment/date/source before reusing it; NOT_YET_MEASURED = a real question with no number; do not invent one; TARGET = a goal or budget; not a measurement - Latency on NoETL is dominated by orchestration per step, not by the work a step does. Count steps and sub-playbook hops before estimating anything else. - Measurements are dated and environment-specific. A figure from kind does not transfer to prod, and an older figure may predate a fix. - Live values beat this file: a running server answers /api/dsl/version; a running worker's limits are its env; actual cost is in the events and metrics listed under metrics. - Every command holds one worker slot from claim until its tool returns, including case evaluation and its lifecycle event emits; a slot is released early only on a failed or refused claim. Waiting inside a tool (an awaited sub-playbook, a polled Kubernetes Job, an LLM call) keeps the slot. [DEFAULT] - The worker sets no wall-clock timeout per command. A step is bounded only by its own tool's timeout; kinds listed with timeout none can hold a slot indefinitely unless the step sets one. [DEFAULT] - ToolConfig.retry (max_retries 3, 500 ms initial, x2, 10 s max) is parsed but no executor applies it; do not budget for automatic tool retries. [DEFAULT] ### Step kinds (execution, slot, default timeout) - python: subprocess; slot: whole run; timeout: none unless the step sets timeout_seconds [DEFAULT] - limit: remote script-source loader: 30 s [DEFAULT] source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/python.rs:23 - test/simple_python playbook end to end on idle kind: 0.66 s warm / 1.20 s cold [MEASURED; kind, 2026-05] source: noetl/ai-meta handoffs/archive/2026-05-23-gke-provision-validation/round-01-prompt.md - provider dispatch moved from a kind: playbook child to inline python (HotelBeds hotels): 9.3 s -> 3.0 s [MEASURED; prod, 2026-08-24] source: noetl/travel wiki playbook-itinerary-planner (#286); noetl/ai-meta wiki Sessions-Log 2026-08-24 - the subprocess shares the worker pod's memory limit; stdout is collected whole - http: in-process (reqwest); slot: whole request; timeout: 30 s, client default; per-request timeout_seconds overrides it [DEFAULT] - limit: response body cap: no number [DEFAULT] (none: the body is read in full) source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/http.rs:249-259 - the response body is held in memory - postgres: in-process (deadpool-postgres, a pool per connection string); slot: whole query; timeout: none set by the tool [DEFAULT] - duckdb: in-process (spawn_blocking); slot: whole query; timeout: none [DEFAULT] - runs inside the worker process: a large query spends the pod's memory limit - rhai: in-process (spawn_blocking); slot: whole script; timeout: none, and no operation limits are set [DEFAULT] - wasm: in-process (wasmtime); one module runs at a time per worker process (the dispatcher sits behind a mutex); slot: whole invocation, plus any wait for the per-process wasm lock; timeout: 30000 ms (NOETL_WASM_TIMEOUT_MS) [DEFAULT] - limit: NOETL_WASM_FUEL: 10000000000 fuel units [DEFAULT] source: noetl/worker@d1c9584 src/plugin.rs:41, 131 - limit: NOETL_WASM_MAX_MEMORY_MB: 1024 MiB linear memory [DEFAULT] source: noetl/worker@d1c9584 src/plugin.rs:47, 138-143 - first dispatch of a ~1.6 MB module on a fresh pod (Cranelift compile ~99%, fetch ~1%); later hops hit the digest cache, and the drive pool pre-compiles at boot behind /readyz: ~1.5 s once [MEASURED; kind (fresh system-pool pod), 2026-06] source: noetl/worker wiki deployment-specification (boot warmup) - mcp: remote JSON-RPC over HTTP; slot: whole call; timeout: 60 s (NOETL_MCP_REQUEST_TIMEOUT_SECONDS), used when the step sets no timeout; clamped to NOETL_WORKER_COMMAND_TIMEOUT_SECONDS (default 180 s, read only here) [DEFAULT] - agent: in-process loop; every model call is a remote HTTP request; mcp_servers / agents entries are answered inside the loop; slot: the whole loop, including every model call and every awaited playbook-bound child; timeout: 300 s, model HTTP client timeout per request (ToolConfig.timeout default) [DEFAULT] - limit: max_turns: 8 [DEFAULT] source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/agent.rs:429-433 - limit: max_tokens per model call: 4096 [DEFAULT] source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/agent.rs:464 - limit: mcp_servers[].timeout: 60 s [DEFAULT] (clamped 1-600) source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/agent.rs:697-700 - limit: NOETL_AGENT_MAX_TOOL_CALLS: 32 tool calls per agent step (hosted loop) [DEFAULT] source: noetl/worker@d1c9584 src/executor/agent_host.rs:50, 94-96 - limit: NOETL_AGENT_TOOL_TIMEOUT_SECS: 180 s per playbook-bound child [DEFAULT] source: noetl/worker@d1c9584 src/executor/agent_host.rs:51, 97 - a hosted LLM (OpenAI / Anthropic) call inside a real playbook step: no number [NOT_YET_MEASURED] - candidate small language model, unoptimised local serving, 2 generations per turn: p50 9.2 s / p95 13.1 s per turn [MEASURED; local] source: noetl/travel wiki Travel-SLM-Journey - the hosted loop lives in one command: if its worker dies the step restarts from turn 1 and children that already ran run again - playbook: remote: POST /api/execute starts a separate child execution on whatever pool its steps target; slot: only for the start request by default (async: returns {status, execution_id}); with return_result: true or return_step the parent's slot is held, polling every 2 s, until the child finishes; timeout: 30 s, start request; the await wait is the step's timeout, else 180 s (return_result) / 300 s (return_step) [DEFAULT] - orchestration cost of one kind: playbook hop, regardless of the work inside it: ~9 s [MEASURED; prod, 2026-08-24] source: noetl/travel wiki playbook-itinerary-planner (#286); noetl/ai-meta wiki Sessions-Log 2026-08-24 - child-playbook hop vs the same work inline (Firestore read: 369 ms of work took 11.4 s as a child; Places: 663 ms of work took 1.5 s inline): ~10-15 s per hop [MEASURED; prod, 2026-08] source: noetl/travel wiki playbook-itinerary-planner#latency - paired A/B: provider children inlined into the planner as python (4 places turns each): median 34.7 s -> 19.4 s (-44%) [MEASURED; prod, 2026-06-29] source: noetl/ai-meta wiki Sessions-Log 2026-06-29 - an awaited child needs a free slot on its own pool while the parent holds one; if both target the same small pool, enough concurrent awaiting parents can occupy every slot - script: remote Kubernetes Job, polled inline every 2 s; slot: until the Job finishes; timeout: 600 s, job.timeout default, also the Job's activeDeadlineSeconds [DEFAULT] - container: remote Kubernetes Job; returns at once with a pending callback; slot: no: the slot is freed after the Job is created (the callback pattern); timeout: the Job's timeout_seconds; none by default [DEFAULT] - snowflake: remote REST; slot: whole statement; timeout: 60 s, statement timeout sent in the request body; the HTTP client sets none [DEFAULT] - subscription: in-process bounded drain (nats / pubsub / kafka / timer); slot: one bounded drain; timeout: 1000 ms, timeout_ms default, cap 5000; batch default 100, cap 1000 [DEFAULT] - task_sequence: in-process; runs its sub-tasks in order through a fresh default registry (so no worker mcp override and no agent); slot: all sub-tasks; timeout: per sub-task [DEFAULT] - limit: NOETL_TASK_SEQUENCE_MAX_ITERATIONS: 1000000 iterations [DEFAULT] source: noetl/tools@v4.6.0 (the version noetl/worker pins) src/tools/task_sequence.rs:253-257 - shell: subprocess; slot: whole run; timeout: none unless the step sets timeout_seconds [DEFAULT] - transfer: in-process (snowflake / postgres / http / duckdb); slot: whole transfer; timeout: HTTP client sets none; chunk_size default 1000 rows [DEFAULT] - noop: in-process; slot: briefly; it still pays the per-step orchestration cost - Profiled: the kinds whose execution was read in the worker and tools code. Also accepted by the parser and not profiled here: artifact, gateway, gcs, iterator, nats, playbooks, secrets, snowflake_transfer, workbook (see the tool-kind manifest); nothing is measured for them. ### Runtime limits - NOETL_WASM_FUEL: 10000000000 fuel units per invocation [DEFAULT]; 0 / off / none / unlimited removes the limit; an unparseable value keeps the default; a trip increments noetl_worker_plugin_limit_trips_total{limit} - NOETL_WASM_TIMEOUT_MS: 30000 ms wall clock per invocation [DEFAULT]; 10 ms deadline granularity - NOETL_WASM_MAX_MEMORY_MB: 1024 MiB linear memory [DEFAULT] - NOETL_AGENT_MAX_TOOL_CALLS: 32 tool calls per agent step [DEFAULT]; positive integers only; the model cannot raise it; exceeding it ends the step with agent.final refused: tool_call_budget - NOETL_AGENT_TOOL_TIMEOUT_SECS: 180 s per playbook-bound child [DEFAULT] - WORKER_MAX_CONCURRENT: 4 slots per worker process [DEFAULT] - NOETL_EVENT_RESULT_CONTEXT_MAX_BYTES: 102400 bytes [DEFAULT]; a step result above it is spilled to the result store and replaced by a reference: a downstream read then costs a fetch - NOETL_COMMAND_CONTEXT_MAX_BYTES: 524288 bytes [DEFAULT]; a command context above it is offloaded as a __context_ref__ - request body limit on /api/execute and /api/events: 2097152 bytes [DEFAULT] - GET /api/executions/{id} events returned: 2000 most recent events [DEFAULT] - NOETL_EVENT_EMIT_OUTAGE_BUDGET_SECS: 120 s [DEFAULT]; extra retry window for event emits during a server outage; the slot stays held - NOETL_WORKER_DRAIN_TIMEOUT_SECS: 150 s [DEFAULT]; on SIGTERM a worker finishes in-flight commands within this budget - catalog URI fetch (source: references): off; the allowlist is empty by default, which fetches nothing (fail closed, HTTP 403) [DEFAULT] - gated by: NOETL_CATALOG_URI_FETCH, NOETL_CATALOG_URI_ALLOW - http(s) fetch timeout, redirects disabled: 15 s [DEFAULT] source: noetl/server src/services/catalog_peer.rs:218-221 - db:// connect / query timeout; the connection is opened for one read: 10 s / 15 s [DEFAULT] source: noetl/server src/services/catalog_sources.rs:46-47 - ehdb:// read: linear scan, 16 records per page, at most 4000 pages, 60 s per page: worst case 64,000 records scanned [DEFAULT] source: noetl/server src/services/catalog_sources.rs:49-50, 312, 320 - fetched body cap (UTF-8 only): 1048576 bytes [DEFAULT] source: noetl/server src/services/catalog_peer.rs:76 - fetch cache: none: every locate fetches again; the imported body is deduplicated by digest [DEFAULT] source: noetl/server src/services/catalog_peer.rs:591-593, 863-867 - retries: none [DEFAULT] source: noetl/server src/services/catalog_peer.rs:896-951 - latency of a URI fetch: no number [NOT_YET_MEASURED] - prod: off on prod (no catalog CAS / pinning / discovery / URI flags in the server env) [DEPLOYED] - version pinning and hot reload: off. With pinning on, a root execution records a catalog watermark before its first event and every bare-path lookup in its tree resolves at or below it; hot_reload only applies to paths the ROOT lists (at most 64 globs) and referenced explicitly as @latest [DEFAULT] - gated by: NOETL_CATALOG_PINNING, NOETL_CATALOG_HOT_RELOAD - pinning adds no fetch: resolution is a bounded catalog lookup. Without pinning, /api/execute reads the latest catalog row on every request (no cache) - prod: off on prod [DEPLOYED] - loops and back-edges (goto): No server cap on iterations, steps, back-edge repeats or sub-playbook depth. A back-edge re-dispatches its target until the arc's when: guard goes false; the playbook's own guard is the only bound. [DEFAULT] - every iteration and every back-edge pass is a full step: it pays the per-step orchestration cost below and holds a slot - loop.spec.mode: parallel dispatches every item at once. loop.spec.max_in_flight is parsed but has no reader, so it does not throttle anything - NOETL_RECONCILE_MAX_NOOPS: 225 consecutive no-progress polls at an 8 s tick (~30 min) [DEFAULT]; how long a stuck execution is re-driven before the server stops ### Concurrency and scaling - A worker process has WORKER_MAX_CONCURRENT slots (a semaphore). A slot is acquired before the worker pulls a command and released when that command's tool returns. A pool's capacity is replicas x slots; commands beyond it queue on the command bus. [DEFAULT] - A worker claims only its pool's subject (commands..>); a step reaches a pool through the pool its command is published to. Orchestration of system/* drives runs on the system pool. [DEFAULT] - pool worker-rust-pool (user pool, Deployment noetl-worker-rust): replicas 2 (KEDA 2-20), 4 slots each, capacity 8 at min, 80 at max slots; cpu 250m request / 2 limit; memory 256Mi request / 768Mi limit [DEPLOYED, 2026-11-11] - pool worker-system-pool (Deployments noetl-worker-system-pool + -shard1): replicas 1 + 1, fixed (no autoscaler), 2 slots each, capacity 4 slots; cpu 750m request / 2 limit; memory 3Gi / 3Gi [DEPLOYED, 2026-11-11] - KEDA noetl-worker-rust: 2-20 replicas on metrics-api: ehdb_feed_subject_lag{subject="commands.shared.shard.0"} from the command-bus writer, target 2, poll 10 s, cooldown 30 s; replicas ~ ceil(lag / 2) [DEPLOYED] - ops/ci/manifests also declares a system-pool ScaledObject (3-10); it is not on the cluster, so the system pool does not scale - user pool under 200 real executions (200/200 completed): replicas 2 -> 4 -> 8 -> 16 -> 20 -> 2 [MEASURED; prod, 2026-07-31] source: noetl/ai-meta wiki Home (#210) - time for the user pool to ramp under load on the EHDB trigger: no number [NOT_YET_MEASURED] - GKE Autopilot node provisioning when a scale-up needs a new node: ~60-90 s [MEASURED; prod] source: noetl/ai-meta playbooks/load-test/README.md - command.issued -> command.claimed, pool not saturated: p50 138.5 / p95 156.4 / p99 181.0 ms (n=60) [MEASURED; prod, 2026-07-30] source: noetl/ai-meta wiki Sessions-Log 2026-07-30 (prod deploy) - same, with 90 commands offered in 1.2 s against 8 slots (2 replicas x 4): p50 2123 / p99 3735 ms [MEASURED; prod, 2026-07-30] source: noetl/ai-meta wiki Sessions-Log 2026-07-30 (prod deploy) - POST /api/execute submit latency by client concurrency (server v3.100.4, 2 reps): c=1 p50 120-150 ms; c=2 p50 140-150 ms; c>=4 bimodal with errors and 30 s timeouts [MEASURED; prod, 2026-09-03] source: noetl/ai-meta wiki Umbrella-Request-Latency - sustained executions per second the platform can complete: no number [NOT_YET_MEASURED] ### Orchestration overhead - Each step pays a fixed orchestration cost (command dispatch, claim, and three lifecycle event emits) on top of its work; each kind: playbook hop pays a whole child execution's worth. - fixed cost per step in the travel itinerary planner; events spaced a near-uniform 0.8-1.2 s apart: ~4.5 s per step [MEASURED; prod, 2026-08] source: noetl/travel wiki playbook-itinerary-planner#latency - share of a planner turn that is orchestration (63 events, ~15-17 s of work in a ~76-107 s turn): ~82% [MEASURED; prod, 2026-08] source: noetl/travel wiki playbook-itinerary-planner#latency - 10-hop probe playbook: share of a turn in two command-dispatch gaps per hop (~510 ms p50 each): 79% [MEASURED; prod, 2026-08-18] source: noetl/ai-meta wiki Sessions-Log 2026-08-18 - server event ingest per emit: command.started / command.completed vs call.done (three emits per hop): ~336 ms / ~1021 ms on prod, ~40 ms on kind [MEASURED; prod and kind] source: noetl/server src/metrics.rs (doc comment on noetl_event_ingest_phase_seconds) - prod smoke turns through the gateway: travel planner turn / ranch concierge turn: 17.0-21.2 s / 6.4-7.0 s [MEASURED; prod, 2026-11-04] source: noetl/ai-meta wiki Sessions-Log 2026-11-04 - first drive of a fresh system-pool pod after a roll (state builder replays the feed): drives stall ~7-10 min [MEASURED; prod, 2026-11-03] source: noetl/ai-meta wiki Sessions-Log 2026-11-03 (early) - command-bus writer restart (lease election): failover window 21 s; plan for a 20-60 s write outage [MEASURED; prod, 2026-11-04] source: noetl/worker wiki cmdbus-writer-tier-service - caveat: Per-step cost varies with load and with the server build: several of these figures predate later dispatch fixes. Measure on the target cluster at the target load before committing to a budget. ### Planning heuristics - [GUIDANCE] Estimate wall clock as steps x per_step_overhead + sum(step work) + hops x hop_overhead, where hops are kind: playbook children. On prod today use ~4.5 s per step and ~9-15 s per hop (orchestration_overhead.measured), and re-measure: the overhead, not the work, decides the total. - [GUIDANCE] Prefer an inline python step to a kind: playbook child when the child only wraps one call: each hop measured ~9-15 s of orchestration on prod, and inlining cut a planner turn by 44%. - [GUIDANCE] Count slots, not steps, for concurrency: a parallel loop dispatches every item at once and max_in_flight does not throttle it, so N items need N free slots on the target pool or the rest queue. The prod user pool has 8 slots at minimum and 80 at maximum; the system pool has 4. - [GUIDANCE] An awaited child (return_result / return_step) or a hosted agent's playbook tool holds the parent's slot while the child needs another. Budget two slots per awaiting parent and avoid sending both to a small pool. - [GUIDANCE] Bound every loop and back-edge with a when: guard on an explicit counter: the server enforces no iteration cap, and each pass is a full step. - [GUIDANCE] Set timeout_seconds on python, shell, postgres, duckdb and rhai steps that call anything slow: those kinds have no default timeout and the worker has no per-command one. - [GUIDANCE] For kind: wasm, plan inside the defaults (30 s, 10e9 fuel, 1024 MiB) and expect one module at a time per worker process: concurrent wasm steps on one pod serialize. Read fuel actually spent from the worker's debug log; no metric reports it. - [GUIDANCE] For kind: agent, the tool-call budget (32) and child timeout (180 s) bound a hosted loop: worst case is about 32 x 180 s = 96 min of one held slot. Size max_turns and the tool list so the expected path uses a few calls, and read agent.final for the real count. - [GUIDANCE] Keep step results under 100 KiB when a downstream step reads them: larger results are spilled to the result store and every read becomes a fetch. - [GUIDANCE] Choose a pool by where the work must run (credentials, network, memory) first; then check its slot capacity against the plan's peak parallelism. Bursts above capacity on the user pool trigger KEDA scale-up, which can add node provisioning time (~60-90 s) on Autopilot. - [TARGET] Orchestrator-driven resume of a hosted agent loop after a worker dies (today the step restarts from turn 1). ### Measure the real run - event call.done: result.context: status, duration_ms, error, exit_code, child_execution_id; an agent step's result data carries usage {input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens}, turns, model, stop_reason - event command.started / command.completed / command.failed: created_at on each event: the difference is the step's wall clock including slot wait after claim - event agent.turn: turn, outcome (tool_calls | final), tool_calls [{id, name}] - event agent.tool_call: turn, tool_use_id, tool, binding, playbook, child_execution_id, is_error, duration_ms, input and result previews (2 KiB) - event agent.final: turns, tool_calls, child_execution_ids, duration_ms; or refused: tool_call_budget - agent.* events come only from the hosted loop (NOETL_WORKER_AGENT_PLAYBOOK_TOOLS=true). Token usage is in call.done result data, not on agent.* events. - metric noetl_worker_dispatch_duration_seconds (histogram, labels tool_kind): full dispatch per command: tool + case evaluation + lifecycle emits; buckets 0.01-120 s - metric noetl_worker_concurrent_dispatches (gauge): slots in use on this worker - metric noetl_worker_command_pickup_phase_seconds (histogram, labels phase (park | outcome)): claim latency split - metric noetl_worker_event_emit_duration_seconds (histogram, labels event_type) - metric noetl_worker_plugin_load_seconds (histogram, labels phase (fetch | compile)): wasm cold start - metric noetl_worker_plugin_limit_trips_total (counter, labels limit (fuel | timeout | memory)) - metric noetl_worker_result_store_put_duration_seconds (histogram): cost of spilling an over-budget result - metric noetl_worker_build_info (gauge, labels version): which worker build answered; absence of a newer metric means that build predates it - from noetl/worker@d1c9584 src/metrics.rs, served on WORKER_METRICS_BIND, else 0.0.0.0:$PORT, else 0.0.0.0:9090 - metric noetl_event_ingest_duration_seconds (histogram, labels event_type) - metric noetl_event_ingest_phase_seconds (histogram, labels phase): where inside event ingest the time goes - metric noetl_orchestrate_in_flight_executions / noetl_orchestrate_oldest_seconds (gauge) - metric noetl_execute_outcomes_total (counter, labels entry, outcome) - from noetl/server src/metrics.rs, served at GET /metrics - GET /api/executions/{id}: started_at, completed_at and up to 2000 most recent events with created_at; no duration field (subtract) - GET /api/executions/{id}/status: progress step counts only, no times - not available: no per-execution or per-step duration histogram on the server - not available: no metric for wasm fuel consumed (debug log only) - not available: no metric for token usage (call.done result data only) - not available: no per-tool memory metric ### Not yet measured (do not invent these numbers) - per-step latency of http, duckdb, postgres and mcp steps on prod - latency of a hosted LLM call inside a real playbook step - sustained executions per second on prod - worker slot occupancy over time - user-pool KEDA ramp time on the EHDB trigger, and scale-down time on prod - catalog URI fetch latency - unsaturated prod command-bus latency after 2026-07-30 ## Rules for generators - Use only enabled tool kinds. A disabled kind is rejected at registration. - A kind with a worker flag registers but fails at run time unless that worker flag is set. - `metadata.expose.*` is inert on a server without its flag; ask that server's /api/dsl/version. - Executability is reported by live workers, not by this file: 'accepted' is not 'runs'. - A workflow needs a step named `start`; step names are unique; `next` targets must exist. - Validate before registering (POST /api/dsl/validate, RFC Phase 2). - Budget slots, time, fuel and tool calls from /dsl/resource-model.json; never present a GUIDANCE or TARGET figure as measured.