9. Workers

Role in the architecture

The engine runs inside Azure with one identity and a fixed set of activity types. Workers are the part of the architecture that runs outside those limits. A worker is a service that implements one or more step types and runs wherever the step needs to run: in Container Apps, on a VM, or on an on-premises server. Workers are built and maintained by the platform team. A workflow uses one by naming its step type, for example parse_pdf. The definition carries no information about where the step runs.

A worker:

1. Polls the Task API for tasks of the types it implements.
2. Performs the task.
3. Posts the result to the Task API.

The engine does not call the worker. The worker calls the engine, and requires only outbound HTTPS to the gateway and an identity. A step can therefore run in a location the engine cannot reach, using a credential the engine does not hold.

Workers are grouped by code area rather than by workflow. An illustrative grouping:

Worker	Example step types	Reason for a worker
Document worker	parse_pdf, extract_excel, classify_document	Extraction rules specific to the organisation, with no API
Data platform worker	reconcile_ledger, bulk_export	Runs for minutes over the data platform
Connector worker	fetch_bank_feed, lodge_return	The credential must stay in its own service

When a step becomes a worker step

A step is an HTTP, CALL_TOOL, Logic Apps connector, or Document Intelligence call whenever it can be. It becomes a worker step type only when one of these holds:

Condition	Example
No API exists for the logic	Spreadsheet extraction with organisation-specific rules
The system cannot be reached from Azure	An on-premises file share or desktop-only software
The step runs for minutes	A month-end reconciliation across all clients
The credential must not be held by the engine	A bank connection whose secret stays in its own service

A step that only calls an HTTP API is an HTTP task, not a worker. A step that contains business rules is a tool on the action service, so the rule exists once and every caller shares it.

Task definition

Every worker task type needs a task definition, registered through POST /api/metadata/taskdefs:

{ "name": "parse_pdf",
  "timeoutSeconds": 300, "responseTimeoutSeconds": 120, "pollTimeoutSeconds": 3600,
  "timeoutPolicy": "RETRY",
  "retryCount": 2, "retryLogic": "EXPONENTIAL_BACKOFF", "retryDelaySeconds": 30,
  "concurrentExecLimit": 10,
  "rateLimitPerFrequency": 100, "rateLimitFrequencyInSeconds": 60,
  "inputSchema": { }, "outputSchema": { } }

Field	Meaning
timeoutSeconds	From queueing to completion.
responseTimeoutSeconds	From a worker taking the task to posting a result. Also the queue lock duration.
pollTimeoutSeconds	From queueing to the first poll. Fires when no worker is running.
totalTimeoutSeconds	Across all attempts.
timeoutPolicy	RETRY (try again if attempts remain), TIME_OUT_WF (fail the run), ALERT_ONLY (record a metric, keep waiting).
retryCount, retryLogic, retryDelaySeconds, backoffScaleFactor	The retry schedule. Logic: FIXED, LINEAR_BACKOFF, EXPONENTIAL_BACKOFF.
concurrentExecLimit	Maximum tasks of this type in progress at once.
rateLimitPerFrequency, rateLimitFrequencyInSeconds	Maximum tasks handed out per window.
inputSchema, outputSchema	JSON Schema, checked when set.

Queues

Each worker task type has a Service Bus queue named after it. The pipeline creates the queue when the task definition is registered. When the interpreter reaches a WORKER task, it sends a message {taskId, workflowId, taskRef, input, attempt} to that queue and waits.

Situation	What Service Bus does
A worker takes a task	The message is locked for responseTimeoutSeconds. Other workers cannot see it.
The worker posts a result	The message is completed and removed.
The worker dies mid-task	The lock expires and the message returns to the queue.
A retry with delay is needed	The message is re-sent with a scheduled delivery time.
Retries are exhausted	The message moves to the dead-letter queue. The engine marks the task FAILED.
A worker asks to be called back later (callbackAfterSeconds)	The message is re-sent with that delay.
A start request maps the type to a domain	The message goes to queue {domain}-{type} instead. Workers polling with that domain take it.
The run has high priority	The message goes to queue {type}-high. Workers poll it first.

The Task API enforces concurrentExecLimit and rate limits when handing out tasks: at a limit, it returns the message to the queue with a 60-second delay.

Worker loop

forever:
    tasks = GET /api/tasks/poll/batch/{type}?count={free slots}&timeout=5000
    if none: wait (1 ms, doubling up to the poll interval); continue
    for each task:
        result = handler(task.inputData)
        POST /api/tasks {taskId, workflowId, status: COMPLETED, outputData: result}

Two packages implement the loop; a worker adds only the handler per step type:

C#:

[WorkerTask("parse_pdf")]
public ParseResult ParsePdf(ParseInput input) => ...;
// WorkerHost.Run() polls forever

Python:

@worker_task("parse_pdf")
def parse_pdf(url: str) -> dict:
    return {"text": ..., "pages": ...}
run()   # polls forever

Both packages support a thread count, a poll interval, callbackAfterSeconds, and posting log lines. A Functions app with a Service Bus trigger is an alternative worker shape with the same result contract.

Hosting

Default: Azure Container Apps, one app per code area, not per task type. A document worker, for example, would implement parse_pdf, extract_excel, and classify_document in one app. The app runs at least two replicas and scales on queue depth, down to zero when idle. Container Apps Jobs host batch workers that run to completion.

Any other host works if it has outbound HTTPS and an identity with the worker role listing its task types (page 15).

Liveness

Every poll is recorded with the worker id and time. GET /api/tasks/queue/polldata?taskType= shows the last poll per type. A type with no poll in the last 10 seconds is reported as having no active worker, and the engine records a warning when it queues work for that type.