Skip to main content

Monitoring, the Reconciler & Engine-Stop Failures

Most of the platform's operational health is self-maintaining. There's a health endpoint for liveness, telemetry for tracing behaviour, and a background reconciler that quietly repairs the one gap that matters most: an execution the platform thinks is running that the engine has actually finished.

Health and telemetry​

The API exposes /health (used by the deploy step and easy to curl from inside the container). Behaviour is traced through telemetry with {domain}.{event} naming — useful domains include workflow_execution, workflow_engine, and reconciler. When something looks stuck, the logs for those domains are the first place to look.

The reconciler​

The API runs a reconciler on a loop — always enabled, no configuration required. Every 120 seconds (WORKFLOW_EXECUTION_RECONCILER_INTERVAL_SECONDS) it scans running executions and reconciles platform state against engine state:

  • If an execution has an external_execution_id and the engine has finished but no callback ever arrived, the reconciler marks it failed_transient. (Engine success does not become platform completed — the platform never received the result, so it will not fabricate one. The reconciler never writes result_payload or calls /complete.)
  • If limits are breached and the engine is still running, it issues a remote stop.
  • If external_execution_id is missing, it can't query the engine — it leaves the execution running and logs "operator intervention required".

When multiple API instances share a database, only one runs the tick at a time, enforced by a Postgres advisory lock (918600, override via WORKFLOW_EXECUTION_RECONCILER_ADVISORY_LOCK_ID only when independent deployments share one database).

Operator intervention​

The "operator intervention required" log means an execution breached its limits but has no engine ID to act on. Resolve it by hand: check the n8n UI for the execution, and if it's finished, terminate it via the platform; if it's still running, stop it in n8n first.

Diagnosing an operator-terminate 502​

Machine callbacks (/complete, /terminate) stop the engine after returning 200 — a failed stop there is just logged, and the reconciler is the safety net. The one path that surfaces a stop failure synchronously is an operator terminating a running execution from the UI: that stop is blocking, so if it can't be confirmed you get 502 ENGINE_STOP_FAILED and the execution stays running. To diagnose:

  1. check stop-attempt logs (workflow_execution_log rows with node_name='system', worker_type='internal');
  2. verify the engine's api_base_url is reachable from the API container;
  3. check the per-engine API key and stop timeout;
  4. read the failure reason in the 502 body — timeout, rejected, network_error, or unconfirmed_response — each points at either connectivity, the API key, or engine-version compatibility.

Once fixed, the operator retries; if the engine already stopped, the retry returns already_stopped, treated as success.