Skip to content
PhramePhrame

Placement & scheduling

Placement & scheduling

Section titled “Placement & scheduling”

Declaring that something should exist is not the same as saying where it runs. The reconciler answers the second question — and writes the answer down.

Orchestration — desired state, actual state, and the loop between DESIRED — what should be running Architect /Operatorauthor intent GitLabphrame-configversioned config-syncvalidate →retained MQTT MQTT brokerdesired topics(retained) commit webhook retained orchestration-reconcilerchooses a node · commits the placement writes node.jsononly to a branch,never to a tag ACTUAL — what is really running node-controlleron every host Dockerinput-proc · mixersTAMS · egress actual stateheartbeat ·host stats mirrors →web-ui converge publish aggregate desired state, per node Git is the source of truth for desired state — a commit is a deployment, and the running system is reported back, not assumed

  • Project: services/orchestration-reconciler (Node / TypeScript)
  • Decides: what containers exist, and on which node
  • Does not: start anything — that is the node controller

The split worth remembering

Section titled “The split worth remembering”

The reconciler decides what should exist, where. The node-controller makes local Docker match.

Neither reaches into the other’s job. That is why the scheduler can be restarted without disturbing a single running container, and why a node can drop out and rejoin without the cluster having to be told.

What it does

Section titled “What it does”

The reconciler watches two things over MQTT — the desired instances, and the live state of the nodes — and for any instance with no running container it:

  1. Picks a node (see below).
  2. Commits a container entry into that node’s node.json, on the active config branch. From there the normal path applies: config-sync publishes the changed file to the node’s retained topic, and that node’s controller converges onto it.
  3. Emits a progress event on the instance’s events topic — scheduled, placed, unschedulable, or error — so a UI can show what happened to a request rather than only its end state.

The placement is therefore not held in the scheduler’s memory. It becomes a commit, in the same version-controlled record as the intent that caused it, and it survives the scheduler being restarted, replaced, or removed entirely.

The whole cycle is idempotent: the plan is re-derived from cluster state on every tick rather than accumulated, so running it again changes nothing unless reality has changed.

Choosing a node

Section titled “Choosing a node”

Eligible nodes are those whose last heartbeat is within the offline threshold — a node that has gone quiet is simply not a candidate, without anything having to mark it down.

Among those, the reconciler picks the least loaded, where load is the sum of CPU and the memory-used ratio reported by the node’s own host statistics. Ties break on node id, which keeps the choice deterministic: the same cluster state produces the same placement, so drift detection and the commit that follows it cannot disagree.

The tag guard

Section titled “The tag guard”

Before writing anything, the reconciler checks whether the active config ref is a branch. If the active ref is a tag or a commit SHA, it does not write.

This is what makes a pinned release genuinely pinned. A deployment running from a tag cannot be quietly mutated by an automated placement decision, so “we are running v1.4.0” stays a true statement about the whole system rather than about its inputs only. Moving a pinned deployment is a deliberate act — publish a new ref (see Configuration & GitOps).

What it can place

Section titled “What it can place”

The reconciler builds container specifications for far more than inputs — it constructs them for input-processor, MinIO and Postgres, the TAMS server, recorder, player and flow-manager, the HLS encoder, SRT / WebRTC / NDI egress, phrame-webrtc-session, and the vision-mixer components (compositor, switch, overlay, scaler, 2D DVE). The specs themselves come from @phrame/orchestration-contracts, so the scheduler and Architect build the same container from the same definition.

It also registers gateway routes for the instances that need to be reachable — file-server and reverse-proxy entries via the Caddy admin API — so an instance becomes addressable as part of being placed, rather than in a separate step somebody has to remember.

A known limitation, recorded

Section titled “A known limitation, recorded”

phrame-webrtc-session needs a distinct UDP port range per instance for ICE. Because the plan is re-derived on every tick rather than stored, ranges are assigned by hashing the instance UUID into a fixed pool of non-overlapping slots — the same UUID always lands on the same slot, so plan and commit never disagree.

Two UUIDs can hash to the same slot. With enough concurrent instances that becomes likely, and the collision surfaces as an OS-level port bind failure with no advance detection and no reassignment. A real allocator — tracking live assignments and reassigning on conflict — is known follow-up work, not something the current implementation does.