12. Operations — Runtime Metrics · Graceful Drain · Readiness

Guide Home | Previous: How Session Binding Works | Next: Options and Defaults

View in another language — C++ · C#/.NET · Java · Kotlin · Node/TypeScript

What you get from this chapter

You can collect runtime metrics and connect readiness and drain to deployment procedures. Each language code tab shows the public runtime surface used in operations.

1. What It Provides

Once a service is in production, you need the following in addition to the event observation covered in the 11. Monitoring chapter.

1. Metrics — see numbers like CCU, queue depth, and request latency on a dashboard.
2. Graceful drain — clean up a node being taken down for a deployment or scale-in without dropping connected users.
3. Readiness — tell the deployment infrastructure "is this node allowed to accept new requests."

The framework provides the metric instruments and the drain procedure at host shutdown. The app wires the meter name into its collection pipeline, and composes the readiness endpoint the deployment environment calls out of the public runtime query API.

Terms that appear for the first time:

Term	One-line explanation
Meter / instrument	A language-standard metric emission unit. counter, gauge, and histogram are instruments
OpenTelemetry (OTel)	The standard for collecting metrics/traces. Exported via Prometheus and other exporters
Relocate	The operation that moves a stateful object to a compatible target and puts the host into Relocated state
Shutdown	The operation that performs bounded cleanup of local resources with no new relocation
Readiness probe	The deployment infrastructure's status check asking "is it OK to accept new requests"

2. Runtime Metrics

The framework emits every instrument through one System.Diagnostics.Metrics.Meter named "zlink.framework". The app registers this canonical meter name in its collection pipeline.

// This one line brings every zlink instrument into the app's OTel pipeline.
builder.Services.AddOpenTelemetry().WithMetrics(m => m
    // The canonical meter name the Framework emits instruments through.
    .AddMeter("zlink.framework")
    .AddPrometheusExporter());

• There's no zlink-specific metrics API. The framework uses each language's standard metrics API directly. To collect without OTel, subscribe directly to the meter name "zlink.framework" from a MeterListener.
• If no listener is attached at all, an instrument update ends on a minimal-cost inactive path. Registering an instrument without turning it on has no effect on messaging performance.
• Choosing a dashboard and exporter is the app's job. The framework doesn't bundle a scrape server.

The catalog below lists the measurements and units for MeshNode, object/STREAM, and location/fanout instruments. For detailed labels, units, and kinds, see Runtime Metrics and Complete Host Relocation Flow.

Instrument	What it measures
zlink.stream.connections.active	Active STREAM connection count (CCU)
zlink.stream.connections.opened	Cumulative STREAM connections started
zlink.stream.connections.closed	Cumulative STREAM connections closed
zlink.spot.count	Active spot count
zlink.actor.count	Active Actor count
zlink.relocation.started	Cumulative Actor/User/Instance Spot relocations started
zlink.relocation.completed	Cumulative relocation terminal results
zlink.relocation.duration	Time from prepare to the terminal phase
zlink.relocation.bytes	Encoded size of the moved relocation payload
zlink.instance_spot.activations	Cumulative Instance Spot activation results
zlink.instance_spot.activation.duration	Time from first address resolution to Ready or terminal failure
zlink.instance_spot.pending.messages	Number of messages waiting at the activation barrier
zlink.instance_spot.pending.bytes	Payload bytes reserved at the activation barrier
zlink.instance_spot.claim.conflicts	Cumulative Instance location claim conflicts
zlink.mesh_node.peers.configured	Peer count present in the descriptor
zlink.mesh_node.peers.connected	Peer count with transport connected
zlink.mesh_node.peers.ready	Peer count that passed admission and handler readiness
zlink.mesh_node.channels.ready_members	Member count available for ChannelName select-one
zlink.mesh_node.channel.selection_failures	Count of times no member was available for select-one
zlink.mesh_node.requests.inflight	Requests currently waiting for a reply
zlink.mesh_node.request.duration	Time from request submit to terminal completion
zlink.mesh_node.request.timeouts	Cumulative request timeouts
zlink.mesh_node.messages.dropped	Cumulative one-way drops with a cause confirmed by the Framework
zlink.fanout.published	Cumulative classic fanout publishes
zlink.fanout.received	Cumulative classic fanout receives
zlink.fanout.dropped	Cumulative classic fanout drops with a cause confirmed by the Framework
zlink.location.store.errors	Cumulative Redis read/write/lease failures
zlink.location.owner_lease.renew.failures	Cumulative owner lease renewal failures
zlink.location.owner_lease.renew.lateness	Owner lease renewal delay versus the scheduled time
zlink.observability.events.overflow	Cumulative monitoring/trace observer queue overflows
zlink.host.state	The current host Framework runtime state
zlink.host.core_hwm.effective_budget	Effective Core HWM byte budget fixed at startup
zlink.host.core_hwm.applied	Sum of ordinary directional queue HWMs, excluding the completion lane
zlink.host.core_hwm.accounted	Current or epoch-peak Core-accounted bytes (state=current|peak)
zlink.host.core_hwm.completion_accounted	Current or epoch-peak completion-accounted bytes (state=current|peak)
zlink.host.core_hwm.blocked_ratio	Core blocked ratio in parts per million
zlink.host.application_job_queue.limit	Effective Application Job Queue permit limit fixed at startup
zlink.host.application_job_queue.jobs	Reserved, queued, in-use, or peak permits (state=reserved|queued|in_use|peak)
zlink.host.application_job_queue.capacity_waiters	Current number of permit-capacity waiters
zlink.host.application_job_queue.capacity_waits	Permit-capacity waits in the current measurement epoch
zlink.host.application_job_queue.capacity_wait_duration	Cumulative permit-capacity wait duration in the current epoch
zlink.host.application_job_queue.pressure_state	Current running · paused state (state)
zlink.host.application_job_queue.pressure_transitions	Transitions by destination state (state=running · paused)
zlink.host.application_job_queue.pause_duration	Current/cumulative pause seconds (state=current · cumulative)
zlink.host.application_job_queue.flow_state_config_failures	Failures to apply an absolute Core flow state
zlink.host.relocation.duration	Time from starting Host Relocate to the terminal result
zlink.host.relocation.blocked	Count of host Relocate calls that ended in Blocked
zlink.host.shutdown.duration	Time from starting Host Shutdown to the terminal result
zlink.host.shutdown.forced	Count of host Shutdown calls that ended via bounded teardown

2.1 Capacity Snapshot and Measurement Reset

The host runtime's capacity snapshot exposes the Core HWM and Application Job Queue status together. Use it to correlate the fixed startup configuration and effective limits with current/peak accounted bytes, reserved/queued/in-use permits, and capacity waits. Read the exact type and member names from the relevant per-language monitoring contract.

Resetting measurements starts a new epoch without changing capacity. Current gauges and configuration stay unchanged, including pressure state and current pause duration. Each peak is rebased to its current value, and epoch waits, pressure transitions, cumulative pause duration, and flow-state configuration failures become zero. An event concurrent with reset belongs to exactly one epoch. Always-on metrics deliberately do not timestamp every job or create a per-job queue-wait histogram; record such distributions only inside a bounded performance fixture. The exact snapshot and reset rules are in Runtime state query and operational diagnostics and the metric names, units, and labels are in Runtime metrics.

3. Relocate — Moving to Another Host While Keeping State

Relocate(...) moves every User Spot, Instance Spot, and Actor alive on this host to another Serving node. It's an operation that targets the whole host, and the call itself doesn't terminate the host.

After the move the logical ids, the work not yet run and the timers remain, and only the application state is carried over by the adapter registered on the factory. What survives, the order the procedure runs in and how far it can be rolled back are covered by Relocation.

This section covers the operator's side — when it can be called, what is configured, and when it is safe to terminate.

When it cannot be called. With no eligible destination it ends as blocked without changing this host's state. While a move or a shutdown is already under way, a new call is not accepted.

3.1 The Unit That Moves

What is bundled into one unit is decided by the Spot kind and the execution mode — Relocation covers that split. For operations it matters for one reason. The larger the bundle, the wider the range that pauses at once.

↗ View larger

A User Spot whose Actors move individually is a shell that carries no state itself, so its factory can only use the policy that creates it fresh on the destination. Each member Actor's factory decides its own policy separately.

3.2 Moving-State Transfer Settings

The moving state travels over the source–target mesh connection in chunks, and four server settings keep it from crowding out ordinary messages on the same connection. The defaults are fine to start with; the runtime never adjusts them on its own, so change them based on what you observe in your deployment.

Setting	Default	Purpose and tuning criterion
RelocationPayloadChunkLimit	256 KiB	Size of one chunk. Lower it if chunk transfers intrude on the latency target of ordinary messages sharing the connection
RelocationInFlightPayloadBudget	16 MiB	Total relocation bytes that may be in flight at once on one peer connection. 0 disables it. Lower it if relocation transfers eat into ordinary message bandwidth; raise it if host move throughput is capped by this budget
RelocationNodeInFlightPayloadBudget	0 (disabled)	Node-wide cap on concurrent transfers. Set it only when a node with many peer connections needs its total occupancy bounded
RelocationCutoverWaitTimeout	1,000ms	How long the target waits for a cutover retransmission. If the cutover_timeout counter is nonzero, adjust it to your deployment's round-trip time

When the budget is full, a new relocation unit waits before its seal, and the Actor/Spot keeps processing messages normally while it waits — no payload size is ever too large to start because of the budget. Internal protocol details such as the chunk format and verification rules are covered by Relocation Flow.

3.3 SafeToShutdown — When It's Safe to Terminate

Relocated means the source finished sending its cutovers, not that every caller caching the old route now points at the new owner. The source runtime publishes the SafeToShutdown state in its own runtime status once every unit's Message Follow can end and the cutover retransmission window has closed. A deployment orchestrator should confirm Relocated, then observe this state before calling Shutdown — terminating before it's published is allowed, but the remaining follow routes disappear and requests from callers still caching the old route can end in Unavailable. State queries and change observation follow Runtime Status Queries And Operational Diagnostics.

The relocation window is observed as three separate segments — source stop (from seal to cutover submission), target resume (from the target's owner confirmation to dispatch opening), and route convergence (from cutover submission until the Message Follow route can be removed). The number of fallbacks that proceeded unverified after waiting for a cutover is published as the cutover_timeout counter. Metric names, units, and labels are owned by Runtime Metrics.

4. Shutdown — Terminating Without Moving

Shutdown(...) terminates this host. Unlike §2, it doesn't move state to another node.

Calling it starts no new relocation; work already in progress either finishes within the given deadline or is settled as a failure. It then notifies the Entry, User, and Instance Spots' OnClosingAsync with the HostShutdown reason, and once that callback finishes, cleans up scope, authority, session, and topology resources. If no deadline is given, it's 30 seconds.

The state of any Spot cleaned up here doesn't survive. If your deployment automation needs to keep state alive while taking a host down, call Relocate(...) before shutting down, confirm the result is Relocated, and only then move on to this call (the example in §4). When possible, also confirm the SafeToShutdown publication (§2.3).

A Spot's lifetime is independent of any request. A User/Instance Spot isn't closed just because an ordinary request finished. Likewise, preparing a nonexistent Instance Spot never starts from a separate address or manager create — only from attaching Instance intent to a SpotId direct call (Spot §5).

5. Wiring Operational Calls and Readiness

The two operations above don't happen automatically. The application calls them directly on the framework runtime. This interface is a DI singleton that owns host maintenance.

The order used in deployment is "move first, and shut down only if it succeeded." Confirming Relocated and then observing the SafeToShutdown publication (§2.3) before terminating avoids failures for callers still caching the old route.

var runtime = app.Services.GetRequiredService<IZLinkFrameworkRuntime>();
var result = await runtime.RelocateAsync(
    new ZLinkFrameworkRelocationOptions
    {
        Mode = ZLinkFrameworkRelocationMode.RollingUpdate,
        // Uses only eligible nodes on the specified new version.
        TargetApplicationVersion = 12,
        Deadline = TimeSpan.FromSeconds(25)
    },
    cancellationToken: ct);

if (result.Outcome == ZLinkFrameworkRelocationOutcome.Relocated)
    await runtime.ShutdownAsync(TimeSpan.FromSeconds(10), ct);
else
    Console.Error.WriteLine($"host relocation blocked: {result.Reason}");

PlannedMaintenance uses only a target on the same application version as the source. RollingUpdate requires a TargetApplicationVersion greater than the source's, and uses only a target exactly at that version. If there's no eligible target, it waits until the deadline and then returns Blocked/TargetUnavailable. Cancellation ends only that waiter — a shared lifecycle operation that's already started keeps running.

A readiness endpoint combines the readiness of the host framework runtime with that of any component runtime required by the application.

app.MapGet("/healthz/ready", (IZLinkFrameworkRuntime runtime) =>
    runtime.Status.IsReady
        ? Results.Ok()
        : Results.StatusCode(StatusCodes.Status503ServiceUnavailable));

In a Kubernetes deployment, the setup looks like this.

# readiness probe → /healthz/ready — excluded from new-traffic targets the moment Draining starts
# preStop hook + terminationGracePeriodSeconds >= drain deadline — allows time for auto-drain to finish

5.1 Calling It Again or Overlapping Calls

Deployment automation retries on failure. So what happens when you make the same call twice is defined by contract.

Situation	Result
Relocate called again with the same mode while one is in flight	Shares the deadline with the first operation. The later call doesn't extend the deadline
Relocate called again with a different mode while one is in flight	Blocked with no wait — meaning an operation is already in progress
Relocate called again after Blocked	Blocked isn't stored, so it re-checks the host's condition from scratch. This is the only result where retrying is meaningful
Relocate called again after Relocated	Returns the original success result as-is. Doesn't move again
Shutdown called again while one is in flight	Shares the same operation and stores the terminal result
Shutdown called again after Stopped	Returns the stored result
Relocate called while starting up, or in an error/stopped state	Blocked without touching admission

A caller's cancellation ends only that call. The shared operation itself isn't cancelled.

Shutdown is never blocked. It proceeds even with no target, insufficient capacity, or no Relocation Store. So if Shutdown gets confirmed while something is waiting on Relocate, the waiter ends in Blocked — this is why you must keep to the order "move first, confirm success, then shut down."

If Shutdown doesn't finish within its deadline, it performs only bounded cleanup and ends in a forced-termination result. A deadline overrun and a callback failure are distinguished by different result values.

5.2 What Stays Alive During a Transition

Relocating, Relocated, and Draining aren't "accepting nothing" states. Only starting something new is blocked — what's already accepted is processed through to completion.

	Relocating	Relocated	Draining
Selection by channel name	Excluded from new selection. The existing owner path is kept	Excluded from new selection	New admission closed
A request that directly names a node	Accepted until the unit seals	Not accepted	Ends with the shutdown result
Spot/Actor creation and join	Rejected	Rejected	Rejected
STREAM	New binding excluded. An existing session is handled via a barrier	New binding excluded	New session not accepted
An already-accepted request	Ends exactly once, via reply, error, timeout, or shutdown	Same	Same

Monitoring or observer callbacks never hold up termination. Even if code observing status runs for a long time, maintenance never waits for it.

6. Location Readiness and Operational Queries

Operational code uses the location readiness API to check whether a needed peer is Ready. It uses the location runtime query for overall status and paged topology.

app.MapGet("/ops/location", async (
    IZLinkLocationReadiness readiness,
    IZLinkLocationRuntimeQuery query,
    CancellationToken ct) =>
{
    var status = await query.GetStatusAsync(ct);
    var page = await query.ListTopologyAsync(
        new ZLinkLocationTopologyFilter(MeshName: "play"),
        new ZLinkPageRequest(PageSize: 100),
        ct);

    var objectPeerReady = await readiness.IsPeerReadyAsync(
        "play",
        ZLinkLocationRole.Spot,
        cancellationToken: ct);

    return Results.Ok(new
    {
        status.StoreHealthy,
        status.OwnerLeaseHealthy,
        objectPeerReady,
        topology = page.Items
    });
});

The operational query returns only health and topology intended for human inspection. Store keys, authority versions, owner tokens, and relocation records are internal Framework information and aren't returned. NodeRid is used only to map operational info back to the actual transport node.

7. MeshNode Runtime Control and Observation

A MeshNode registered with AddRouteMesh is operated through two DI singletons.

Runtime options. Values that can be changed while serving are below. The remaining socket options (HWM, timeout) are exclusive to ConfigureRouterSocket() before startup.

var meshOptions = app.Services.GetRequiredService<IZLinkRouteMeshRuntimeOptions>();
// Excludes it from new object placement
meshOptions.Mesh("game.room").PlacementWeight = 0;
// Excludes it from new channel select-one
meshOptions.Channel("game.room").Weight = 0;

The two weights are independent and take effect on new selections while running. Placement weight is used only for Actor/Spot create and relocation target selection. Channel weight is used only for selecting new select-one targets for that server membership. Looking up an unregistered mesh or membership is a configuration error.

Status query — RouteMesh runtime. It provides one consistent snapshot and an ordered component event stream for one mesh. Host termination is owned by the framework runtime.

var meshRuntime = app.Services.GetRequiredService<IZLinkRouteMeshRuntime>();

// The immutable current status of nodes/peers/channels
var status = meshRuntime.GetStatus("game.room");
var ready = status.IsReady;

await foreach (var observed in meshRuntime.ObserveAsync("game.room", cancellationToken: ct))
{
    // observed.Status carries state/peer transitions in Sequence order.
    // observed.Loss is how many this observer missed — see the shared rule in 11-monitoring §2.
}

8. Host Lifecycle

The Framework runtime is tied to the host's start/stop as its lifecycle service. The channel/SPOT/STREAM runtime is created based on the roles registered at startup, and cleaned up at shutdown.

↗ View larger

• Configuration phase — finish every declaration before app.Run(). A bad configuration is rejected as an exception at host startup.
• Stopping — once the host shutdown signal arrives, it goes down in the order hosted service stop() → channel/SPOT/STREAM runtime cleanup.
• Fold background work into the same lifecycle using the host's standard lifecycle service.

8.1 Observing Status

Host Relocate/Shutdown state transitions are observed through the framework runtime's bounded status stream. The per-MeshName runtime provides a component snapshot, but doesn't create a separate termination authority or partial-drain operation.

var runtime = app.Services.GetRequiredService<IZLinkFrameworkRuntime>();

await foreach (var observed in runtime.ObserveAsync(cancellationToken: ct))
{
    // Records the whole-host state, effective intent, and terminal outcome in sequence order.
    var hostEvent = observed.Status;
    logger.LogInformation(
        "host lifecycle: {State} {Relocation} {Termination}",
        hostEvent.State,
        hostEvent.RelocationResult,
        hostEvent.TerminationResult);
}

Observe the host lifecycle states as-is (preparing, serving, relocating, relocated, draining, stopped, error). The notation follows the language. The status's relocation/ termination results must match that operation's terminal result. To view it as numbers, use the zlink.host.* instruments in Runtime Metrics.

8.2 Restarting with the same RoutingId

A server restarted immediately after forced termination with a fixed RoutingId can encounter the previous process's active owner lease. Publishing the descriptor for the same MeshName and RoutingId returns REJECTED_CONFLICT and fails host startup, so the host start call of every language ends with a failure. Check the public runtime status after successful startup instead of treating an open port as readiness.

Deployment should confirm the previous server's normal termination result before starting its replacement. After forced termination, start a new process once the previous owner lease has expired. The default owner lease TTL is 15 seconds and can be configured. Framework does not retry the conflict or select another RoutingId within the same startup operation. Also check that two processes are not using the same fixed RoutingId concurrently.

Normal shutdown cleans up the host's descriptors and owner lease. Forced termination may prevent that cleanup, leaving the active claim until lease renewal stops and the lease expires. Check the termination result to confirm that cleanup finished within the shutdown deadline. Process exit or a shutdown call alone does not guarantee that an immediate restart can claim the same RoutingId.

9. Related Documents

• Runnable verification examples for this chapter's contract: 13. Interface Catalog chapter §7 — the verification class FrameworkRuntimeContracts
• The formal contract: Complete Host Relocation Flow · Runtime Metrics
• Status observation and diagnostics: the 11. Monitoring chapter
• The Spot where the application decides the relocation boundary: 06-spot §7