Skip to content

Monitoring

Guide Home | Previous: Location | Next: The Execution Model

View in another language — C++ · C#/.NET · Java · Kotlin · Node/TypeScript

What you get from this chapter

You can read what is ready right now, receive that state whenever it changes, and record where one message ended up. The code is the minimal call on each language's observation surface from the tutorial README in the examples repository; follow its Download, Build, and Run sections to read the processes' state and records.

The earlier chapters covered registering and calling. Once it is running, something else is needed — whether the connections are ready, which peer dropped out, and where a message failed. No amount of reading handlers answers that. The framework exposes them on public surfaces.

There is no surface that delivers runtime events to a handler. Start an operational endpoint with a status lookup, and add a subscription only when it must react to state changes. Use diagnostic records for an individual message's path and the meters in Operations and Lifecycle for dashboard values.

1. Start by Reading Operational State

↗ View larger

This flow lets an operational endpoint answer from the current state while observing changes and message handling separately where they are needed.

2. Reading the Current State

A lookup returns one value as of the call. Use it to answer an operational endpoint or to check something once.

ZLinkMeshNodeSnapshot initial = runtime.snapshot(ZoneWorldNames.MESH);
registry.applyLiveRoutingIds(readyRoutingIds(initial));

Read the readiness and the state value together. Readiness alone does not say what to do, and the reason for not being ready comes separately — no eligible peer, a stalled record store and a drain in progress each call for a different response.

The value carries only the peer's identifier, its current state and why it is unusable. Reconnect attempt counts and internal socket state are not part of the public contract.

3. Subscribing to Changes

A subscription gives the complete value after each change, not an event carrying only the fields that changed. If a comparison with the previous value is needed, the subscriber keeps it.

observation = runtime.observe(ZoneWorldNames.MESH, 32);
observation.subscribe(
        new Flow.Subscriber<ZLinkObservedStatus<ZLinkMeshNodeSnapshot>>() {
            @Override
            public void onSubscribe(Flow.Subscription subscription) {
                subscription.request(Long.MAX_VALUE);
            }

            @Override
            public void onNext(ZLinkObservedStatus<ZLinkMeshNodeSnapshot> observed) {
                registry.applyLiveRoutingIds(readyRoutingIds(observed.status()));
            }

            @Override
            public void onError(Throwable error) {
                System.out.println(
                        "node observation error mesh="
                                + ZoneWorldNames.MESH
                                + " detail="
                                + error.getMessage());
            }

            @Override
            public void onComplete() {}
        });

Values can be missed. A slow receiver skips intermediate values past the retention limit. The missed count comes with each item and splits into those coalesced away and those that can no longer be received past the limit. The first means the latest value did arrive; the second does not.

The order of two values is decided by the sequence number carried on the item. The time that comes with it is for display.

A lookup is for reading once; a subscription is for recording or reacting to state transitions. Only a subscription skips intermediate values.

A subscription stays open until it is cancelled, so run it somewhere tied to the host's lifetime. That place differs by language — a background service, the observation object's lifetime, a cancellation signal.

4. Setting the Diagnostics Level

Where and how one message ended is what diagnostics record. The levels are as follows.

options.configureDispatch().messageFlow(ZLinkMessageFlowLogMode.NORMAL);
Level What it records
Off Nothing. No trace item and no string is created
Errors (default) Dispatch failures and backpressure
Normal The above plus major transitions such as receive, dispatch and completion
Detailed The above plus byte sizes and elapsed time

Keep it at the errors step in production and raise it only when needed. The detailed step records per message, so in a high-throughput stretch it becomes load in itself. A logging filter that only discards the output is not the same as the off step — the cost of producing it remains.

The value that ties one request to its reply, and the value that ties that request to the calls it started, are both carried on the record. Use them to see pieces from several nodes as one.

Where the records are stored and where they are exported is owned by the application's existing configuration. The framework exposes neither an observation callback nor a file path; it writes to the standard providers the application has configured. A failing provider call does not change the result of the original message processing.

5. Common Problems

  • I want runtime events delivered to a handler — there is no such surface. State changes come from Subscribing to Changes, and per-message results from Setting the Diagnostics Level.
  • The subscription gives nothing — it is quiet when that name has not changed. If the current value is needed, read Reading the Current State first and then start the subscription.
  • I expect a health endpoint — the framework creates no HTTP endpoint. Wire the readiness into the application's existing endpoint — Operations and Lifecycle is that place.
  • I want to see what lives on which node — use the lookup in Location and the topology query in Operations and Lifecycle.
  • I want to know about messages with no handler — at the errors step or above they are recorded as dispatch failures. A request comes back as an error reply while a send is dropped quietly, so the send side shows up only in diagnostics — How Channels Work covers that difference.

6. Confirming the Result

After running the Server and Client, confirm the readiness and peer state through a status lookup, then raise diagnostics to confirm the sent message's receive, dispatch, and completion records.