Framework Error Model¶

Foundation topic index · Spec index · Previous: 06. Framework API · Next: 08. Layering Boundaries And Identifiers

Defines the shared ErrorKind that the Framework delivers to an Application when Send, Request, or a lifecycle operation fails, the completion/failure boundaries of Send and Request, and the rules for deciding whether to retry.

1. Scope¶

This document defines the shared error the Framework delivers to an Application when Send, Request, or a lifecycle operation fails. An error represents a failure category an Application must distinguish, not an internal function or state-machine step.

The Framework does not provide an indication of whether to retry alongside an error. An Application checks the operation's completion condition, idempotency, and business state before deciding whether to start a new operation.

2. The Shared `ErrorKind`¶

All five languages project the following names and numbers using that language's enum naming convention. The value 0 is also a valid error value.

Value	Kind	Meaning
0	`NotFound`	The Actor, Spot — a logical instance with an address and state — handler, route, or target doesn't exist.
1	`AlreadyExists`	The same identity or registration already exists.
2	`TypeMismatch`	The stable type differs from the requested Application type.
3	`NotConfigured`	The required role, handler, or Store isn't registered.
4	`Rejected`	A Framework admission, filter, or runtime policy rejected the operation without producing a typed result.
5	`Unavailable`	The target, route, Store, or worker is currently unavailable.
6	`DeadlineExceeded`	The operation did not complete within its set deadline.
7	`ShuttingDown`	The runtime isn't accepting a new operation.
8	`ProtocolError`	The wire, payload, or reply contract could not be processed.
9	`InvalidOperation`	The operation can't run in the current object, session, or runtime state.
10	`DataLost`	The published Relocation payload is missing or failed validation.
11	`InternalFailure`	A Framework failure that can't be expressed by the categories above.

Generation, owner fence, moving phase, worker-queue state, and a Relocation processing step are internal causes. When an Application doesn't need to choose a separate response, they are recorded in logs and traces rather than exposed as a new public kind.

The five server packages and the HTTP client package share these 12 kinds. A per-language interface document defines only the enum name and the exception/result representation — it does not add a kind or a retry boolean.

2.1 Wire Failure Codes and `ErrorKind`¶

Each failure code of a Framework failure a remote node sends (a non-none value of framework-error-code in the service wire schema) maps to one ErrorKind. Whether the code comes with a terminal result (typedFrameworkFailure) or alone, as in a relocation failure (command 53), the receiver classifies it with the table below, regardless of the command or path it arrived on. To send an ErrorKind on the wire, use the failure code in that kind's row that names the cause; if there is none, use the representative code. In a reply that carries a terminal result, ShuttingDown is sent as terminated and the representative of InvalidOperation as invalidState, both with no failure code, and the combination follows the schema's terminal-failure-integrity. NotConfigured has no wire form and is sent as requestFailed. In a failure that carries only a code with no terminal result, such as a relocation failure (command 53), ShuttingDown is sent as routeNotConnected (to the receiver, an Unavailable node it can't use for this operation), and an InvalidOperation with no cause code is sent as requestFailed.

ErrorKind	Representative code	Other codes received as the same kind
`NotFound`	`requestTargetNotFound`	`actorRouteNotFound`, `spotRouteNotFound`, `handlerNotFound`, `routeHandlerNotFound`, `actorDispatchHandlerNotFound`
`AlreadyExists`	`actorAlreadyExists`	—
`TypeMismatch`	`spotTypeMismatch`	`actorTypeMismatch`
`Rejected`	`requestRejected`	`actorCreateRejected`
`Unavailable`	`routeNotConnected`	`workerQueueFull`, `actorLocationStale`, `spotMoving`
`DeadlineExceeded`	`workerTimedOut`	—
`ProtocolError`	`requestProtocolError`	`payloadDecodeFailed`
`InvalidOperation`	none (`invalidState`)	`actorSessionNotBound`, `spotGenerationStale`
`DataLost`	`relocationDataLost`	—
`InternalFailure`	`requestFailed`	`workerFailed`, `actorCreateFailed`, `spotCreateFailed`

3. Errors Checkable Before the Call¶

A problem that can be checked immediately at the call site — such as an invalid argument or an already-closed handle — is delivered as that language's standard argument or invalid-operation error. A startup configuration error is also delivered as a per-language configuration exception. Neither of these is turned into a remote error reply.

A Framework failure discovered while waiting on outbound queue acceptance, route resolution, or a remote reply is delivered as a per-language Framework exception or as a result's ErrorKind. A binding NOT_FOUND maps to NotFound when a named target is absent. A pending send whose route was removed ends with Unavailable as Submit and completion §7 states. Channel selection results follow the Framework API channel selection result. Other results are classified by the meaning of the Core result on that call path.

4. `Send` Completion and Failure¶

The normal Send completion boundary is defined by Submit and completion §4.

Condition checked before completion	Result
The logical target or route doesn't exist (including TargetNotFound)	`NotFound`
The connection or current owner is currently unavailable (including RouteNotConnected)	`Unavailable`
The Classic fanout publisher's queue doesn't accept the message before the send timeout	`DeadlineExceeded`
The runtime has stopped new admission	`ShuttingDown`

Even if target activation, admission, or handler execution fails after Send completes, that failure does not change the result of the already-completed call. The Framework records it in metrics, logs, and the message-flow trace. The resubmission boundary is defined by Submit and completion §5.

5. `Request` Completion and Failure¶

Request completes normally when it receives a typed reply. When a normal reply can't be produced, it completes exactly once with one ErrorKind (§2). §2.1 decides the kind of a failure a remote node sent. The cases this node decides itself are the following.

NotFound if the target or handler doesn't exist.
Unavailable if the route, connection, or current owner is unavailable. It also means the selected receiver could not admit this operation after losing first-message Reserve.
DeadlineExceeded if the reply isn't received within the deadline.
ProtocolError if the wire, payload, or reply type can't be processed.
ShuttingDown if the runtime is shutting down.
InternalFailure for a Framework execution failure that can't be expressed by the kinds above.

A full Framework-owned queue is not an error. The work waits until room appears. The only way to slow what is coming in is the PAUSED state of §6.

Every queue is the same in this respect. The Spot/Actor queues in the same runtime, the worker queue, a slot to hold the reply, the in-progress call table and the completion slot all behave this way. While waiting, the work still holds its host permit, so the number of permits in use rises and PAUSED goes out at the threshold.
Another node's queue is the same. When the target falls behind, its PAUSED and Core's byte limit slow this side's send. Errors are not split by who owns the queue.
Observation is outside this rule. A status snapshot, a trace, or an observer notification is not work to be done but a value that reports what happened. To keep a slow observer from holding work back, the newest may be kept or several coalesced. That is not throwing work away.
Having nowhere to place something is different. If no node can host the Spot, slowing down does not produce one. That case completes with Unavailable (Spot Actor, Spot address messaging).
The Message Follow relay queue — the mechanism that forwards a message that arrives at the previous owner node, on behalf of the new owner, after a relocation — and the relocation ingress hold have no record-count or byte bound defined by relocation itself.
The amount retained in this queue or hold does not by itself produce an error. While retained, the record keeps holding its host permit, so as more pile up the number of permits in use rises and PAUSED goes out at the threshold.
The negotiated limit for one message and the limits set by transport, the deadline, and cancellation still apply.
Once retained work is admitted to an ordinary application execution lane, that lane's reservation applies, but this reservation is not used as a storage bound for the relay queue or hold.
If one of those limits causes a failure, the error follows the rules above (Spot Actor, Location runtime).

Cancellation is delivered as that language's cancelled awaitable. DeadlineExceeded and cancellation mean the caller has stopped waiting for the reply. They don't mean the remote handler never ran, and a reply that arrives late does not produce a second result.

6. Typed Results and `Rejected`¶

For an operation whose contract has Accepted and Rejected, such as Actor create or join, the Application callback's decision is returned as a typed result. In this case, Rejected is not a Framework exception.

The shared ErrorKind.Rejected is used only when a filter, admission, or runtime policy without a typed result rejects the operation. Not every result an Application's business rule rejects is converted into a Framework exception.

7. Retry Judgment¶

A public exception, error object, or typed failure does not carry a retry hint such as RetryAdvice, isRetriable, or retriable. Even for the same ErrorKind, the likelihood that the operation already ran and the impact of a duplicate can differ.

To start a new operation, an Application directly checks the following.

Check the previous operation's completion condition and whether it could have run remotely.
Check whether the operation is idempotent, or whether an idempotency key prevents a duplicate effect.
Re-query business state if needed, then start the new operation.

The binding operation's HWM retry and Framework resubmission boundary are defined by Submit and completion §5.

8. Application Job Queue Saturation¶

For the Application job queue — the shared supply-permit queue that a Framework host instance holds until an application callback actually starts — a manual queue value outside 1..2,147,483,647, or a calculation overflow, is a configuration error before socket bind.

A runtime shared-cap shortage is a cancellable wait, not a public error, typed reject, or drop reason.

A per-execution-object FIFO behaves the same way. When it is full the work waits, as §5 defines, and does not end with an error. Execution contract §7 owns FIFO scope.

9. Verification Requirements¶

Verification uses only the public surface — each language's ErrorKind enum and its numeric values, the return value/exception of Send/Request, and typed Rejected results. Each item below leads to one contract test.

ErrorKind values and count

Each language's 12 ErrorKinds and numbers match.

Send completion boundary

Verify Send completion and the result after a remote failure against Submit and completion §4.

Request completion boundary

Request doesn't produce a second result from a reply that arrives late after a timeout or cancellation.

Bounded queue errors

No Request ends because a Framework-owned queue had no room, whether that queue is in the same runtime or on another node.
When no node can host the Spot, the call ends with Unavailable.

Typed Rejected distinction

A typed Rejected result is distinguished from an ErrorKind.Rejected exception.

Absence of a retry hint

The public error surface (exception, error object, typed failure) carries no retry hint.