Why Reliable AI Work Needs More Than a Good Model

1. Introduction

A recurring failure mode in agentic systems is deceptively simple: the worker finishes running, therefore the task is treated as finished. Those are not equivalent statements. A process can terminate cleanly while required evidence is absent, an acceptance condition is unmet, or the produced result does not match the requested outcome.

SignalRivet's internal BOSS architecture was changed to make that distinction explicit. The lower-level worker dispatcher remains responsible for lifecycle facts such as launch, acceptance, execution, termination, timeout, teardown, and durable task identity. A governed coordinator adds a second layer whose job is to decide whether the claimed outcome has actually been demonstrated.

2. Related work

Two recent lines of work influenced the framing of this architecture. SOPStruct converts natural-language procedures into structured graph representations and evaluates graph soundness and completeness, reinforcing the value of representing procedures as something more machine-checkable than prose alone [1]. SOP-Bench evaluates agents against executable business procedures, functioning tools, and ground-truth outcomes, and reports that model-agent performance varies materially by procedure and configuration [2].

The present note is narrower. It does not introduce a public benchmark or claim a general solution to procedural agent reliability. It documents an internal control architecture for requiring stronger completion evidence inside one operating system.

3. System design

3.1 Separation of lifecycle and semantic completion

The governed path is intentionally layered. A lower-level dispatcher records execution facts. A coordinator then validates a structured final completion message against the task contract and routed SOP requirements.

3.2 Structured worker completion

For governed tasks, the worker receives a compact completion contract before execution. Its final message must use the schema boss.worker-completion.v1. The contract instructs the worker not to claim work, evidence, approvals, or outcomes that did not occur and to return HOLD when completion cannot be proven.

3.3 Machine companions for SOPs

The human-readable SOP remains normative. Each controlled SOP also has a machine companion appropriate to its primary role so that the runtime can resolve, expose, and validate relevant completion claims.

4. Method

Validation was divided into procedure-level and integration-level checks. Procedure-level benches tested routing, control dispositions, governance state behavior, and valid terminal claims. Integration tests exercised task capsules, runtime SOP resolution, harness contracts, coordinator behavior, lifecycle dispatch, and command-line compatibility.

The validation set is intentionally described as bounded internal validation. It is structural and synthetic evidence produced against known test cases. It is not a substitute for accumulated real-task evidence.

5. Results

The controlled architecture passed the targeted validation suites listed below.

5.1 Lifecycle PASS can coexist with overall HOLD

The architecture preserves the lifecycle dispatcher's result rather than rewriting it. If lifecycle execution passes but the semantic completion fails, the coordinator records the dispatcher terminal state and returns HOLD_GOVERNED_COMPLETION.

5.2 Observed internal completion-contract mismatch

During a local SignalRivet website integration run, the worker successfully performed filesystem work and the lifecycle layer completed. Its final structured completion payload did not satisfy the exact semantic evidence and SOP-claim contract expected by the governed coordinator. The coordinator therefore held the task rather than inferring completion.

This case did not prove the underlying website work was incorrect. It proved that completion had not been demonstrated in the form required by the governed path. The local artifact was subsequently reviewed on separate evidence.

6. Discussion

6.1 Completion claims become more useful when they are testable

Natural-language instructions alone leave substantial discretion at the moment of completion. A structured claim contract narrows that discretion by giving the surrounding system a known object to validate.

6.2 Procedure structure is not the same as independent evidence

Machine companions can verify routing, required paths, control dispositions, and claim structure. They do not automatically make every worker-reported intermediate event independently true. High-consequence tasks still require stronger evidence from tools, tests, hashes, API responses, browser state, database state, or qualified review.

6.3 Fail-closed behavior preserves uncertainty

When semantic evidence is missing, HOLD is preferable to a false PASS. The design goal is not to maximize completion rate. It is to preserve the difference between “we have evidence” and “we have a plausible statement.”

7. Limitations

1. The validation set does not establish blanket real-world effectiveness across task classes.
2. Full SOP conversion does not qualify every model or worker for every governed task.
3. Some procedure completed-node claims remain worker-reported rather than independently observed.
4. High-consequence work still requires stronger independent evidence through competence, QA, and harness controls.
5. Legacy workflows without explicit SOP metadata remain on their existing path and are not silently assigned procedures.
6. Real-world cross-task effectiveness remains dependent on accumulated production evidence.

8. Implications for benchmarking

The model name is only one component of the evaluated system. Task contract, system prompt, tools, procedure representation, evidence requirements, routing, and validation architecture can materially change observed performance. This aligns with SOP-Bench's finding that agent design and procedure choice can substantially affect outcomes [2].

Accordingly, future SignalRivet model evaluations should identify the full working configuration and report results by task class rather than collapsing everything into a single “best model” score.

9. Conclusion

A capable model can propose, execute, and describe work. A reliable operating system still needs an explicit definition of what counts as evidence that the work is complete. The governed completion architecture described here separates lifecycle truth from semantic completion, preserves uncertainty through HOLD states, and makes procedure claims more inspectable through machine companions.

The next phase is empirical rather than architectural: accumulate governed real-task evidence, convert actual failures into regression cases, and qualify workers for specific jobs based on what they repeatedly demonstrate.