Enterprise Pilot Roadmap
English | 中文
Advance through exit criteria instead of starting with a universal Agent
Add one risk class at a time: prove the task and evidence, then read-only tools, controlled writes, recovery, and multi-team composition.
One task unit, one owner, and one completion definition.
Capabilities grow by stage; high-risk actions do not skip stages.
Observable evidence decides whether to continue, correct, or stop.
Stage roadmap
The roadmap does not prescribe a fixed number of weeks. System integration, data classification, and security review vary widely between enterprises. Define entry conditions, allowed capabilities, and exit evidence for each stage instead.
| Stage | Allowed capability | Required evidence | Exit condition |
|---|---|---|---|
| 0 Frame the task | No production-system integration | Task samples, human baseline, completion definition, risk classification | The team can consistently judge whether one run is complete |
| 1 Observe only | Fixed material and model, no state-changing tool writes | Session records, error types, human-rating disagreements | Output form is stable and main failures are classifiable |
| 2 Read-only tools | Search, read, query, and controlled computation | Tool trajectory, citation or query evidence, takeover records | Findings return to sources and errors do not change business state |
| 3 Controlled writes | Reversible or low-risk writes with per-action approval | Approval pairs, business idempotency, compensation and exception records | Unapproved calls have no effect and retries do not duplicate effects |
| 4 Recovery and scale | Persistent sessions, continuable subagents, second preset | Recovery drill, capability isolation, cost and capacity observation | Crash and switch behavior match expectations; a second scenario does not fork the core |
Phase 0: Express the task as an acceptable unit
Collect real historical samples and record the human process, input material, allowed actions, and final artifact. Replace “improve efficiency” with an observable question such as “find all three field classes in the supplied material and provide a source for each,” rather than inventing a benefit percentage first.
This stage also defines stop conditions. If inputs are repeatedly missing, business rules have no owner, outcomes cannot be reviewed, or one task contains inseparable high-risk actions, return to process governance instead of handing all uncertainty to the model.
Phase 1: Observe the model and session only
Process samples with a fixed Agent composition but no tools that change business state. Check whether the session completely records input, model output, errors, and corrections. Establish failure classes instead of reporting only “answer satisfaction.”
At minimum, distinguish missing material, ambiguous task, model reasoning error, unmet tool need, conflicting domain rule, and output-format error. Classification identifies whether to change process, prompt, tool, or model instead of calling every issue “model instability.”
Phase 2: Connect read-only tools
Wrap the most stable sources as read-only tools: repository search, document read, ticket query, or data snapshot. Each tool needs a clear schema, timeout, errors, and presentation intent. A reviewer should be able to locate the tool result instead of receiving only a model summary.
The read-only stage validates whether answers have evidence and failures can transfer to a person. If the Agent cannot cite sources or distinguish no result from system error, writes will only amplify uncertainty.
Phase 3: Add reversible controlled writes
Prefer drafts, suggestions, temporary labels, and other reversible actions. Give each write tool policy and approval; the business interface uses caller identity, idempotency, and an explicit transaction result. Approval copy names the object and impact instead of asking only whether to “allow the tool.”
Exercise denial, unavailable approvers, tool timeout, partial business failure, and duplicate requests. Exit evidence compares live state with the log: unapproved actions have no side effect, successful calls have one business effect, and failures give an operator a clear recovery entry.
Phase 4: Validate recovery, delegation, and multi-team composition
After the main path stabilizes, validate session recovery, continuable subagents, and a second Agent preset. Recovery drills cover process termination, persistence reads, and follow-up messages. Delegated work completes under fixed authority and returns to the parent Agent when it needs more access.
The second preset should serve a different task with a different tool set. If the new scenario still copies the agent loop, session, or Host, capability seams and plugin composition have not delivered the intended foundation.
Pilot scorecard
The scorecard defines metrics and evidence sources without fabricating target values. The pilot owner sets thresholds from the human baseline, risk tier, and sample distribution.
| Dimension | Suggested metric | Evidence source |
|---|---|---|
| Task quality | Share of samples accepted against the checklist; reasons for human rework | Business acceptance sheet and final artifacts |
| Traceability | Share of key findings that return to a source or tool result | Session events, citations, and tool cards |
| Control effectiveness | Whether denial, approval, retry, and exception branches behave as specified | Approval events and business-system state |
| Recoverability | Share of specified faults that restore context or explicitly report non-recovery | Persistence drills and recovery records |
| Human load | Where approval, takeover, review, and correction time occurs | Operation records and interviews, not total duration alone |
| Run cost | Model tokens, tool time, remote environment, and storage consumption | Request events, tool timing, and platform billing |
Roles and responsibilities
| Role | Decision it must own | Responsibility it must not transfer |
|---|---|---|
| Business owner | Task boundary, completion, exceptions, and final acceptance | Letting the model define business correctness |
| Product/process owner | Human takeover, approval copy, and delivery interface | Hiding an undefined process in prompt wording |
| Platform engineering | Presets, capability providers, persistence, and observability | Inventing authorization and transaction semantics for business systems |
| Security and compliance | Data classification, execution environment, credentials, and retention | Treating session-log existence as completed compliance |
| Evaluation owner | Samples, human baseline, failure classes, and thresholds | Using one average score to hide high-risk failures |
Expand, correct, or stop
When should the pilot expand?
The current task meets exit criteria on the specified sample, main failures are classifiable and transferable, control branches have been exercised, and a new scenario can reuse the runtime spine.
When should the pilot correct course?
When value exists but evidence breaks, tool interfaces are unstable, approval load is too high, or one failure class dominates, repair the process, tool, or composition and repeat the stage.
When should the pilot stop?
Stop when completion remains undefined, irreversible work cannot be separated, data or identity conditions prohibit the design, takeover costs exceed task value, or value depends only on untestable future assumptions.
Prove judgeable, controllable, and transferable before faster, broader, and more autonomous
Plugins and multi-Agent capability make scope easy to expand. Pilot discipline requires every expansion to rest on passed evidence instead of using more features to hide an unresolved earlier stage.
Evidence status: Harness capabilities and limitations are confirmed against current source. Stages, metrics, and organization advice are a pilot method; the target enterprise sets samples and thresholds.