Consider a CI/CD pipeline configured to deploy to production on every successful merge to the main branch. No human approval gate. No manual sign-off. The tests pass, the build succeeds, the pipeline runs, and the code is live. The pipeline executed instructions that a human engineer wrote. It evaluated criteria that a human team defined. It deployed to an environment that a human architect configured. And yet, at no point did a human decide to deploy. The decision — if it can be called that — was made by the satisfaction of preconditions in a system designed to act when those preconditions were met.

Is the pipeline a tool the team used? Or is it an actor that acted on the team's behalf?

The question is not rhetorical, and it is not merely philosophical. The answer determines who is responsible when the deployment introduces a critical failure. It determines whether the organisation's change management process has been followed or circumvented. It determines whether the system's action constitutes a decision that should have been made by a human, and whether the absence of that human decision is a governance gap or an efficiency achievement. These are not the same answer, and the failure to distinguish between them — to identify precisely where the automation threshold was crossed and whether crossing it was deliberate — is one of the most consequential unexamined questions in the practice of contemporary software engineering.

This essay argues that the automation threshold — the point at which a system crosses from tool to actor — is not defined by the system's technical capability but by the locus of consequential decision: a tool executes what a human decides; an actor decides what to execute. The crossing of this threshold is rarely announced, rarely governed, and frequently recognised only in retrospect, when the consequences of having an unacknowledged actor in the system become impossible to ignore.

The Tool: Capability Without Initiative

A tool extends human capability without substituting human intention. This is the definition that matters, and it is worth holding precisely. A hammer multiplies the force a human can apply to a surface; the human decides where the nail goes, when to strike, and how hard. A spreadsheet performs arithmetic across thousands of cells in milliseconds; the human decides what data to enter, what formulas to write, and what the results mean. A search engine retrieves documents matching a query; the human decides what to search for, which results to examine, and what to do with the information they contain.

The sophistication of the tool is irrelevant to its category. A statistical model that predicts customer churn is a tool as long as a human reviews its predictions and decides what to do about them. A computer vision system that identifies defects in manufacturing output is a tool as long as a human operator decides whether to halt the line. A recommendation engine that surfaces the ten most relevant documents is a tool as long as the human decides which to read. What defines the tool is not its complexity, its speed, or the degree to which its operation exceeds human cognitive capacity. What defines it is the presence of a human decision at the point of consequential action — not at the point of tool operation, but at the point where the tool's output is translated into a real-world effect.

The tool category preserves human agency not by limiting the system's capability but by structuring the system's relationship to consequence. The tool produces outputs. The human decides what those outputs mean and what to do about them. The causal chain from system output to real-world effect passes through a human decision, and that decision is where accountability lives.

The Threshold: Where Execution Becomes Initiative

The automation threshold is crossed at the precise point where that human decision is removed from the causal chain. The system no longer produces an output for human interpretation — it produces a consequence directly. The action and the effect are connected without a human in between, and the question of who decided becomes, in the strict sense, unanswerable: the system decided, in the sense that no human did.

The examples are more numerous than is generally acknowledged. A CI/CD pipeline with auto-deploy enabled is an actor; the same pipeline with a manual approval gate is a tool. A fraud model that blocks transactions automatically is an actor; the same model that flags transactions for human review is a tool. An inventory system that places purchase orders automatically when stock falls below a threshold is an actor; the same system that generates a recommended order for human approval is a tool. In each case, the technical difference between the two versions is minimal — a configuration flag, an approval step, a human notification. The categorical difference is absolute: in one version, a human decided; in the other, the system did.

Organisations cross this threshold incrementally, and the increments are individually defensible. The manual approval on the deployment pipeline creates friction; the team is disciplined; the tests are comprehensive; the auto-deploy configuration is a reasonable trust extension given the evidence. The fraud analyst reviewing model flags is a bottleneck; the model's precision is high enough that human review adds little value in the straightforward cases; the automatic block is a reasonable efficiency gain. Each crossing is a local optimisation. The aggregate is a system whose actor status has never been acknowledged, whose trust boundaries have never been formally defined, and whose accountability structure assumes a human decision that no longer exists.

Agents and the Recursive Threshold

Agentic AI systems — systems capable of planning multi-step actions, selecting and invoking tools, and adapting their behaviour in response to intermediate results — introduce a version of the threshold problem that is structurally more complex than any single automated action.

A single automated action has a definable threshold: either a human approved it or the system executed it without approval. An agent operating on a multi-step task crosses the threshold repeatedly, at multiple points within a single execution, at varying levels of autonomy, in ways that were not individually specified when the agent was given its instructions. An LLM-based agent instructed to "research this topic and draft a summary" may browse multiple websites, execute search queries, extract and synthesise content, and write a document — a sequence of actions each of which involves tool invocations, data access, and content generation decisions made by the agent without explicit human authorisation at each step. An agent instructed to "fix the failing tests in this repository" may read code, write modifications, execute the test suite, observe failures, revise the modifications, and commit the result — a sequence that includes consequential changes to a production codebase, made autonomously within the scope of a loosely specified instruction.

The governance frameworks developed for single-action automation — approval gates, circuit breakers, audit logs of discrete transactions — are not adequate for systems whose autonomy is recursive and compositional. The agent's action boundary is defined by the scope it was given, which is typically specified in natural language, interpreted by the agent, and expanded by the agent's own intermediate decisions about what is necessary to complete the task. The trust that was extended when the instruction was given applies, in the agent's execution, to a set of actions that the human who gave the instruction may not have anticipated and may not be aware of until after they have occurred.

Counter-Argument: Trust as Calibration

The most practically grounded counter-argument to the tool/actor distinction is that the distinction, however philosophically coherent, is not the right frame for governing automation. All useful automation involves system initiative of some kind — the elevator decides when to stop; the autopilot manages trim and heading; the autocorrect modifies text. In each case, the system takes actions that the human did not explicitly authorise at the moment of execution, and in each case the arrangement works well enough that the philosophical category of the system is irrelevant to its safe and effective use. The relevant question, on this account, is not whether the system has crossed an actor threshold but whether the trust placed in its autonomous actions is appropriately calibrated to its demonstrated reliability, its error rate, and the reversibility of its failures.

This framing is not wrong. It describes accurately how most automation is governed in practice, and it works adequately in domains where the failure mode is well characterised, the error rate is measurable and stable, and the consequences of error are bounded and recoverable. The spam filter that occasionally misclassifies an email is a system whose autonomous action is trusted within a calibration that accounts for its known failure rate. The calibration is reasonable because the failure is recoverable.

The rebuttal is that the calibration model has a defined scope, and outside that scope it fails in ways that the tool/actor distinction directly illuminates. The CI/CD pipeline that auto-deploys a breaking change was trusted within a calibration that assumed the test suite was complete — and the test suite was not. The agent that interpreted its scope more broadly than its operator intended was trusted within a calibration that assumed the instruction was unambiguous — and it was not. The fraud model that developed a systematic demographic bias was trusted within a calibration that assumed the training data was representative — and it was not. In each case, the failure occurred at the boundary of the trust that had been extended, in conditions that the calibration did not anticipate, and the accountability gap was a direct consequence of having an actor in the system whose actor status had never been formally acknowledged.

Conclusion: The Trust Boundary as Design Artefact

The automation threshold is not crossed once. It is crossed in increments — one approval gate removed, one alert suppressed, one agent scope expanded — each increment locally justified, the aggregate ungoverned. The result is a system landscape in which actors are everywhere and acknowledged nowhere: systems that decide, that act, that produce consequences, whose authority to do so was never formally granted and whose accountability for what they do is structurally unclear.

The governance response to this condition is not to reverse the increments — the efficiency they represent is real and the competitive pressure to retain them is persistent. It is to make the crossing of the threshold a deliberate and documented act: to define, for each system that acts without human approval, what it is authorised to decide, under what conditions that authority holds, what would cause it to be reduced or revoked, and who bears responsibility when the system acts within its authorised scope and the consequence is nonetheless harmful.

The trust boundary — the explicit, maintained definition of what a system may do without asking — is the governance mechanism for the automated actor. It is not a technical constraint. It is a design artefact that requires the same rigour, the same version control, and the same review cycle as the system it governs. In its absence, the system defines its own scope through the accumulation of its own actions. That is not a tool that got out of hand. It is an actor that was never named.

References

  1. Fowler, M. "Continuous Delivery." martinfowler.com. https://martinfowler.com/articles/continuousDelivery.html
  2. Amazon Web Services. "AWS Auto Scaling." aws.amazon.com. https://aws.amazon.com/autoscaling/
  3. Weng, L. et al. "LLM-powered Autonomous Agents." arXiv:2302.04761. arxiv.org. https://arxiv.org/abs/2302.04761
  4. Platform Engineering. "What is Platform Engineering?" platformengineering.org. https://platformengineering.org/