How we’re building an autonomous AI engineering team

The question that started it

Terrain’s autonomous engineering experiment began with a practical constraint: one founder could not review every piece of work. We wanted to see how much of the team’s operation agents could take on while keeping the work accountable.

That question is bigger than whether a model can write a useful function. A team has to decide what happens next, preserve context, review changes, handle failures, and know when to stop. Those responsibilities became the center of the experiment.

This account draws on the four-day operating snapshot documented in our internal thesis. It describes an early system, including its failures. The observation window is too short to establish long-term reliability.

What we mean by autonomous

The human still sets the goals and the boundaries. Agents carry out the work inside those boundaries: coordinating tasks, building, reviewing, and handing responsibility between sessions. Live corrections from the human change the operating rules when a gap appears.

Some decisions remain with a person. Destructive actions involving real user data, commitments beyond an established budget, and certain identity or credential steps are among the boundaries described in the experiment. The scope of autonomy has to be explicit before a team can be evaluated against it.

That distinction matters when describing results. A useful measure is how often the human still has to intervene, along with why each intervention was needed. Calling a system autonomous tells you little without that context.

Roles that outlive a session

We organized the team around durable roles. An orchestrator coordinates the work. Captains carry missions forward. Workers handle bounded tasks. An analyst examines the system’s performance, including the orchestrator’s mistakes.

A role has a mission, success criteria, authority, and a handoff. A model session is one temporary holder of that role. Separating the two lets responsibility survive a session ending.

That separation introduced its own work. A successor has to identify the current state, understand its authority, and verify what the previous session left behind. A long handoff can be expensive to read; a short but stale handoff can be dangerous to trust. We use pointers to current records for information that changes quickly.

Review belongs to another role

The role producing a change should not be the only role certifying it. We separated writing from review, and gave the analyst a way to report certain problems without routing them through the orchestrator it was examining.

The checks also have to refer to the actual code being shipped. If a change is made after review, the evidence needs to be refreshed or its continuing validity established. Otherwise a green result can describe a version of the work that no longer exists.

Separate roles do not make errors impossible. Both review layers in this experiment are still internal. Agreement between them is useful evidence, with a narrower meaning than an independent external audit.

A failure should change the system

One of the most useful practices has been recording what failed and what is supposed to prevent it happening again. We distinguish between a written rule, an implemented check, and work that is still pending.

That distinction makes the gaps visible. A rule can be clear and still rely on an agent remembering to apply it. A mechanical check can refuse an action when a required condition is missing. Repeated failures under a written rule are candidates for stronger enforcement.

In the early record, stale identity pointers were caught across four consecutive successions. The review process caught them, but each recurrence was another opportunity for the wrong thing to happen. That history helped justify a verification gate.

The useful question is what changed after the failure.

Measure the interruptions, too

We track who catches a problem. If the founder finds it after the responsible agent and another review layer missed it, the system has demanded attention it was meant to save. A peer agent catching that same problem tells a different story.

Those categories need careful definitions. A mistake caught by the responsible agent before damage is different from a mistake that escaped and was caught later. Combining them into one success rate can make the result look better without improving the system.

Our internal grading rubric helped make judgments more consistent, but it was fitted against known history. Matching those earlier judgments is not evidence that it predicts future performance.

What we can say so far

The early experiment produced a working operating structure and concrete lessons about where it failed. It also exposed the cost of handoffs, the limits of internal review, and the danger of treating a written rule as if it were enforced.

We cannot use four days of operation to claim reliability over months. The cost estimates in the source account are modeled, and direct per-role token metering was missing. We also changed several parts of the system at once, making it difficult to isolate which change caused an improvement.

The next useful work is to extend the observation window, improve measurement, and keep examining where human attention is still required. That is the experiment we want to keep documenting.