---
title: "What a useful AI pilot should prove"
description: "A useful pilot supports a concrete decision. Test representative work, establish a credible baseline and account for the effort of operating the process."
slug: "what-a-useful-ai-pilot-should-prove"
url: "https://withlodge.com/writing/what-a-useful-ai-pilot-should-prove"
published: true
published_at: "2026-05-26T12:00:00.000Z"
updated_at: "2026-11-02T15:16:14.951Z"
image_url: null
image_alt: null
image_width: null
image_height: null
word_count: 786
---
A pilot should leave a business able to make a decision. Continuing, narrowing the scope, changing the approach and stopping are all useful outcomes when the evidence supports them.

That purpose changes how a pilot is designed. A successful demonstration shows what can happen under chosen conditions. A useful pilot tests whether the proposed way of working deserves a place in everyday operations, including the effort required to keep it there.

Consider a hypothetical professional services team evaluating an agent that prepares meeting follow-up tasks. The question is whether people can rely on the resulting tasks within a defined process. An attractive list is only the starting point.

## Write the decision before running the experiment

The team first needs to describe what it would do with a positive result. Perhaps it would allow draft tasks to enter a review queue for a specified kind of meeting. That is a concrete next scope, with a known audience and a person responsible for the decision.

It should also name the findings that would prevent that move. Unclear ownership, invented commitments or information appearing in a task that the recipient may not see could each make the proposed process unsuitable. These conditions belong in the pilot design before anyone sees an impressive demonstration.

The decision needs an owner who can weigh the tradeoffs. The person running the experiment can collect evidence, while the relevant business and information owners determine what behavior is acceptable. An agent cannot approve its own expansion or interpret enthusiasm as a broader grant.

A short decision statement keeps the experiment focused: under these conditions, with these review requirements, should this work proceed to the proposed next scope? Every observation can then help answer a question somebody is authorized to decide.

## Establish a baseline people recognize

Before introducing the agent, the team should examine how follow-up tasks are created now. Which meetings produce them? Who checks the wording? How are owners confirmed? Where do missing or disputed commitments get resolved?

The baseline should include the work around the artifact. A task list may take little time to write but require careful checking against the meeting record. Another may be quick to review because participants already agree on the commitments. Treating these as equivalent would make the comparison misleading.

The team can use a consistent record of preparation, review, correction and acceptance for both the existing process and the pilot. It should also record what remains unresolved. A fast draft with an unanswered ownership question has not reached the same finish as an accepted task.

Permission matters in this comparison. The experiment may use only meeting material and participant information that its actors are authorized to access. A pilot label creates no exception to that requirement.

## Include the cases that could change the decision

For the team, representative cases would include ordinary planning meetings, discussions where no commitment was made and conversations where participants disagree about the next action. An unavailable recording or an ambiguous speaker should have a defined place in the evaluation too.

The aim is to learn where the process works and where it needs a human judgment. A refusal to infer a commitment from an unclear discussion may be the correct result. A confident task based on that same discussion may be a serious failure, even if it reads well.

Keep the cases comparable across revisions. If the operating brief changes, replay the relevant examples under the new instructions. Otherwise, a better result may reflect an easier selection of work rather than a useful improvement.

People reviewing the output should capture specific disagreements and their reasons. That creates material for a decision about scope or process design. A general impression of quality is harder to act on and easier to influence with a memorable success.

## Count the work of keeping it running

The pilot also needs an operating record. Someone will maintain the instructions, investigate missing records, review exceptions and answer questions from colleagues. Those responsibilities consume capacity even when the underlying generation works as intended.

Include that effort in the continuation decision. A process may be worthwhile with a review step if the overall work becomes easier to complete. It may also need a narrower scope because maintenance or correction falls on people who cannot sustain it.

Finish with a decision memo that names the tested scope, the baseline, the difficult cases, the unresolved issues and the proposed next boundary. Preserve the evidence needed to revisit the choice without exposing material to a wider audience.

Before starting a pilot, write the final decision memo with its evidence fields left empty. If the team cannot agree what would belong in those fields, settle that first. It will make the experiment much more useful.