Choose a task with boundaries

Start by describing one task in ordinary language. For example, a team might need to find information in approved internal documents, extract fields from a standard form, or prepare a draft summary for review. Each example has an input, a desired output, and a person who can judge whether it is useful.

Broad ambitions can make evaluation difficult. A narrower task gives the team a clearer basis for testing the approach and deciding whether it deserves a place in the workflow.

Understand the information available

Identify where the information comes from, who maintains it, and who is allowed to use it. Check whether it is current, sufficiently complete, and suitable for the task. Different document formats, inconsistent records, and outdated instructions can affect the usefulness of results.

Set access boundaries before connecting tools. A person should not gain access to information through an assistant that they could not access through the organization’s normal systems.

Decide how results will be assessed

Prepare representative examples and define what a satisfactory result looks like. Consider more than whether an answer reads well. Is it grounded in the available information? Does it include the required fields? Can a reviewer see where the information came from?

Include difficult or incomplete examples. Decide what the system should do when it cannot produce a reliable result. Asking for clarification or referring a task to a person may be more useful than producing a confident answer.

Keep responsibility visible

Define which steps can be automated and which require review. For consequential decisions, the system should support the responsible person with information rather than obscure who is making the decision.

Make the limits of the tool understandable to its users. A clear interface should distinguish source information, generated material, and decisions that still need to be made.

Pilot, learn, and integrate

Introduce the use case to a small, relevant group. Observe the work around it: the preparation required, the review effort, and how the output is used. A tool that appears useful in isolation may introduce extra work elsewhere.

Compare the whole workflow before and after the pilot. Use the findings to improve the approach, adjust its scope, or decide that another use case is a better starting point. The goal is a useful operational capability, supported by evidence from actual use.

Before the pilot: five decisions

  • Define the task and the result the user needs.
  • Confirm permission to use the source information.
  • Select representative examples, including difficult cases.
  • Name the reviewer and the route for uncertain results.
  • Set success criteria and a rule for stopping or revising the pilot.

Illustrative workflow: an internal policy assistant

An employee asks a question → the assistant retrieves approved policy material → the answer points to its sources. If the material does not support an answer, the request goes to a responsible person. Evaluate correctness, source relevance, and unanswered questions—not just the number of replies.