Computer-Using Agents and Human Checkpoints
How I think about when computer-using agents can run autonomously and where human approval must stay in the loop.
When agents can click/type/navigate software, value goes up fast. Risk also goes up fast.
I treat these systems as supervised operators, not independent decision makers.
The practical question is not "can the agent use the computer?" It is "which actions can it perform without creating irreversible damage?" That is where human checkpoints become product design, not just safety theater.
The operator model
A computer-using agent is a very fast operator. It can follow instructions, recover from minor UI changes, and handle repetitive digital tasks. It still needs checkpoints near irreversible actions.
This framing helps because an operator can be delegated routine work, but a responsible team still defines the operator's boundaries. The agent should know when to continue, when to ask, and when to stop.
Typical loop
The loop is simple: read screen state, choose the next action, execute it, verify that state changed as expected, and repeat.
This is more adaptive than classic RPA. It is still weaker than humans on ambiguous or high-stakes tasks.
Example: browser-based back-office work
Consider an agent that updates records in a vendor admin panel. Reading the record, finding missing fields, and drafting the update may be safe. Pressing "save" on a customer-visible change is different. If the UI has no undo path, that final action should usually require human approval.
The same pattern applies to refunds, permission changes, data exports, and compliance submissions. The agent can do the legwork; the human owns the irreversible boundary.
Action lanes I use
- Green: read/search/summarize
- Yellow: low-risk changes with audits
- Red: explicit human approval (money/data/permissions/legal)
If everything is "green," incident probability rises quickly.
Designing checkpoints
A good checkpoint should show what the agent is about to do, which evidence it used, what will change after confirmation, whether the action can be undone, and what the fallback is if the action fails.
"Approve?" is too vague. The reviewer needs enough context to make a fast, confident decision.
Where checkpoints are not optional
Payments, refunds, data deletion, permission changes, compliance submissions, legal submissions, healthcare, and safety-critical operations need human checkpoints.
Logging I want by default
For every meaningful action, I want a trace of the observed screen state, selected action, tool or input used, result observed after the action, and human approval record when applicable.
This is partly for debugging, but it is also for trust. If users cannot reconstruct what the agent did, they will either over-trust it or stop using it.
When autonomy is reasonable
More autonomy is reasonable when the task is reversible, low impact, repetitive, and easy to verify. Summarizing a dashboard or collecting links can be mostly autonomous. Changing a production permission should not be.
The line can move over time as evals, auditability, and product confidence improve. It should move deliberately, not because the demo looked smooth.
Product copy matters too
The UI should not imply more autonomy than the system can safely provide. I prefer labels like "draft update," "prepare refund," or "review permission change" over vague labels like "complete task." The label should tell the user whether the agent is preparing work or committing work.
This sounds small, but it shapes trust. Users are more comfortable supervising an agent when the product is honest about what still needs human judgment.
Trend signals behind this note
- Google announced AI Mode and Project Mariner updates on May 20, 2025: Google Search AI Mode update.
- OpenAI added computer-use capabilities in Responses API updates on May 21, 2025: New tools and features in the Responses API.
Takeaway
Let computer-using agents execute routine work. Keep humans on risky boundaries, and make every approval carry enough context to matter.