The basic pattern: instruction, tools, approval, log
Every agent I build has four parts. A clear instruction about what it is for. A small set of tools, each doing one thing with the narrowest rights that work. An approval step wherever the result is binding. And a log of every call: who asked, which tool, with which parameters, what came back.
Some actions always need a human: anything sent to a customer, payments, decisions about people, deletions, and anything legally binding. The AI can prepare them, a person decides.
Pattern 1: the automation never holds the credential
In the M365 Control Plane I built for multi-tenant Microsoft 365 operations, PowerShell 7 jobs run in Semaphore. The obvious design would give Semaphore each tenant's certificate. I did the opposite: the certificate never leaves the API container. When a job starts, the API issues a ticket bound to tenant, template and task, and passes it in the job's envelope. The script exchanges the ticket for a short-lived app-only token:
# Simplified from the runner module: read the envelope, exchange the ticket for a Graph token
$e = $env:SEMAPHORE_TASK_DETAILS_MESSAGE | ConvertFrom-Json
if ($e -is [string]) { $e = $e | ConvertFrom-Json } # the message arrives double-encoded
$base = ([string]$e.runner.url).TrimEnd("/")
$hdr = @{ Authorization = "Bearer $($e.runner.ticket)" }
$tok = Invoke-RestMethod -Method Post -Uri "$base/api/runner/v1/token" -Headers $hdr `
-ContentType "application/json" -Body (@{ resource = "graph" } | ConvertTo-Json)
Connect-MgGraph -AccessToken (ConvertTo-SecureString $tok.accessToken -AsPlainText -Force) -NoWelcome
# ... read-only work against the tenant ...
Invoke-RestMethod -Method Post -Uri "$base/api/runner/v1/evidence" -Headers $hdr `
-ContentType "application/json" -Body (@{ title = "Licence report"; data = $result } | ConvertTo-Json -Depth 10)The broker only answers requests from the internal network, only for graph, exchange or the tenant's own SharePoint host, and stores only a SHA-256 hash of each ticket. The ticket is revoked when the task finishes, expires after a time limit (60 minutes by default) and dies with a restart of the API. Results go back into the evidence archive, labelled with template, task and the person who started it.
One honest limitation: the token carries the permissions of the tenant's app registration, so the action label of a job is not enforced at token level. The protection is in who may start which template, in the short lifetime, and in the log. That is why audit-only tenants are connected with a separate, read-only app, and write actions for them are refused before a job even starts.
Pattern 2: switches that default to off
The Control Plane exposes an MCP endpoint so an AI assistant can work with it. Three rules apply:
- API keys are shown once and stored only as hashes.
- Per-tenant permissions: every MCP capability is a switch per tenant, and every switch is off until someone turns it on. The same applies to AI-driven write actions, such as importing a policy and assigning it to a pilot group.
- Audit log: every MCP call is logged with key, action and result.
What the AI may read without a switch is the baseline: the written operating rules. That way it advises "according to baseline 1.2" instead of inventing its own rules. Rules are not customer data.
Pattern 3: the model sees context, not access
The ticket copilot in the same tool suggests solutions for service desk tickets. It works with live tenant data, such as licences or Conditional Access state, but the model never gets tenant or certificate access. The server collects the context with its existing, tested functions and hands the model a finished context object. The model can be wrong; it cannot act.
Pattern 4: scoped tools instead of a shell
My own server is operated by an AI assistant through MCP tools: deploy a site, create a DNS record, create a reverse proxy host, send a mail. Each tool does exactly that, with its own guardrails: only create, change existing things only when explicitly asked, and secrets referenced by placeholder so they are never returned to the model. Giving an agent a root shell would be simpler to build and impossible to review.
Pattern 5: AI suggests, a person confirms
In DokAudit, my PDF accessibility product, AI suggests alternative texts for images. A person checks each one, and only then does the tool write it into the document. The time saving is real, and responsibility stays where it belongs.
What I would check in any AI integration
| Question | Good answer |
|---|---|
| Which credential does the agent hold? | None, or a short-lived token for one resource |
| What can its tools do? | One thing each, with documented limits |
| Who turns a capability on? | A person, per tenant or system, default off |
| What needs approval? | Everything binding, outward-facing or destructive |
| Can we reconstruct what happened? | Every call logged with requester, parameters and result |
| What does the model see? | Prepared context, no secrets |
Checklist
- Instruction, tools, approval and log defined for every agent
- No long-lived credentials in the automation; tickets or managed identities instead
- Tokens short-lived and limited to named resources
- Capabilities switched on per tenant or system, default off
- Binding, outward-facing and destructive actions approved by a person
- Read-only mode enforced on the server, not only in the UI
- Every call logged; logs kept and reviewed
- Secrets referenced, never returned to the model