Check a finished job against the playbook it was meant to follow

MaintenanceIntermediate~10 min

Before you start

Plugin version 1.10.1 or newer. Run this from an administrator connection, because the Activity Log read needs the manage-options capability, which a day-to-day agent user should not have. List playbooks, Get playbook and Get activity log must be switched on. Nothing here writes.

An agent ran your Shop tidy-up playbook overnight and sent a tidy summary. That summary is the agent’s own account. The Activity Log is this plugin’s record of what was actually called, and the playbook is what you asked for, so the useful check is where those two disagree.

The prompt
I want to check a job against its playbook. Read only: change nothing.
1. List the Active playbooks, find [Shop tidy-up] and load it. List its
numbered steps and the tools each step uses.
2. Read this plugin's Activity Log newest first, 200 per page, until you
pass the start of [today 02:00 to 03:00]. Times in the log are UTC; my
site time is [UTC+5], so convert before you filter. Keep only rows
where the user login is [agent-shop].
3. Match each kept row to the playbook step it belongs to. If a row fits
no step, mark it "outside the playbook".
4. Tell me:
- steps with no matching row (skipped, or done without a tool call)
- rows that were denied or ended in error, with their detail codes
- any write in the log that the playbook does not name
- any step that ran far more calls than the playbook implies
5. Give a one-line verdict per step: done, partly done, skipped, or cannot
tell. Mark anything you inferred as inference.

Why this is safe to run

Each of the three tools only reads. The Activity Log returns the ability name, outcome, user, time, the argument keys and a short identifier per call, never the content that was sent and never an IP address.

Start with any write that shows up in the log but not in the playbook. For a plain session review without a playbook, verify what an agent did from the audit log covers it, and the admin Activity Log tab adds the refusal reason and source address for any denied row.

Already connected from an admin account? Ask your agent to check last night’s run against the playbook it used, and give it the time window, your UTC offset and the agent user.

How it works

  1. Find the playbook in the Active list, load it, and list its tools in step order.
  2. Page back through the Activity Log until it passes the start of your time window.
  3. Keep the rows for the agent user and match each one to a playbook step.
  4. List skipped steps, refused or failed calls, and writes the playbook never names.
  5. Give a verdict per step, with inference labeled as inference.

Questions

Can the log prove the agent followed every instruction?
No. The log records which of this plugin's tools ran, the outcome, the user and an identifier such as a post ID. It cannot show whether the agent read a post carefully or wrote a good summary. Checking the tool trail against the playbook catches skipped steps and writes nobody asked for.
Why does it page through so much of the log?
The Activity Log tool filters by status and by ability, not by user or date. The agent reads newest first and stops once it passes your window, so a busy site takes more pages. Log times are UTC, which is why the prompt asks for your offset.
Does the log include tools from other plugins?
It records calls to this plugin's own tools, including bridged ones served through this plugin. Tools another plugin serves on a different MCP server do not appear.