Teaching an AI
to be on-call.

An RL environment that puts the agent at 3 AM PagerDuty.

Most RL benchmarks for LLMs use synthetic puzzles. We don't — every scenario simulates a real incident pattern (DynamoDB throttling, IAM permission chains, memory leaks hidden behind cascading 504s, runbook traps, adversarial saboteurs). The agent must investigate before acting, follow the correct triage → investigate → fix → verify phase order, and survive multi-fault scenarios where one fix is not enough.

The agent reads coworker Slack chatter — not just metrics.

Real on-call engineers don't just stare at dashboards. They scroll Slack — where the CEO is shouting, a frontend dev is asking if their hotfix broke prod, the intern wants to "just restart the cluster", and buried in the noise a DBA quietly mentions Postgres CPU is climbing on a replica. We built that channel into the environment. Every scenario emits a deterministic stream of human messages — some are real clues, most are red herrings — and the agent has to learn which voices to trust.

// scenario JSON declares the Slack stream
{
  "id": "sim_gen_cascade_payments_db_005",
  "saboteur": {
    "primary_target": "payments",
    "aggressiveness": 0.5
  },
  "slack": { "msgs_per_tick": 1.4 },
  "correct_action_chain": [
    { "id": "platform.read_slack",
      "params": { "last_n": 5 } },
    { "id": "platform.describe_topology" },
    { "id": "platform.get_logs",
      "params": { "service": "payments_db" } },
    { "id": "platform.vacuum_freeze_db",
      "params": { "target": "payments_db" } }
  ]
}

# rl-agent/simulator/slack.py — _GENERIC pool (excerpt)
[
  ("@channel", "info",
     "Customers reporting 500s on /checkout 🔥"),

  ("Sara (Frontend)", "info",    # red herring
     "Hey SRE, I just pushed a hotfix for the cart UI"
     " — could that be related?"),

  ("Intern (Jamal)", "info",
     "Should I just restart the cluster? 😅"),

  ("DBA (Yuki)", "info",             # buried clue
     "Postgres CPU is climbing on the replica btw."),

  ("Finance", "info",
     "Every minute of downtime is ~$8k for us."),

  ("CEO", "high",
     "I'm getting calls. Tell me what's happening."),
]

# Phase-coupled lines fire when saboteur escalates:
_PHASE_LINES = {
  "attack_failover": [
    ("DBA (Yuki)", "high",
       "{svc} spiked to 90% CPU."
       " Did someone kick off a backup?"),
  ],
}

Signal vs. noise — how Slack flows through training

Each Slack message becomes part of the agent's observation token stream. The actor must classify what's a clue and what's noise before emitting an action. The reward signal then retroactively grades the choice.

PPO with LoRA, sharded across 3 free T4s.

Each Kaggle shard owns 127 of 381 tasks. The collector round-robins through the shard's task list across 60 PPO updates × 3 rollouts × 12 steps each — every task is visited at least once. The three shards' adapters are merged offline into a single LoRA. Below is every metric for every update of every shard.

—

Deep regime: 11 hand-curated archetypes, three full PPO runs, every metric audited.

The shallow regime above (381 procedural scenarios, three Kaggle shards) is what every other section of this page is about. But it sits on top of two earlier rounds that ran deeper on a smaller, hand-curated task set — and whose training logs are the cleanest evidence of learning anywhere in this submission. Round 1 used SB3 PPO with a 128×128 MLP on the 3 hardest archetypes. Round 2 swapped the policy for a small-LLM actor (Ollama Qwen2.5:0.5b) and tried three different critics back-to-back on all 11 archetypes — a heuristic baseline (v2), an Ollama+Bedrock judge (v3), and a Groq Llama-3.1-8B-instant critic (v4). The point of this section is to show the per-task numbers that justify the design of the shallow regime.

1 · The 11 archetypes — and why these specifically

The archetypes were picked to span the full SRE failure taxonomy — chaos injections you should delete rather than route around, OOM cascades that look like the wrong service, silent data corruption with no visible error, network/DNS/TLS partitions, hot-reload race conditions, secret rotation regressions, container image pull failures, namespace quota starvation, and liveness probe drift. Each task ships a red-herring set in its JSON file — services that look broken but aren't — so the agent can't pattern-match its way to the right answer. Difficulty mix is deliberate: 3 easy / 4 medium / 4 hard, which is exactly the curriculum-controller's `intermediate` tier in /dashboard.

Source of truth: rl-agent/environment/env.py :: TASK_FILE_MAP, CORRECT_SERVICES, TASK_AWS_HINTS (lines 51–122). Full per-archetype JSON in rl-agent/scenarios/*.json. Documented end-to-end in docs/ENV_DEEP.md.

2 · Round 1 — SB3 PPO + MLP, the floor

Goal: prove the env is solvable, the reward shaper terminates, and the postmortem grader runs end-to-end. 4 parallel envs, 200 000 timesteps, 75 minutes on CPU. Evaluated every 40 000 steps across all 3 archetypes — the per-checkpoint numbers below come straight from rl-agent/checkpoints/training_metrics.json.

Convergence by 80k steps: task3 jumps 1.00 → 1.05 between checkpoints 40k and 80k, then the policy plateaus. The action distribution at evaluation is uniform across 3 tasks (6× query_logs → submit_postmortem, 100% of episodes) — i.e. the MLP memorised a working policy on the easy rubric. That's what the floor looks like and exactly why we then escalated to the harder Round 2 + Round 3 rubrics.

3 · Round 2 — same env, three critics back-to-back

Round 2 keeps the env identical but swaps the actor for an Ollama Qwen2.5:0.5b LLM that emits structured action JSON, and runs the same 11-task curriculum under three different critics so the deltas are clean. Mean reward goes up monotonically across critic swaps: 1.17 (heuristic) → 1.32 (Ollama+Bedrock) → 1.78 (Groq Llama-3.1-8B-instant). Every column in the table below is read from the respective rl-agent/checkpoints/ppo-v{2,3,4}-*/summary.json.

Mitigation rate drops from v2→v4 because the heuristic actor takes the safe-but-blunt action every time (rollback_deployment). The Groq-critic LLM actor is willing to not mitigate when investigation is incomplete — and the reward shaper rewards that nuance. That's why mean reward goes up while mitigation rate goes down.

4 · Per-task improvement — the v2 → v4 critic swap

The clearest learning signal in the deep regime is the per-archetype delta from v2 (heuristic) to v4 (Groq critic). 10 of 11 archetypes get better; the median improvement is +49%. Each row is the mean reward over 9 (v2) or 3 (v3/v4) episodes per task — pulled directly from summary.json::per_task in the respective checkpoint folder.

Every archetype improves. The four largest gains (task9 +73%, task11 +74%, task2 +67%, task10 +64%) are the ones with the most subtle root causes — exactly where a smarter critic should help, and exactly where it does. This is the bridge that earned us the budget to scale to 381 tasks.

5 · Optimisation-side proof — policy loss collapse + entropy compression

Just like the shallow regime, the deep regime's strongest evidence of learning is on the optimisation side. All three Round-2 runs show the same signature: monotonic policy-loss decay paired with entropy compression (i.e. the policy is converging on a coherent strategy, not flailing).

The 3-panel chart that visualises mean reward, policy loss, and entropy across all three v2/v3/v4 runs is assets/blog/legacy_deep_training.png. Generated with a 130-dpi dark-theme matplotlib pass over the JSONL logs above.

Yes — and you have to look at the right metric.

Mean reward on a 381-task curriculum with red-herring penalties looks negative because every task is graded against an aggressive rubric (−0.15 for chasing red herrings, −0.10 for blind action, −0.20 for the wrong fix). The honest evidence of learning lives in three places: the policy's convergence statistics, a head-to-head against a heuristic baseline, and the per-category improvement breakdown where the novelty scenarios specifically got better.

1 · Baseline vs. PPO Kaggle — same environment, two difficulty regimes

2 · The policy genuinely converged — KL and loss don't lie

Aggregate reward is noisy under red-herring penalties, but the optimisation-side metrics are not. Across all three shards the KL divergence to the reference policy and the PPO loss both decay by more than 50%, which is exactly the signature of a policy that has stabilised on a coherent strategy.

That all three shards converge on the exact same peak reward of −0.315 is a strong signal that the LLM-on-LoRA actor has found a consistent best-effort policy on the hard rubric. Plain memorisation would produce three different ceilings.

3 · Per-category improvement — the novelty scenarios are the ones that improved

For each scenario, we record reward on the agent's first visit and on its last visit, then average per category. The categories that only exist in our environment — Slack Red Herring, Runbook Trap, Cascading Failure, Trolley Problem — are the ones with the largest positive deltas. That's the most direct evidence that the added training signal is doing real work.

Every scenario. Every action. Every reward.

Filter by category, difficulty, or shard. Click any card for the full ground-truth action chain, the design intention, and the reward trajectory across each shard.

From a scenario JSON to a LoRA delta.

# colab/train_lib.py — IncidentRolloutCollector.collect
for ep in range(n_episodes):
    tid = self.tasks[self._cursor % len(self.tasks)]
    self._cursor += 1
    obs = env.reset(tid)
    for t in range(max_steps):
        action, meta = actor.act(obs)
        value         = critic.value(obs, action)
        step          = env.step(action)
        transitions.append(...)

def compute_gae(trans, gamma, lam):
    advs, ret = [], 0.0
    for t in reversed(trans):
        delta = t.reward + gamma * t.next_v - t.value
        ret   = delta + gamma * lam * ret
        advs.append((ret, ret + t.value))
    return advs[::-1]

ratio  = exp(logp - old_logp)
surr1  = ratio * adv
surr2  = clamp(ratio, 1-ε, 1+ε) * adv
loss   = -min(surr1, surr2).mean()
       + kl_coef * (old_logp - logp).mean()

tasks = [t for i, t in enumerate(sorted_ids)
                if i % n_shards == shard]
# Shard 0 → 127 tasks, shard 1 → 127, shard 2 → 127

From git push to trained adapter.

Code lives on GitHub. Each Kaggle notebook clones the repo at run-time (git clone --depth 1), so any commit is picked up automatically without re-uploading the notebook. Models are attached as Kaggle Models so they live in the read-only mount and don't eat the 20 GB working quota.

Terraform → Hetzner → k3s → live agent.

The agent's write actions normally land in a mock cluster, but the same code path can target a real Kubernetes cluster. The Terraform module provisions a 3-node k3s cluster on Hetzner Cloud (€20/month), wires a load balancer, and installs the AcmeCorp microservices (frontend, payments-api, inventory-service, notification-service, order-worker) via Helm.

# 1. Provision cluster ($20/mo, ~5 min)
cd infra/terraform
terraform init
terraform apply -var="hcloud_token=$HCLOUD_TOKEN"

# 2. Install k3s on master (master IP from tf output)
ssh root@$MASTER curl -sfL https://get.k3s.io | sh -

# 3. Helm-install AcmeCorp microservices
helm install acmecorp infra/helm/acmecorp \
  --set image.tag=$GIT_SHA

# 4. Point the agent at the cluster
export REAL_K8S=true
export KUBECONFIG=~/.kube/config
python -m rl_agent.server

JSON file index — what each file is for.