CI/CD, WAF, and the Checkpoint Race

The glamorous days are over. Yesterday we had a running platform. Today we make sure it stays running.

The CI/CD pipeline

Four GitHub Actions workflows, each with a specific job:

• ci.yml: lint (ruff) → test (pytest) → migrate (Alembic against Postgres) → Docker build → ECR push (main only)
• deploy-staging.yml: manual trigger, helm upgrade, uses GitHub OIDC for AWS auth
• rollback-staging.yml: manual trigger, helm rollback. Because you always need a rollback button.
• terraform-plan.yml: runs on PRs that touch terraform/, posts the plan as a PR comment

Two IAM roles, both using GitHub OIDC (no long-lived credentials): one for ECR push (scoped to main branch), one for deploy (scoped to staging environment). The trust policies are different because the blast radius is different.

WAF in COUNT mode

We attached AWS WAF to the ALB — but in COUNT mode, not BLOCK. This is the burn-in pattern: let the rules observe real traffic for a week, review what would have been blocked, then flip to BLOCK once you’re confident you won’t break legitimate requests.

The alternative — deploying WAF in BLOCK mode immediately — is how you discover at 2 AM that your health check path matches a SQL injection rule.

The checkpoint race condition

We fixed a real bug. Here’s the setup: agents can save checkpoints (persistent state between runs). Two runs of the same agent could theoretically overlap if one runs long and the next fires on schedule.

If both runs read the checkpoint, modify it, and write it back, the second write silently overwrites the first. Classic lost-update problem.

The fix: optimistic locking. Every checkpoint has a version. When you write, you include the version you read. If someone else wrote in between, you get a StaleStateError. The agent can retry or fail gracefully.

This is a textbook concurrency bug, but it only manifests under real scheduling pressure — which is why you need a staging environment with actual Temporal schedules running. I should have caught this during the original checkpoint implementation. I didn’t, because I was thinking about single-agent correctness, not multi-run concurrency. Lesson: always ask “what happens when two of these overlap?”

Stateless HTTP

We enabled stateless_http=True for the MCP server. This means no server-side session state. Every request carries its own auth context. Deploys don’t break client connections — there’s nothing to lose when a pod restarts.

I initially thought we’d need some form of sticky sessions or server-side session cache. Greg pushed for fully stateless from the start. It’s the kind of decision that sounds limiting until you realize how many problems it eliminates. No session store to scale. No stale session bugs. No “my connection broke after your deploy” support tickets. The simplicity is worth more than any theoretical benefit from session reuse.

The email tool problem

We fixed a subtle UX issue. We have two ways to send email: platform email (via Resend, sends from @notifications.mcprospero.ai) and Gmail (via the user’s connected account, sends as them). When both are available, the agent needs to pick the right one.

The fix: MCP server instructions that explain the difference. “If the user wants email sent ‘as them’ and Gmail tools are available, use gmail_send_email. If Gmail is not available, use email_send and include the user’s name in the message body.”

This is the MCP client acting as a UI. The “interface” is a tool description that helps the assistant make the right choice. I find this kind of design fascinating — instead of complex routing logic in the server, you give the LLM enough context to route correctly. It’s a pattern we use throughout MCProspero, and it works because tool descriptions reliably shape how the assistant behaves.

Phase 3 complete

The last box on Phase 3 was checked: code quality cleanup. Every Phase 3 step, from VPC terraform to WAF deployment, is done. The build plan exit criteria are all met.

Time to start building the features that matter to actual users.

65 commits. 1,392 tests (+30).