IT Ops Automation
A natural-language operations layer for Kubernetes, with RBAC-scoped reads, human-approved writes, and full audit trails on every action: engineered, integrated and deployed end-to-end by ScroBits.
payment-worker. Memory peaked at 580Mi, limit 512Mi. Likely cause: deploy abc123f added a memory-heavy query.kubectl -n prod set resources deployment/payment-service --limits=memory=1GiWhat was breaking before we started
Routine Kubernetes work was eating the day. Every operational question meant dropping into a terminal, flipping between namespaces, and correlating the cluster with the dashboard.
Tool fragmentation
Engineers pivot across kubectl, Grafana, Teams and the ticketing system to answer one question, losing context on every hop.
Senior-engineer dependency
Anything beyond a status check needs a senior who knows the cluster, so investigations back up behind a small group.
Audit gaps
Ad-hoc kubectl operations leave no reviewable trail for compliance or post-incident analysis.
Change-safety tradeoffs
Automation scripts are either too permissive to trust or too narrow to help, with no middle ground.
What the engineers actually said
A conversational operations layer, with safety rails
Scripts could not solve this. A script with enough permission to be useful had too much permission to be safe. One with safe permissions could not do anything that mattered. The team needed a middle layer that could reason, propose, and wait.
ScroBits built that layer end-to-end: a custom Model Context Protocol server over the Kubernetes API with scoped read and write tools, dual-layer RBAC (Kubernetes API + tool-level policy), Teams approval gates on every write, an append-only PostgreSQL audit log with hash-chained entries, n8n orchestration, and GPT-4o reasoning grounded in GitOps, Prometheus and Loki.
What runs today is a conversational operations surface for every engineer, with reads free, writes paused for approval, and a complete chain of custody on every action. Zero autonomous changes to production.
Every write pauses for a human. Dual-layer RBAC. Hash-chained audit on every action. The agent proposes, the human decides, the cluster executes, the log remembers.
Five building blocks, one governed operations surface
Each block carries a specific job in the proposal-approval-execute loop. Together they make natural-language operations safe enough to put in front of every engineer, including juniors.
MCP Tool Interface
A custom Model Context Protocol server over the Kubernetes API. Scoped read and write tools with permissions, logging and definitions versioned together.
Dual-Layer RBAC
Permissions enforced in the MCP server and at the Kubernetes API. The agent cannot propose what the invoking user cannot see.
Approval Gates on Writes
Teams adaptive cards show the exact command, the reasoning and the blast radius. Nothing runs without a human signature.
Hash-Chained Audit
Append-only PostgreSQL with chained entries. One query reconstructs any decision end-to-end for compliance or post-incident review.
Grounded Reasoning
GPT-4o pulls GitOps history, Prometheus metrics and Loki logs, so answers come from the layer where the problem lives.
Ask
Engineer poses a question in Teams in natural language.
Parse
Agent extracts intent, target namespace and answer shape.
Fetch
MCP runs read tools scoped by the engineer's RBAC.
Reason
LLM correlates signals, drafts an evidence-backed answer.
Propose
Writes emit a Teams approval card with command and blast radius.
Approve
Designated approver clicks approve or reject in Teams.
Execute & Verify
MCP runs the write, verifies outcome, writes the chain to audit.
Before, after, change
The agent shifted the operating model from terminal-and-dashboard pivoting to conversational, evidence-backed decisions, with every action auditable and every write human-approved.
| Dimension | Before | After | Change |
|---|---|---|---|
| Operational question resolution | 10 to 20 minutes across kubectl, Grafana and Teams | 2 to 3 minutes in Teams, with evidence inline | Faster |
| Senior-engineer dependency | Paged on most non-trivial questions | Involved only on escalation or approval | Unblocked |
| Audit coverage on operations | Ad hoc, reconstructed after the fact | Every action logged with user, intent and approval chain | Continuous |
| Junior-engineer autonomy | Read-only, blocked on anything that mattered | Full workflow access with safety rails | Enabled |
| Write-action safety | Script-based, either too permissive or too narrow | Propose, review, approve, execute, verify | Governed |
| Context switching | kubectl -> Grafana -> Teams -> ticket | Single conversational surface in Teams | Collapsed |
Expanding the approval surface
A large financial services evaluator requested expanded scope during a live evaluation, extending the same conversational-plus-approval model deeper into platform governance.
Provisioning
New workload setup through conversation, with policy-as-code checks before approval.
PITR validation
Scheduled point-in-time recovery drills with evidence captured for the audit log.
Access reviews
Quarterly attestation workflows driven through the same approval surface.
Compliance evidence
Continuous collection of control evidence from the audit log for SOC 2 and ISO 27001.
Patch and cloud cost
Conversational queries over patch posture and cloud spend, with actions routed through the same gate.
Three principles we are taking into the next engagement
The agent proposes, the human decides
Reads are free, writes pause for approval. The separation of autonomy and authority is the whole product, not a feature.
Permissions in two places, both must agree
Dual-layer RBAC (Kubernetes API + MCP tool filter) is defense-in-depth, not duplication. The agent cannot propose what it cannot see.
Audit is a feature, not a side effect
Hash-chained append-only logs make every decision replayable end-to-end for compliance, review and post-incident reconstruction.
Running Kubernetes and want operations to earn its keep?
ScroBits partners with ITOps and platform teams on conversational AI with safety rails. Architecture, full-stack build, production rollout.