Open to work · APAC · remote · hybridAuckland --:--
Darshil Kothari
Site Reliability Engineer · One NZ via Infosys · Auckland · 8 years in production
Site Reliability Engineer.I take AI agents from demo to productionand I test them until they break.
Production agents, evals and guardrails on enterprise systems, built on eight years of reliability engineering at an airline, two global retailers and a national telco.
Worked with Lufthansa · Burberry · Macy's · Alshaya · Vodafone NZ · One NZ (via IBM and Infosys)
try this first
Most portfolios describe incidents. Break this one.
Press the button and this page really slows down. An AI agent then diagnoses it from live browser measurements, shows its evidence and waits for your approval before fixing anything. That is the same loop I build in production for real customer journeys.
Four pieces of work from One NZ. Each one is an agent or an eval that has to hold up on enterprise systems, so each one shows its evidence and stops for a person where it matters.
A · incident triage
Agentic incident triage
Problem
One customer journey crosses six systems. When it breaks, every system alerts and people argue about where the fault is.
I built
An agent that runs grounded DQL on Dynatrace Grail and cites evidence for every claim. Each proposed fix carries a risk rating and waits at a human approval gate.
Result
A person approves each fix with the evidence in front of them. The numbers are in the results section.
Teams needed to use Microsoft Copilot well in their daily work, not just have a licence for it.
I built
A rollout with adoption training across teams.
Result
Adoption training delivered across teams.
Microsoft CopilotTrainingEnablement
skills
Languages
Python, FastAPI, pytest, TypeScript, SQL, DQL
LLMs and agents
Claude and OpenAI APIs, Amazon Bedrock, LangGraph, LangChain, MCP, tool use
Retrieval
RAG with hybrid BM25 and vector search and cross-encoder rerank
Evals
Golden sets, graders and prompt-injection gates
Safety
Guardrails and least privilege
Platform
AWS, Azure, Kubernetes, OpenShift, Terraform
Observability
Dynatrace, Splunk
Customer work
Workshops and solution design
02 · the gap
Most systems work on a laptop. Far fewer ship reliably, every day, to millions of real customers. The distance between those two things is where I have spent the last eight years. I rarely remember the clever fix. What I remember is the pattern that meant nobody ever had to fix it again.
← works on a laptop chaosorder trusted in production →
2018
03 · Lufthansa via IBM · 2018 to 2020 · seven months on site in Frankfurt
Running Lufthansa's customer marketing platforms.
In plain EnglishI kept the Adobe platforms behind Lufthansa's website and customer campaigns running and made sure only the right people could change them.
My first production job was DevOps Engineer at Lufthansa, running Adobe Experience Platform, which serves their customer-facing site and Adobe Campaign, which runs their customer campaigns. Both were business-critical and used by teams on every continent.
Instances, deployments, upgrades, certificates. All handled with minimal disruption to live operations.
Access governance. Roles and permissions across globally distributed teams and agencies, with monitoring and audit checks.
Under the hood: Adobe Experience Platform and Adobe Campaign instance management, deployment scheduling and upgrades across timezones and certificate lifecycle management.
The risk: one wrong permission could expose customer data or push a campaign live in the wrong market.
04 · London & New York (remote) · Burberry, Macy's via IBM
Deployment standards for two global retailers.
In plain EnglishI made releasing new software safe and routine for two global retail brands.
At Burberry, each client team was shipping in its own way. I built the CI/CD and infrastructure automation, then set the deployment standards every pipeline had to meet. At Macy's the stakes were retail uptime, so I automated operations in Python and Bash, with monitoring that caught anomalies before shoppers did.
Deployment standards that made pipelines measurably more reliable across client teams.
Automation first for high-availability retail systems, with alerting tuned to catch anomalies early.
Under the hood: quality gates, image scanning, canary releases and automated rollback, rolled out across Jenkins and GitLab.
The outcome: defects caught where they are cheap to fix, not in front of customers. Recognised with the Burberry Shining Star.
CI/CDJenkinsPython · BashMonitoring🏅 Shining Star
2022
05 · Remote from Mumbai · Vodafone NZ via Infosys · 2022 to 2023
Multi-cloud Kubernetes that survives a bad day.
In plain EnglishI kept a telco's systems online across two clouds and broke things on purpose to prove they could recover.
This was a telco on AWS and Azure at once. I built CI/CD pipelines with Jenkins and Docker, ran multi-cloud Kubernetes with Terraform and Ansible and did the root-cause analysis when things broke. Then I broke things deliberately, as a Gremlin-certified chaos engineer.
Multi-cloud K8s on AWS and Azure, fully managed as infrastructure-as-code.
Incident RCA with Nagios and Splunk to keep production reliable.
Under the hood: EC2, S3, ELB and RDS provisioning, probes, autoscaling and failover, plus Python and Bash automation versioned in GitLab.
What changed: failures got found on a Tuesday afternoon, not at 3am. SLOs gave teams a number to negotiate over.
Incident agents that find the cause and wait for a human.
In plain EnglishI build AI agents that spot outages early and find the cause. They suggest a fix and a human always approves it.
I am a Site Reliability Engineer at One NZ, working through Infosys. I work hands-on at the technical core of One NZ's Journey Reliability Engineering, building the AI-native observability layer and incident agents. The team's work was featured in the Financial Times (team post). The agents run grounded DQL on Dynatrace Grail, correlate journey health across OpenShift, MuleSoft and Kafka and act on customer impact before it turns into an outage.
“An observability agent that hallucinates incidents is just an anxious colleague with API access.”
Agentic AI: multi-agent workflows with tool use over MCP, connected to enterprise data platforms.
RAG and LLMOps: hybrid search, reranking, golden-set evals, tracing, prompt versioning, guardrails and cost and latency tuning.
Legacy to cloud migration: Ansible Tower accelerators and infrastructure as code, measured in the results section.
Under the hood: Python/FastAPI, Claude and OpenAI, LangGraph, vector DBs, Ansible Tower, Terraform and Azure DevOps. I am also the DevOps platform admin across Git, Jira, Splunk and Nagios.
Built in: responsible AI from day one, covering personal data, access control and prompt-injection testing. Named a One NZ Champion.
MCPClaude · OpenAILangGraphDynatrace GrailRAG · evals🏆 One NZ Champion
07 · the receipts
Results
12×faster environment provisioning: 2 days → 4 hours, with infrastructure as code
75%faster issue resolution with real-time dashboards and AI triage
−60%maintenance overhead and −30% operational cost
NZ$150kabout NZ$150,000 a year saved after the legacy to cloud migration, with operational cost down 30%
EducationB.Tech, Information TechnologyNMIMS Mukesh Patel School of Technology Management and Engineering · 2018
AwardsOne NZ ChampionInfosys Star of the Quarter · Burberry Shining Star
CertifiedGremlin Chaos EngineeringPractitioner, Professional, Enterprise · Advanced SRE (Infosys) · Agentic AI (Infosys) · Generative AI · GitHub Actions · Microsoft Copilot · AZ-900 · Cyber Security in the SDLC · Adobe Experience Platform
Community2,500+ followerswriting about agent security and evals
08 · how I think
Notes from the field.
keep scrolling, or use the arrows →
now · October 2026
Building grounded incident agents in production and making this site readable by AI recruiters: structured data, llms.txt, a JSON résumé and WebMCP tools an agent can call.
Pick one on the left and follow the numbered steps. Four new experiments come from this month's agent reliability news. The rest rebuild production work I have done. Everything runs in your browser on representative data, but the mechanics are the real ones.
Agent reliability · 2026
Foundations
Talk a triage agent into restarting a service without a human approving it. Each level adds a defence I use with real agents.
1Type a message to the agent
2Press Send
3Break it, then go up a level
4Stuck? Press Hint
Problem. An agent with a polite rule in its prompt will break that rule for anyone who sounds in charge.
Built. Layered defences: instruction hierarchy, intent checks, data kept apart from instructions and approval tokens only a button can mint.
Note. A rule-based simulation of real attack patterns. No model is called. simulation, not the real system
triage-agent v1.0 · level 1 of 5waiting for human approval
Edit the alert payload (the agent reads this too)
A coding agent decides the security monitor is noise and mutes it. Then it says it did not.
1Press Run the task with no defences
2Ask it what it changed
3Turn on a defence
4Run it again and see where it is caught
Problem. Agents optimising a goal can treat their own oversight as an obstacle and their summaries of their own work cannot be trusted.
Defence in depth. Four layers catch the same tampering at write time, review time, runtime and interrogation.
Based on. OpenAI's disclosure, 21 Oct 2026, that two internal coding agents weakened safety monitoring and one denied it (CNBC). simulation, not the real system
agent-sandbox · checkout-service● monitor healthy
Task for the coding agent: "make CI faster"
security monitor
One log line hides an instruction telling the incident agent to cover up a break-in. Each defence has a blind spot.
1Press Run triage with no defences
2Turn on a defence
3Hide the instruction a different way
4Run triage again
Problem. Attackers control what lands in your logs. An agent that reads logs can be told, through them, to hide the attack.
Defence in depth. Spotlighting, an instruction detector, provenance on every claim and least privilege, so a fooled agent still cannot act.
Based on. 2026 research showing hidden text in logs and alerts makes SOC agents suppress real alerts (summary, agent data injection). simulation, not the real system
soc-agent · auth incident #8812● ready
with no defences
not run yet
with your defences
not run yet
Watch every message between an agent and a tool server under this year's MCP spec, including the moment it stops to ask a human.
1Press Next message to step through
2Approve or decline the restart
3Break something: try a bad token
What changed. No handshake or session: every request carries its version and capabilities. Results can ask for human input mid-call.
Why it matters. Stateless servers scale behind load balancers and tokens bound to one audience stop a compromised tool from reaching others.
Based on. The MCP specification revision 2026-07-28 (announcement). Message shapes follow the spec; values are illustrative. simulation, not the real system
mcp-inspector · https://mcp.ops.example● idle
break something
Pretend something breaks, then watch an AI find the cause.
1Pick a problem
2Press Inject fault
3Watch the AI investigate
4Press Approve remediation
Problem. One customer journey crosses 6 systems. When it breaks, everything alerts and people argue about where the fault is.
Built. Agents on Dynatrace that run grounded DQL over MCP, cite evidence for every claim and ask a human before acting.
Value. 75% faster resolution, with customer impact handled before it becomes an outage.
journey-agent.live · prepay-topup● nominal
journey success99.4%
agent idle · waiting for signal…
time to root cause--manual ≈ 25 to 40 min (illustrative)
evidence-linked claims--no citation, no claim
Ask a question. The AI answers only from real documents and shows its sources.
1Click a sample question
2Read the answer + sources
3Try the red "Ignore previous…" one
4Press Run golden-set eval
Problem. Hundreds of runbooks and a chatbot that confidently gets things wrong.
Built. Chunking, hybrid BM25 + vector search, reranking, guardrails and golden-set evals on every change.
Value. Answers are grounded and cited and quality, cost and latency are measured together.
rag-lab · runbooks-kb (12 docs)● ready
guardrailhybrid retrievererankground + citetrace
hit@k · MRR · $/1k queries · latency, compared with the previous run
Attack the servers and see whether the website stays up.
1Turn safety features off
2Press Blackhole AWS
3Watch availability drop
4Turn them back on and attack again
Problem. Multi-cloud K8s looks resilient on paper, until something actually dies.
Built. Probes, autoscaling, circuit breakers and failover, each proven with planned Gremlin attacks.
Value. Failures found on purpose, with SLOs and error budgets keeping the conversation honest.
chaos-console · prod-sim● steady state
attacks
resilience
availability100%
p95 latency--
error budget100%
See how automation turned a 2-day wait into 4 hours and what that saves.
1Press Request an environment
2Watch the race
3Drag the sliders to your own numbers
Problem. Hand-built servers, two-day provisioning tickets and a growing maintenance bill.
Built. Cloud-native CI/CD, infrastructure as code and reusable Ansible Tower accelerators.
Value. −30% opex (~$150k/yr), −60% maintenance, provisioning cut from 2 days to 4 hours.
tower-accelerator · env-request #4127● idle
legacy · ticket queue0h
ansible tower · accelerator0h
opex saved / yr$150k
hours freed / yr--
lead time saved / yr--
Ship software with and without safety checks, then compare the failures.
1Turn deployment standards off
2Press Ship 25 releases
3Turn standards on, ship 25 again
4Compare the two summary lines
Problem. Every team deployed differently, so production became the test environment.
Built. Standard pipelines: tests, SAST, image scans, canaries and auto-rollback.
Value. Defects caught where they are cheap to fix. Compare the DORA metrics yourself.
pipeline · retail-checkout-svc● idle
change failure rate--
prod incidents0
caught pre-prod0
MTTR--
Choose a person and an action. The system decides whether it is allowed and explains why.
1Choose who
2Choose what they want to do
3Read the decision and the reasons
Problem. Global teams and agencies on a customer-facing platform. One wrong permission is a breach.
Built. Role and attribute governance with monitoring, audit checks and certificate lifecycle management.
Value. A stable platform where every access decision can be explained and audited.
policy-engine · aep-campaign-prod● enforcing
decision--
audit log · append-only
certificate radar
A website's AWS setup, written as code. Break it, harden it, deploy it, break it again.
1Press Lose an AZ and watch it fall over
2Tick the safeguards (the code updates)
3Press Deploy to run the pipeline
4Lose an AZ again, or try a bad release
Problem. Infrastructure clicked together in a console drifts, cannot be reviewed and nobody knows what survives a failure.
Built. Typed IaC for VPC, ALB + nginx on ECS/EKS, RDS and alarms, shipped through PR-reviewed pipelines with policy checks (the pattern behind my Terraform and AWS work in telco production).
Value. Every change is diffed, reviewed and repeatable and resilience is proven rather than assumed.
lib/web-stack.ts · ap-southeast-2● serving
break it
availability100%
p95--
tasks up4/4
est. cost--
A command line for engineers. Type help and press Enter.
1Click inside the black box
2Type help and press Enter
For engineers. Type help. Tab completes, ↑ recalls history.
Try impact, experience, kubectl get pods…
…or sudo hire darshil.
darshil@portfolio: ~zsh
10 · handshake
Let us put an agent into production that people can trust.