-
151
Projects submitted
-
109
Published projects
-
47
Comments left
-
66
Number of votes
-
52
days until voting ends
Observe Platform: open observability with AI root-cause analysis
-
Project manager on customer side
Kamran Karimov
ABB Bank OJSC
Head of Site Reliability Engineering
-
Category
-
Nomination
-
Goals
Give the bank one place to see how every digital service behaves, and find the cause of an incident in minutes instead of days.
1. Replace fragmented, per-team monitoring with one in-house platform built on open standards (OpenTelemetry), owned by the bank's SRE team.
2. Collect metrics, logs and distributed traces from every business service, from mobile banking to core banking and payments, without per-host license costs.
3. Cut the cost of observability: about 1 million AZN saved compared with commercial monitoring licenses.
4. Detect problems before customers report them, with service level objectives (SLOs) and burn-rate alerts per service.
5. Let AI agents work directly on live telemetry, so engineers and developers can ask "what is wrong with my service" and get evidence and a root cause, not a guess.
6. Make every major incident end with a documented, evidence-based root cause and action items.
-
Timeline
january, 2026 — october, 2026 -
Project completion year
2026
-
Project scale
5900 human-hours -
Results
About 1 million AZN saved compared with commercial monitoring and APM licensing.
600+ business services send metrics, logs and traces to one platform, across all production OpenShift clusters.
Tens of thousands of spans and metric points per second processed and stored on-premises.
Root causes established for all major production incidents of 2026, with evidence, using the platform and AI agents.
AI agents used daily: the Grafana MCP endpoint serves about 5,000 requests per day.
Service level objectives and burn-rate alerts in place for key customer journeys, including mobile banking.
Developers self-serve: one AI skill lets any squad triage its own service after a release.
Platform fully delivered as code through GitOps, with documentation built from the same repository.
-
Project uniqueness
Built in-house by a small SRE team, entirely on open-source components, and run as code: every change to the platform is a reviewed Git merge request deployed automatically by GitOps.
The AI layer is the distinctive part. Instead of copying data into a separate AIOps product, AI agents connect to the bank's own telemetry at analysis time through the Model Context Protocol (MCP). They query metrics, logs and traces with the same access an engineer has, and stay read-only.
On top of that the team wrote its own AI skills, versioned and tested like code: a platform knowledge skill, a service health triage skill that any developer can run against their service, and a postmortem skill that gathers evidence, builds the incident timeline and drafts the report. Each skill is evaluated against real incidents before release.
The result is a bank-grade observability platform with AI-assisted root cause analysis, at a fraction of commercial cost and with all data kept inside the bank.
-
Software used
Telemetry collection: OpenTelemetry (SDKs, auto-instrumentation, a sharded collector fleet with tail sampling).
Metrics: Prometheus and Thanos (long-term storage, global query).
Logs: Grafana Loki. Traces: Grafana Tempo. Profiles: Pyroscope.
Visualization and alerting: Grafana, Alertmanager, Sloth for SLOs, blackbox probes for synthetic checks.
Storage: MinIO S3-compatible object storage on the bank's own servers.
Platform: Red Hat OpenShift (Kubernetes); GitOps delivery with Argo CD, Kustomize and Helm; Sealed Secrets.
AI layer: MCP servers for Grafana, Kubernetes, Zabbix and GitLab; AI coding agents (Claude Code, Cursor, Codex) with in-house SRE skills; HolmesGPT evaluated for automated investigation.
Integrations: Zabbix (infrastructure), Jira (incidents and postmortems), Confluence (postmortem pages), GitLab.
Documentation: docs-as-code site built from the same repository.
-
Solution from the Global CIO catalog
The project does not use solutions from the Global CIO catalog
-
Project implementation complexity
Scale and volume: 600+ services on several OpenShift clusters, high-cardinality span metrics, and log volumes that needed sampling and size limits to keep storage costs flat.
Architecture: a sharded OpenTelemetry collector fleet with pinned routing, tail sampling for traces, and separate Prometheus shards for application, platform and self-monitoring metrics, all feeding long-term storage on S3.
Adoption: 59 squads in 14 tribes had to instrument their services and label workloads with their owner. The team ran this as a bank-wide program, with documentation, dashboards per tribe and direct help to squads.
AI in a regulated bank: agents had to get useful access to production telemetry without write access or data leaving the bank. The team chose MCP-based, read-only access over copying data into an external AIOps tool, and built evaluation suites to prove the AI skills give correct answers before release.
Built and operated by the bank's SRE team, with no external integrator.
-
Description
Initial situation
Monitoring was fragmented. Infrastructure was watched in Zabbix, a few applications in a commercial APM tool, and most of the 600+ business services had no shared metrics, logs or traces. Logs lived on servers or in team-specific tools. During an outage several teams looked at different screens, and the cause was often found by guessing or by customer complaints. Expanding commercial APM to the whole bank was too expensive, and the bank required telemetry data to stay on-premises.
Implemented solution
The SRE team built Observe Platform, an in-house observability platform on OpenTelemetry. Applications on all production OpenShift clusters send metrics, logs and traces to a central collector fleet, which routes each signal to its store: Prometheus and Thanos for metrics, Loki for logs, Tempo for traces, all on the bank's own S3 object storage. Grafana gives one view across signals, with links from a metric spike to the exact log lines and traces. Every service gets standard dashboards (Service Health), SLOs with burn-rate alerting, and ownership labels that route alerts to the right squad.
On top of the platform the team added an AI layer. MCP servers expose Grafana, Kubernetes, Zabbix and GitLab to AI agents, so an engineer or developer can ask an agent to investigate a service and get an answer grounded in live data. In-house AI skills encode the team's knowledge: how to query the platform, how to triage a service step by step, and how to write a blameless postmortem with evidence.
Implementation specifics
Everything is delivered as code. One GitOps repository defines the whole platform, and merging deploys it: in 2026 alone, about 1,000 reviewed merge requests. The hardest part was scale: tens of thousands of spans and metric points per second, with high-cardinality span metrics. The team designed a sharded collector fleet where each edge cluster pins traffic to specific collector instances, so per-service metrics stay consistent. A second challenge was trust in AI: agents run read-only in production, every AI skill is versioned, reviewed and evaluated against real incidents before release, and answers must cite the query behind each claim.
Changes after implementation
Incident investigation moved from "who has access to which tool" to one shared view. Engineers now establish the root cause of major incidents with evidence: database stalls in core banking, login failures in identity services, memory exhaustion in payment consumers, and agent crashes after runtime upgrades were all traced to their cause using the platform and AI agents. Developers check their own service health after a release instead of waiting for the SRE team. Postmortems are written from evidence, and action items are tracked in Jira.
Executive sponsor
Nihat Senyuva, CIO, ABB Bank./
-
Project geography
Azerbaijan. The platform runs in the bank's own data centers and covers all production and test OpenShift clusters of the bank. It monitors the services behind every customer channel: mobile and internet banking, cards, payments including instant payments, lending, branches and ATMs, and corporate banking. All telemetry data stays inside the bank's infrastructure in Azerbaijan.
-
Additional presentations
-
Customer
ABB Bank OJSC
Only authorized users can comment.
We suggest you log in or register.