OPEN SOURCE KUBERNETES INCIDENT AGENT

Your monitoring stack can tell you the pod is dying. 01Agents brings it back.

An open-source Kubernetes incident resolution engine. It pairs real-time cluster triage (L1) with secondary telemetry reconstruction (L2) via SigNoz, Grafana Loki & Prometheus to diagnose root causes and suggest verified GitOps fixes.

01. 404 Resilient
Postmortem Log Retrieval
Queries SigNoz & Loki when deleted pods return 404 from K8s API.
02. Zero Code Changes
Non-Invasive Deployment
No SDK or code instrumenting required. Plugs into current OTel & logs.
03. HITL Guardrails
Operator Approval Gate
High-risk cluster remedies pause for human approval before execution.
04. GitOps Verified
ArgoCD & FluxCD Sync
Verifies declarative manifest state and rollback snapshots automatically.
NATIVE TELEMETRY & GITOPS INTEGRATIONS

Supported Native Observability & GitOps Stack

Zero application code modifications required. 01Agents natively interface with your cluster's existing telemetry collectors, log streams, metric stores, and GitOps delivery engines.

SigNoz(L2)
Grafana Loki(WebSocket)
Grafana Alloy(Telemetry)
Prometheus(Real-Time)
ELK Stack(Elasticsearch)
Datadog(Enterprise)
ArgoCD(Declarative)
FluxCD(Kubernetes)
SigNoz(L2)
Grafana Loki(WebSocket)
Grafana Alloy(Telemetry)
Prometheus(Real-Time)
ELK Stack(Elasticsearch)
Datadog(Enterprise)
ArgoCD(Declarative)
FluxCD(Kubernetes)
Native

SigNoz

L2 Telemetry & ClickHouse

Native integration with SigNoz v5 for historical ClickHouse log queries & distributed APM tracing.

ClickHouseAPM TracingL2 Reconstruction
Native

Grafana Loki

WebSocket Live Logs & Pod Storage

Direct Loki Tail WS proxy & PVC volume mounting to retrieve logs even from terminated & deleted pods.

Live WS ProxyPostmortem LogsPVC Storage
Native

Grafana Alloy

Telemetry Collector Pipeline

Next-gen OpenTelemetry pipeline & agent for streaming node-level metrics and container logs.

OTel CollectorMetric StreamingUnified Agent
Native

Prometheus

Real-Time K8s Metrics

PromQL metric parsing for instant CPU, memory, OOMKilled, and crash-loop backoff detection.

PromQL QueriesAlertmanagerNode Exporters
Native

ELK Stack

Elasticsearch & Kibana

Seamless search & indexed log analysis via Elasticsearch REST APIs and Kibana dashboard audit sync.

ElasticsearchLogstashKibana Audit
Native

Datadog

Enterprise Cloud Telemetry

Bi-directional Datadog monitor webhook intake & automated incident remediation triggers.

Cloud APMMonitor WebhooksEnterprise Sync
Native

ArgoCD

Declarative GitOps Delivery

Automated GitOps state verification, commit rollback, and sync status checks upon incident resolution.

GitOps SyncAuto-RollbackK8s Declarative
Native

FluxCD

Kubernetes GitOps Operator

Native reconciliation with Flux Kustomization and HelmRelease objects for safe cluster state restoration.

HelmRelease SyncFlux KustomizeContinuous Delivery
ARCHITECTURE SPECIFICATION

Why Standard K8s Diagnostic Tools Fail on Pod Termination

When a pod enters CrashLoopBackOff or gets evicted due to OOMKilled, Kubernetes eventually removes the pod resource. Running kubectl logs or querying the API server at that point returns:

Error from server (NotFound): pods "auth-service-7f9b" not found

01Agents solves this by decoupling live cluster event triage (Level 1) from postmortem telemetry query execution (Level 2):

Level-1 (L1) In-Cluster Triage AgentMonitors real-time Kubernetes events, node health, and container state transitions. Evaluates immediate low-risk remedies and handles active resource constraints.
Level-2 (L2) LangGraph Telemetry EngineWhen live APIs return 404, L2 initiates a lookback query across secondary backends (SigNoz ClickHouse, Grafana Loki WS proxy, Prometheus PromQL) to reconstruct full pre-failure log streams and memory curves.
incident-triage-trace.log
L1 → L2 Pipeline
17:04:12 [L1 Monitor] Event Received
Reason: OOMKilled | Pod: auth-api-9d2a | Namespace: production
17:04:13 [L1 Monitor] Pod Evicted (404 API)
K8s API: Pod deleted. Initiating L2 Secondary Telemetry lookup...
17:04:14 [L2 Telemetry Agent] Querying ClickHouse & Loki
SigNoz ClickHouse lookup window: [-15m]. Fetched 1,420 historical log entries.
17:04:15 [L2 RCA Agent] Root Cause Verified
Heap exhaustion in auth-v2.1 binary (Limit: 512Mi | Usage Peak: 518Mi).
17:04:16 [L2 Remediation Agent] Patch Generated
Increase limit to 1Gi + trigger Helm update. Risk Assessment: HIGH.
17:04:17 [HITL Safety Gate] Approval Awaited
High-risk patch queued. Awaiting Slack callback from operator @sre-oncall.
17:04:22 [HITL Safety Gate] Approved & Executed
Operator APPROVED. Patch applied with GitOps rollback snapshot.
Active Trace LogLookback: -15mRCA Confidence: 0.96
AGENT CAPABILITIES MATRIX

Domain-Specific Intelligence Across L1 Triage & L2 RCA

From live event monitoring to secondary telemetry postmortems, 01Agents handle targeted Kubernetes failure patterns out of the box.

Level-2 Secondary Telemetry Engine
L2 Telemetry Agent

Secondary Telemetry Reconstruction

Connects to SigNoz, Grafana Loki, and Prometheus to pull pre-failure logs, metric trends, and APM traces after pod deletion.

SigNoz · Loki · PrometheusActive
L2 RCA Agent

Multi-Agent Root Cause Analysis

Executes a 4-agent LangGraph workflow to cross-correlate log stacktraces, OOM events, and Git commit history.

LangGraph WorkflowActive
L2 Safety Agent

Human-in-the-Loop Safeguards

Auto-applies low-risk remedies while queuing high-risk cluster state mutations for SRE confirmation via Slack or API.

HITL GatekeeperActive
Level-1 Real-Time Pod Failure Skills
CrashLoopBackOffL1 Event Skill

CrashLoop Triage

Diagnoses container crash loops, missing env variables, panics, and unhandled exception stack traces.

Real-Time Triage
OOMKilledL1 Event Skill

Memory Exhaustion

Traces heap spikes, memory leaks, container memory limit breaches, and recommends scaled limits.

Resource Limits
ImagePullBackOffL1 Event Skill

Registry & Image Audit

Verifies container image availability, registry authentication tokens, and secret volume mounts.

Image & Credentials
CreateContainerErrorL1 Event Skill

Runtime Diagnostics

Identifies container runtime failures, cgroup misconfigurations, and volume mount permission errors.

Runtime Failure
FailedSchedulingL1 Event Skill

Scheduling Taints & Affinities

Analyzes node taint mismatches, unschedulable pod requirements, and node resource shortfalls.

Scheduler Analysis
NonZeroExitCodeL1 Event Skill

Process Exit Analysis

Correlates process exit codes (e.g. exit 137, 127) with binary startup scripts and entrypoint parameters.

Exit Code Tracing
l2-telemetry-audit-#4821.json
// L2 Secondary Telemetry Decision Entry
{
"agent_tier": "L2_LangGraph_Pipeline",
"telemetry_backend": "SigNoz_V5_ClickHouse",
"lookback_window": "-15m",
"logs_reconstructed": 1420,
"rca_confidence": 0.96,
"hitl_approval": "OPERATOR_APPROVED",
"rollback_available": true
}
Observability & Integrations

Enterprise-Grade Observability & Instant Incident Auditability

Every L1 triage decision and L2 secondary telemetry analysis is fully logged, exportable, and queryable. 01Agents integrate natively into your existing observability and GitOps stack without modifying application code.

Decision history is queryable via API, exportable in JSON or CSV, and structured for postmortem review. When something needs to be explained — the exact evidence and RCA path are ready for audit.

Full Decision Provenance
Complete L1/L2 audit trails for postmortem analysis.
Instant Multi-Format Export
Export structured evidence in JSON, CSV, or query via API.
HITL Safety Controls
Human approval guardrails for critical cluster actions.
GitOps Verified Rollbacks
Automated rollback verification via ArgoCD & FluxCD.
TECHNICAL GUARANTEES

Engineered for Production Kubernetes Clusters

Designed from the ground up to solve postmortem log loss, prevent accidental cluster mutations, and integrate natively into existing DevOps tooling.

Telemetry PersistenceSigNoz / Loki API

Postmortem Log Reconstruction

Queries ClickHouse (SigNoz) & Loki WebSocket streams to retrieve logs after pods are deleted or terminated.

Production Tested
Safety & GovernanceHuman-in-the-Loop

Deterministic HITL Approval Gates

Classifies remediation actions by risk level. High-risk mutations (scaling, config patches) require operator approval.

Production Tested
Agent ArchitectureLangGraph Engine

LangGraph 4-Agent Workflow

Orchestrates dedicated agents for Triage, Telemetry Querying, Root Cause Analysis (RCA), and Remediation Generation.

Production Tested
State ConsistencyArgoCD & FluxCD

GitOps Rollback Verification

Verifies application declarative state with ArgoCD and FluxCD before marking any incident resolved.

Production Tested
Non-Invasive SetupOperator / Agent

Zero Application Code Instrumentation

Operates entirely at the cluster control plane & telemetry backend layer without requiring app SDK changes.

Production Tested
Audit ProvenanceJSON / CSV Audit API

Exportable Incident Decision Records

Outputs structured JSON/CSV decision traces for postmortems, compliance reviews, and team handoffs.

Production Tested
WORKFLOW SPECIFICATION

Four Steps from Incident to Verified Fix

How 01Agents integrates into your deployment pipeline and handles pod failures from detection to resolution.

01

Cluster Event Monitoring & Ingestion

Deploys non-invasively into your Kubernetes cluster to listen for real-time pod events, OOMKilled signals, and state transitions.

02

Secondary Telemetry Backend Query

When a pod terminates and returns 404, 01Agents automatically initiates a lookback query across SigNoz ClickHouse, Grafana Loki, and Prometheus.

03

Multi-Agent Root Cause Analysis

The L2 LangGraph engine correlates pre-failure logs, metric memory spikes, and container configuration history to pinpoint the exact root cause.

04

Human-in-the-Loop Safeguards & GitOps Sync

Low-risk remedies execute automatically, while high-risk changes pause for SRE confirmation before verifying manifest restoration with ArgoCD or FluxCD.

OPEN SOURCE · APACHE 2.0 LICENSE

Start Resolving Kubernetes Incidents Automatically

Install the 01Agents operator on your cluster, connect SigNoz or Grafana Loki, and stop losing logs when pods are deleted.

Maintained by BerryBytes & 01Cloud Open Source Community