•
Performs complex reliability engineering work autonomously; recognized subject matter expert within the team and adjacent teams.
•
Designs and builds production software systems, reliability tooling, and automation frameworks; treats operational problems as engineering problems to be solved through code.
•
Owns SLO governance, error budget management, and observability architecture for assigned systems; leads engineering-driven incident response including failover scenarios.
•
Holds NERC/CIP compliance responsibility for assigned systems; formally mentors less experienced specialists; may coordinate team delivery and on-call activities.
The following expectations apply at all Systems Reliability Specialist levels. Scope and independence expand with each level.
•
Engineer reliability solutions: when a process is manual and repeatable, automate it; when a failure mode is opaque, instrument it; when a system is fragile, redesign the failure boundary.
•
Define and own SLIs and SLOs for assigned systems; treat error budgets as a shared engineering contract with development teams, not an operations metric.
•
Respond to production incidents as an engineer: form a hypothesis, isolate the failure, resolve it, and close the loop with a post-mortem that addresses root cause.
•
Instrument systems so that on-call responders have sufficient telemetry to diagnose and act without tribal knowledge.
•
Participate in 24/7 on-call rotation; treat every alert as signal — either actionable or worth eliminating.
•
Write production-quality code: reliability tooling, automation frameworks, and operational software are held to the same engineering standards as application code.
•
Partner with development teams as a peer in design reviews; reliability is designed in, not bolted on after deployment.
Senior specialists design and build the engineering systems that make production software reliable. This is software engineering applied to operational problems — the output is code, frameworks, and automated systems, not tickets and runbooks alone.
•
Design, build, and maintain reliability tooling: automated remediation systems, self-healing infrastructure components, and operational software that reduces human intervention in production.
•
Own SLO and error budget definitions for assigned systems; review error budget consumption with development teams and drive engineering decisions based on budget status.
•
Architect and implement chaos engineering programs: define failure injection scenarios, automate resilience tests, and validate recovery behavior against defined SLOs.
•
Build and maintain CI/CD reliability gates: automated canary analysis, progressive delivery validation, and rollback triggers based on SLI thresholds.
•
Design capacity planning models for assigned systems; build tooling to project resource needs and surface capacity risks before they affect availability.
•
Contribute to production readiness reviews: define and enforce the engineering criteria that a system must meet before it ships to production.
•
Reduce operational toil through engineering: measure toil, track reduction targets, and build the automation that eliminates it.
Observability & Instrumentation
Observability is an engineering discipline. Senior specialists design and build the telemetry systems that make production behavior understandable — not just monitored.
•
Architect MLTP (Metrics, Logs, Traces, Profiling) observability solutions using the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir), Dynatrace APM, Splunk, and Datadog.
•
Define and enforce instrumentation standards: structured logging schemas, metric naming conventions, trace context propagation, and continuous profiling configuration for assigned systems.
•
Build distributed tracing coverage across service boundaries; identify and close observability gaps that produce blind spots during incidents.
•
Design SLI instrumentation: translate user-facing reliability requirements into specific, measurable signals that accurately represent system health from the user’s perspective.
•
Build and maintain alerting frameworks: alerts must be actionable, calibrated to SLO burn rate, and free of noise; own alert quality as an engineering output.
•
Correlate application performance data — JVM heap behavior, GC pressure, thread contention — with infrastructure events to enable root cause analysis across layers.
Incident Response & Problem Management
Incident response at this level is an engineering activity. Senior specialists lead the technical response to high-severity events, own the post-mortem process, and drive the engineering work that prevents recurrence.
•
Lead high-severity incident response for assigned systems, including dual-datacenter failover execution; own the technical resolution from detection through remediation.
•
Apply structured root cause analysis: distinguish symptoms from causes, identify contributing factors across system layers, and drive remediation that addresses root cause rather than surface behavior.
•
Author post-mortems that produce actionable engineering work items — not process improvements alone; track remediation to completion and validate effectiveness.
•
Diagnose complex cross-layer failures: Java/JVM application failures, distributed system race conditions, database connection pool exhaustion, messaging system backpressure, and cross-datacenter synchronization issues.
•
Build and maintain incident response runbooks as engineering artifacts: automated where feasible, version-controlled, and validated during chaos engineering exercises.
•
Participate in blameless post-mortem facilitation; model the engineering culture that treats incidents as system failures, not human failures.
Java Application Reliability
The primary application platform is Java/Spring Boot. Senior specialists are expected to operate at the intersection of application engineering and reliability — understanding the runtime deeply enough to diagnose, tune, and improve production behavior.
•
Diagnose and resolve Java application performance problems in production: heap memory pressure, garbage collection tuning, thread pool exhaustion, connection leak detection, and class loading anomalies.
•
Perform JVM performance analysis using heap dumps, thread dumps, and continuous profiling; translate findings into engineering recommendations for development teams.
•
Instrument Spring Boot applications with production-grade observability: Micrometer metrics, structured logging with correlation IDs, and distributed trace integration.
•
Diagnose failures across the Java application stack: Spring Boot service behavior, PostgreSQL and Oracle query performance, Kafka and ActiveMQ messaging reliability, and REST/SOAP API integration failures.
•
Contribute to Java application design reviews with a reliability lens: identify failure modes, single points of failure, and observability gaps before code ships to production.
Platform & Infrastructure Engineering
Senior specialists build and maintain the platform engineering components that reliability depends on — container orchestration, infrastructure automation, deployment tooling, and environment governance.
•
Design and operate Kubernetes and OpenShift workloads for reliability: resource quotas, pod disruption budgets, horizontal pod autoscaling, and liveness and readiness probe engineering.
•
Build infrastructure-as-code for reliability infrastructure: Terraform modules, Ansible/AAP playbooks, and Azure Resource Manager templates that are tested, version-controlled, and peer-reviewed.
•
Own dual-datacenter reliability architecture for assigned systems: synchronization validation, automated failover triggering, traffic management, and recovery time objective verification.
•
Design and automate environment promotion pipelines: ensure that configuration, secrets, and infrastructure state are consistent and validated across development, test, staging, and production.
•
Build and maintain automated patch compliance workflows; integrate CVE remediation into CI/CD pipelines rather than treating it as a manual operational process.
NERC/CIP compliance for assigned systems is an engineering responsibility at this level — not a documentation exercise. Senior specialists implement controls through code and automation wherever possible.
•
Own NERC/CIP compliance for assigned systems: interpret applicable reliability standards, implement required controls, maintain evidence documentation, and prepare for regulatory audit.
•
Engineer compliance controls into the platform where possible: automated hardening scripts, configuration drift detection, access control validation, and audit log integrity verification.
•
Maintain currency on applicable NERC/CIP standards and ERCOT-specific regulatory requirements; escalate emerging compliance risks to the Lead or Manager.
•
Participate in regulatory audit preparation: produce control evidence, respond to auditor inquiries, and coordinate with compliance stakeholders on findings remediation.
Technical Leadership & Mentoring
•
Hold formal mentoring responsibility for Systems Reliability Specialist I and II team members: structured coaching on SRE practices, code review for reliability tooling, and career development conversations.
•
Serve as the recognized technical authority on reliability engineering and Java application operations for the team; adjacent teams and development engineers seek out this specialist for guidance.
•
Lead design reviews for systems within the team’s scope; identify reliability risks and observability gaps before systems reach production.
•
Set engineering standards for the team: post-mortem quality, observability instrumentation, chaos engineering practices, and on-call readiness.
•
Contribute to the broader engineering organization: internal technical talks, SRE practice documentation, and shared tooling that other teams can adopt.