As a Site Reliability Engineer, you balance the day-to-day operations on production systems with long-term software engineering improvements to reduce operational toil and foster the reliability, availability, and performance of these systems.
• Design, build, and maintain scalable, highly available and fault-tolerant infrastructures to support our web services and ML workloads
• Make sure our platform, inference and model training environments are always highly available and enable seamless replication of work environments across several HPC clusters
• Operate systems and troubleshoot issues in production environments (interrupts, on-call responses, users admin, data extraction, infrastructure scaling, etc.)
• Implement and improve monitoring, alerting, and incident response systems to ensure optimal system performance and minimize downtime
• Implement and maintain workflows and tools (CI/CD, containerization, orchestration, monitoring, logging and alerting systems) for both our client-facing APIs and large training runs
• Participate occasionally in on-call rotations to respond to incidents and perform root cause analysis to prevent future occurrences
• Drive continuous improvement in infrastructure automation, deployment, and orchestration using tools like Kubernetes, Flux, Terraform
• Collaborate with AI/ML researchers to develop and implement solutions that enable safe and reproducible model-training experiments
• Build a cloud-agnostic platform offering an abstraction layer between science and infrastructure
• Design and develop new workflows and tooling to improve to the reliability, availability and performance of our systems (automation scripts, refactoring, new API-based features, web apps, dashboards, etc.)
• Collaborate with the security team to ensure infrastructure adheres to best security practices and compliance requirements
• Document processes and procedures to ensure consistency and knowledge sharing across the team
• Contribute to open-source projects, research publications, blog articles and conferences