Founding Engineer, AI Infra
Read straight from Cox Exponential's careers page, verified 3d ago.Seekless can tailor your résumé to it.
Seekless can write your résumé for this posting first.
Before you apply
LocationSan Francisco Bay Area, CA, SF Bay Area, United States
Work modehybrid
Typefull-time
SenioritySenior
Experience5+ yrs
DepartmentGoaly
Company size10+ people
PostedJun 11, 2026 · 4mo ago
Verified live3d ago
At a glanceSummarised by Seekless from the posting.
5+ years building or operating ML infrastructure at scale, ideally supporting large language or multimodal models
Deep understanding of GPU architecture, distributed training frameworks (PyTorch, DeepSpeed, Megatron, Ray), and parallelism strategies
Hands-on experience running inference stacks (vLLM / SGLang, TGI, Triton) and optimizing them via low-level profiling
Strong software engineering fundamentals in Python and one of C++/Rust/Go, with clean, reliable code shipped to production
Working knowledge of modern data pipelines, feature stores, and vector databases used in production AI systems
Comfort automating infrastructure with Kubernetes, Terraform/Pulumi, and observability stacks (Prometheus, Grafana, OpenTelemetry)
Experience deploying open-source LLMs (Llama 3, Qwen, DeepSeek) or training custom foundation models
Contributions to ML systems tooling (compilers, kernels, inference runtimes) or open-source infrastructure projects
Background in reinforcement learning, evaluation harnesses, or alignment tooling that hardens production AI systems
Skills
Python
C++
Rust
Go
Kubernetes
Terraform
Pulumi
Prometheus
Grafana
OpenTelemetry
PyTorch
DeepSpeed
Megatron
Ray
vLLM
SGLang
TGI
Triton
FlashAttention
CUDA
Llama 3
Qwen
DeepSeek
About Goaly
At Goaly, our mission is to make custom AI affordable for every business. Our founding team comes from the front lines of top AI labs and tech giants (Meta MSL, TikTok AI, Google DeepMind, xAI, Microsoft Research, etc.), where we built large-scale training infrastructure powering trillion-parameter models and scaled GenAI models to a global user base. Now, we are building something we wish we had before: a platform that makes training and adapting custom AI affordable for all modern companies, not just Big Tech. Our north star is ambitious: for a domain-specific task, reach 90% of SOTA performance at less than 10% of the cost. To get a taste of what we are doing, see our first tech blog.
About the Role
You will sit at the intersection of systems engineering and applied ML, building specialized infrastructure that keeps large language and multimodal models fast, reliable, and cost-effective. You will partner with research, product, and infra teams to ship production-ready platforms for training and serving AI at scale.
Key Responsibilities
•
Efficiency & performance: Improve LLM training and inference efficiency through better memory utilization, optimized parallelism, and kernel-level innovations (e.g. FlashAttention, CUDA/Triton).
•
Training & RL robustness: Build scalable, stable training and RL pipelines with strong reproducibility, observability, and debuggability.
•
Serving & inference optimization: Design and tune high-throughput, low-latency model serving systems, including quantization, caching, and speculative decoding.
•
Scalability & infrastructure: Own end-to-end training and inference infrastructure — from data ingestion and checkpointing to multi-GPU and multi-cloud orchestration.
•
Production enablement: Work closely with researchers and product engineers to turn new algorithms into reliable, production-ready systems.
Requirements
•
5+ years building or operating ML infrastructure at scale, ideally supporting large language or multimodal models.
•
Deep understanding of GPU architecture, distributed training frameworks (PyTorch, DeepSpeed, Megatron, Ray), and parallelism strategies.
•
Hands-on experience running inference stacks (vLLM / SGLang, TGI, Triton) and optimizing them via low-level profiling.
•
Strong software engineering fundamentals in Python and one of C++/Rust/Go, with clean, reliable code shipped to production.
•
Working knowledge of modern data pipelines, feature stores, and vector databases used in production AI systems.
•
Comfort automating infrastructure with Kubernetes, Terraform/Pulumi, and observability stacks (Prometheus, Grafana, OpenTelemetry).
Bonus Points
•
Experience deploying open-source LLMs (Llama 3, Qwen, DeepSeek) or training custom foundation models.
•
Contributions to ML systems tooling (compilers, kernels, inference runtimes) or open-source infrastructure projects.
•
Background in reinforcement learning, evaluation harnesses, or alignment tooling that hardens production AI systems.