Member of Technical Staff (Data Scientist/Engineer, Online Metrics)
Location
New York City
Work mode
hybrid
Type
full-time
Department
AI Research & Systems
Last seen
1h ago
Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and specialized data sources. The Answer Quality team ensures that our prompts, tools, search, and specialized datasets, combined with both frontier and in-house models, create the best possible experience for our users. As a Data Scientist/Engineer on this team, you will derive online signals from user interactions to bridge the gap between changes in answer quality and observed user behavior.
Responsibilities
•
Discover and validate online signals from user interactions that serve as reliable proxies for true answer quality
•
Design and implement novel online metrics to be tracked both in A/B testing and on product health dashboards, ensuring alignment with ground-truth evaluations
•
Analyze experimental results to validate these metrics, ensuring they accurately predict user satisfaction and drive product decisions
•
Build and maintain the data pipelines that calculate these metrics at scale, delivering actionable quality signals to Search, Product, and model training teams
•
Communicate findings and bring clarity through close collaboration with Product and Search teams
•
Operate in a small, high-impact team where your work directly shapes how Perplexity measures and improves Answer Quality
Qualifications
•
MS in a technical field or equivalent experience
•
4+ years of experience working as a Data Scientist, Analytics Engineer, or related role
•
Experience working on search, recommendation, or LLM-based products, with an emphasis on designing online metrics and analyzing A/B experiments
•
Strong proficiency in Python and SQL (expected to write production-grade code)
•
Deep knowledge of statistical analysis
•
Experience with Business Intelligence (BI) tools for visualization and reporting
•
Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster
Preferred Qualifications
•
Proficiency with Apache Spark and Databricks
•
Experience with the development or validation of LLM-as-a-judge systems
•
Prior work supporting customer-facing products at scale