At a glanceSummarised by Seekless from the posting.
Must have7
5+ years of software development using Python
Solid understanding of distributed systems and algorithms
Experience building & working with complex data pipelines or data systems
Strong SQL skills, ideally with Snowflake or similar analytical databases
Strong hands-on experience with Apache Spark – Databricks (PySpark or Scala) in production
Experience with AI/ML-assisted systems (embeddings, inference, re-ranking)
Experience with, or strong interest in, fuzzy and semantic matching techniques including Levenshtein, token-based similarity, BM25, vector embeddings, cosine similarity, approximate nearest-neighbor
Nice to have9
Experience with data orchestration tools such as Airflow (or equivalents)
Experience building entity resolution, deduplication, or record linkage systems
Familiarity with search or retrieval systems (e.g., Elasticsearch, OpenSearch, vector databases)
Experience operating data pipelines at very large scale (100M+ records)
Background in data quality frameworks, validation automation, or QA at scale
Experience with Kubernetes or Containerized Functions (e.g., Azure Container Apps, AWS Fargate)
Experience with Serverless Functions
Azure or other Cloud Experience
GitHub Actions
Skills
Python
SQL
Apache Spark
Databricks
PySpark
Scala
Airflow
DBT
Snowflake
Kubernetes
Azure Container Apps
AWS Fargate
GitHub Actions
Elasticsearch
OpenSearch
vector databases
Azure
Serverless Functions
About the company
GT was founded in 2019 by a former Apple, Nest, and Google executive. GT’s mission is to connect the world’s best talent with product careers offered by high-growth companies in the UK, USA, Canada, Germany, and the Netherlands.
About the role
On behalf of the client, GT is looking for a Senior ML Engineer / Applied ML Engineer who is interested in solving complex matching and search problems at scale, applying ML and algorithms to hundreds of millions of real-world data records.
About the Client
Our client is a leading global management consultancy known for tackling some of the world’s most complex business challenges. With a focus on strategy, transformation, and performance improvement, the firm partners with major organizations across industries to drive lasting impact.
Recognized consistently as a top workplace, it combines deep industry expertise with a collaborative, innovative culture. Its centralized European hub plays a key role in supporting operations across the EMEA region, ensuring excellence and efficiency at scale.
About the Project & Role
We are looking for a Senior ML Engineer / Applied ML Engineer to build and evolve the client’s internal entity resolution system — a core part of the data platform that uses machine learning, LLMs, and search and matching algorithms to turn complex, noisy data into trusted, unified entities.
The system operates at significant scale, processing hundreds of millions of records, and the role will focus on developing and improving the ML models, matching logic, algorithms, and service capabilities behind it.
This is a hands-on engineering role requiring strong Python and SQL, practical ML engineering experience, experience with large-scale data pipelines, and strong algorithmic and problem-solving skills.
This is not a traditional Data Engineering role. The main focus is on building intelligent, production-ready ML systems and solving complex matching and algorithmic problems, with Data Engineering technologies such as Spark/PySpark used to support scalability and productionization.
Responsibilities:
Entity Matching & Distributed Algorithms
•
Design and improve entity matching, clustering, and deduplication algorithms at scale
•
Implement distributed matching approaches such as:
•
blocking strategies
•
multi-pass matching pipelines
•
nearest-neighbor and similarity-based methods
•
Apply and combine rule-based, statistical, and ML-assisted techniques (including embeddings where relevant)
•
Optimize candidate generation and scoring to balance accuracy, recall, performance, and cost
•
Translate algorithmic ideas into scalable implementations using Spark and SQL transformations
•
Continuously experiment with different approaches and iterate based on performance metrics and results
Large-Scale Analytic Engineering
•
Design, build, and operate a large-scale analytic system processing hundreds of millions to billions of records.
•
Implement and optimize Apache Spark pipelines for entity matching, deduplication, and clustering.
•
Build and maintain complex workflows using Airflow and DBT.
•
Ensure pipelines are fault-tolerant, observable, and cost-efficient in distributed environments.
SQL & Data Modelling
•
Develop and maintain analytical SQL models using Snowflake or a similar cloud data warehouse.
•
Optimize large joins, aggregations, and window functions over very large datasets.
•
Design data models that support both matching pipelines and downstream consumers.
Data Quality, Validation & Iteration
•
Build validation logic and metrics to measure match rate, precision, recall, and accuracy.
•
Support continuous improvements to the matching engine through iterative releases.
•
Debug and resolve data quality issues across heterogeneous and imperfect data sources.
Essential knowledge, skills & experience
Core Requirements:
•
5+ years of software development using Python, ideally in a team lead capacity.
•
Solid understanding of distributed systems and algorithms (partitioning, shuffles, joins, scalability trade-offs).
•
Experience building & working with complex data pipelines or data systems
•
Strong SQL skills, ideally with Snowflake or similar analytical databases.
•
Strong hands-on experience with Apache Spark – Databricks (PySpark or Scala) in production.
•
Experience with AI/ML-assisted systems (embeddings, inference, re-ranking).
Matching, Search & Similarity (Required or Willingness to Learn)
•
Experience with, or strong interest in, fuzzy and semantic matching techniques, such as:
•
Levenshtein / edit distance
•
Token-based similarity
•
BM25 or other lexical ranking methods
•
Vector embeddings and cosine similarity
•
Approximate nearest-neighbor or vector search concepts
•
Strong willingness to learn and apply advanced semantic matching techniques if not already experienced.
Nice-to-Have:
•
Experience with data orchestration tools such as Airflow (or equivalents).
•
Experience building entity resolution, deduplication, or record linkage systems.
•
Familiarity with search or retrieval systems (e.g., Elasticsearch, OpenSearch, vector databases).
•
Experience operating data pipelines at very large scale (100M+ records).
•
Background in data quality frameworks, validation automation, or QA at scale.
•
Experience with Kubernetes or Containerized Functions (e.g., Azure Container Apps, AWS Fargate)
•
Experience with Serverless Functions
•
Azure or other Cloud Experience
•
GitHub Actions
What Success Looks Like
•
Matching pipelines scale reliably across hundreds of millions to billions of records.
•
Measurable improvements in matching accuracy, recall, and runtime efficiency.
•
Faster iteration on matching algorithms without compromising system stability.
•
A robust, high-quality entity foundation that enables advanced analytics and downstream applications.
•
Understanding of algorithmic and resource constraints to balance system cost and performance
Interview Process
1.
Interview with GT Recruiter
2.
Cultural Fit Interview
3.
Technical Interview
4.
Final Interview (optional)
5.
Offer
Why join our client?
•
Join a fast-growing, high-impact team in a top-tier company
•
Contribute to an ambitious effort to create the highest quality, most comprehensive business directory in the world.
•
Be part of a dinamic-style group within the company that’s redefining how they deliver consulting through productization and data innovation.
•
Work with cutting-edge data tools, including AI/ML enrichment, semantic matching, and modern cloud-based infrastructure.