Basic 5+ years of experience in data engineering or related roles Experience with Python for data engineering and automation Experience with distributed computing frameworks like Apache Spark or Ray Experience integrating LLMs (e.g., OpenAl, Azure OpenAl, Anthropic, or similar) into applications via APIs, including prompt design, response parsing, and error handling Experience with distributed systems concepts including data partitioning and sharding strategies, fault tolerance and replication, consistency models and distributed consensus, load balancing and resource management Experience with distributed file systems (Azure Data Lake, S3, HDFS) Experience implementing REST APIs using Python frameworks such as FastAPI, Flask, or similar Experience with containerization and orchestration (Docker, Kubernetes) Experience with SQL and database optimization techniques Experience with data structures, algorithms, and software design patterns Experience with version control systems (Git) and CI/CD pipelines Bachelor’s or Master’s degree in Computer Science, Engineering, or related field, or equivalent practical experience Preferred Experience working with unstructured data (text, images, video, audio) and associated processing techniques Experience with multiple cloud platforms (e.g., AWS, Azure, GCP) Knowledge of data governance and security best practices Experience with machine learning pipelines and MLOps Experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex) Familiarity with responsible Al practices, including bias mitigation, content filtering, and token cost optimization for LLM-based applications Contributions to open-source projects Excellent problem-solving skills and ability to work with complex, ambiguous requirements Strong communication skills and ability to collaborate across teams
We are seeking an experienced Machine Learning Engineer to join our team and drive the design, development, and optimization of large-scale data processing systems. This role requires deep expertise in distributed computing, data pipeline architecture, modern big data technologies, and applied Large Language Model (LLM) integration. This position is an individual contributor role reporting to the Machine Learning Engineering Manager. Responsibility Architect and maintain high-performance, fault-tolerant distributed systems to ensure scalability and availability Design, build, and optimize scalable data pipelines and ETL processes using Python, Spark, and Ray to process complex, unstructured datasets Develop batch data processing workflows to support analytics and machine learning initiatives Integrate and operationalize Large Language Models (LLMs) such as OpenAl, Azure OpenAl, or similar platforms into production applications, including PII detection and redaction workflows Design and maintain prompt engineering strategies, fine-tuning pipelines, and evaluation frameworks for LLM-based solutions Optimize data processing jobs and system performance for cost-efficiency and reliability Collaborate with applied scientists, SMEs, and engineering teams to understand data requirements and deliver robust solutions Implement data quality frameworks and monitoring systems to ensure data integrity Troubleshoot and resolve complex data processing issues in production environments