Data Engineer
Read straight from Beyondsoft Consulting's careers page, verified 1d ago.Seekless can tailor your résumé to it.
Seekless can write your résumé for this posting first.
Before you apply
LocationIndia, UNAVAILABLE, IN
Work moderemote
SenioritySenior
Company size10,000+ people
PostedOct 18, 2024 · 2y ago
Verified live1d ago
May no longer be actively hiring
Posted by the company on Oct 18, 202496% of this company's open roles have been listed for 90+ days
At a glanceSummarised by Seekless from the posting.
Strong hands-on experience with Azure Databricks
Strong programming skills in Python
Strong proficiency in SQL, including complex queries, data transformation, and query optimization
Hands-on experience designing and developing production-grade ETL/ELT data pipelines
Experience working with large-scale structured and semi-structured datasets
Solid understanding of data modeling, data warehousing, and modern cloud-based data architectures
Experience troubleshooting and optimizing data pipelines and data processing workloads
Good understanding of Large Language Models (LLMs) and Generative AI concepts
Familiarity with integrating LLM APIs or AI services into applications or data processing workflows
Understanding of common LLM concepts such as prompt engineering, embeddings, vector search, and Retrieval-Augmented Generation (RAG)
Strong problem-solving and analytical skills
Good communication and collaboration skills
Experience with Azure OpenAI Service or other enterprise LLM platforms
Experience developing RAG-based applications or solutions
Familiarity with vector databases or vector search technologies
Experience preparing and managing enterprise data for LLM-based applications
Experience with Azure Data Factory for data integration and orchestration
Experience with Azure Data Lake Storage (ADLS)
Experience with Azure SQL, Azure Synapse Analytics, or other cloud-based data platforms
Experience with REST APIs and integrating external services into data pipelines
Experience with CI/CD, Git, and Azure DevOps
Knowledge of data security, governance, access control, and data quality best practices
Understanding of responsible AI, data privacy, and security considerations when working with LLMs
Relevant Azure, Databricks, or AI certification is a plus
All candidates must complete a video screen with our corporate recruiting team (for remote positions)
Skills
Python
SQL
Azure Databricks
ETL
Azure OpenAI
REST APIs
Git
Azure DevOps
Azure Data Factory
Azure Data Lake Storage
Azure SQL
Azure Synapse Analytics
Overview
WHO WE ARE:
Beyondsoft is a leading mid-sized business IT and consulting company that combines modern technologies and proven methodologies to tailor solutions that move your business forward.Our global head office is based in Singapore, and our team is made up of a diversely talented team of experts who thrive on innovation and pushing the bounds of technology to solve our customers’ most pressing challenges.When it comes time to deliver, we set our sights on that sweet spot where brilliance, emerging technologies, best practices, and accountability converge. We have a global presence spanning four continents (North America, South America, Europe, and Asia). Our global network of talent and customer-centric engagement model enables us to provide top-quality services on an unprecedented scale.
WHAT WE’RE ABOUT:
We believe that collaboration, transparency, and accountability are the values that guide our business, our delivery, and our brand. Everyone has something to bring to the table, and we believe in working together with our peers and clients to leverage the best of one another in everything we do. When we proactively collaborate, business decisions become easier, innovation is greater, and outcomes are better.
Our ability to achieve our mission and live out our values depends upon a diverse, equitable, and inclusive culture. So, we strive to foster a workplace where people have the respect, support, and voice they deserve, where innovative ideas flourish, and where people can unleash their brilliance. For more information regarding DEI at Beyondsoft, please go to https://www.beyondsoft.com/diversity/.
ADDITIONAL REQUIREMENT FOR REMOTE POSITIONS:
For remote positions, all candidates must complete a video screen with our corporate recruiting team.
Responsibilities
POSITION SUMMARY:
About the role
We are seeking an experienced Data Engineer In India with strong expertise in Python, SQL, and Azure Databricks to design, build, and maintain scalable cloud-based data solutions.
The ideal candidate will have hands-on experience developing production-grade data pipelines, transforming and managing large datasets, and building reliable data solutions on Azure. In addition, the candidate should have a good understanding of Large Language Models (LLMs) and Generative AI, with experience or strong interest in integrating LLM capabilities into data and AI solutions.
You will work closely with engineers, analysts, and business stakeholders to build reliable data platforms and enable data-driven and AI-powered applications.
WHAT YOU WILL BE DOING:
•
Design, develop, and maintain scalable data pipelines using Azure Databricks, Python, and SQL.
•
Develop ETL/ELT workflows for ingesting, transforming, validating, and publishing data from multiple data sources.
•
Build and maintain Azure Databricks notebooks, workflows, and jobs for production data processing.
•
Develop reusable Python components for data ingestion, transformation, automation, and integration.
•
Write and optimize complex SQL queries for data transformation, analytics, and reporting.
•
Work with large-scale structured and semi-structured datasets.
•
Design and maintain reliable data models and data processing workflows.
•
Monitor data pipelines and troubleshoot data quality, reliability, and performance issues.
•
Implement appropriate logging, monitoring, error handling, and data quality controls.
•
Explore and integrate LLM and Generative AI capabilities into data engineering solutions and applications.
•
Develop data preparation and processing workflows to support LLM-based applications.
•
Work with LLM APIs and services to enable use cases such as information extraction, summarization, classification, and intelligent data processing.
•
Collaborate with engineering and business teams to identify opportunities where LLM technologies can improve data processing and automation.
•
Participate in technical design, code reviews, testing, deployment, and production support.
•
Follow engineering best practices for source control, CI/CD, security, documentation, and maintainability.
Qualifications
MINIMUM QUALIFICATIONS:
•
Strong hands-on experience with Azure Databricks.
•
Strong programming skills in Python.
•
Strong proficiency in SQL, including complex queries, data transformation, and query optimization.
•
Hands-on experience designing and developing production-grade ETL/ELT data pipelines.
•
Experience working with large-scale structured and semi-structured datasets.
•
Solid understanding of data modeling, data warehousing, and modern cloud-based data architectures.
•
Experience troubleshooting and optimizing data pipelines and data processing workloads.
•
Good understanding of Large Language Models (LLMs) and Generative AI concepts.
•
Familiarity with integrating LLM APIs or AI services into applications or data processing workflows.
•
Understanding of common LLM concepts such as prompt engineering, embeddings, vector search, and Retrieval-Augmented Generation (RAG).
•
Strong problem-solving and analytical skills.
•
Good communication and collaboration skills.
Required Skills: Python, SQL, Azure Databricks, ETL/ELT, data pipeline development, data processing and transformation, LLM/generative AI fundamentals, LLM API integration, prompt engineering, RAG fundamentals.
PREFERRED QUALIFICATIONS:
•
Experience with Azure OpenAI Service or other enterprise LLM platforms.
•
Experience developing RAG-based applications or solutions.
•
Familiarity with vector databases or vector search technologies.
•
Experience preparing and managing enterprise data for LLM-based applications.
•
Experience with Azure Data Factory for data integration and orchestration.
•
Experience with Azure Data Lake Storage (ADLS).
•
Experience with Azure SQL, Azure Synapse Analytics, or other cloud-based data platforms.
•
Experience with REST APIs and integrating external services into data pipelines.
•
Experience with CI/CD, Git, and Azure DevOps.
•
Knowledge of data security, governance, access control, and data quality best practices.
•
Understanding of responsible AI, data privacy, and security considerations when working with LLMs.
•
Relevant Azure, Databricks, or AI certification is a plus.
Preferred Skills: Azure OpenAI, embeddings/vector search, RAG application development, Azure Data Factory, Azure Data Lake Storage, Azure SQL/Azure Synapse Analytics, REST API integration, Git/Azure DevOps/CI/CD, data governance and security.
Legal
Beyondsoft provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type with regards to race, color, religion, age, sex, national origin, disability status, genetics, veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws. This policy applies to all terms and conditions of employment, including recruiting, hiring, and the full employee lifecycle up through and including termination.
More roles at Beyondsoft Consulting