Support EST Hours, at least through 2pm EST (MUST)
We are seeking a Big Data Lead with hands-on Python ,Spark experience and proven abilities to lead software development on Big Data product in an Agile methodology. We are seeking a well-rounded senior data engineer to lead a cloud based Big Data product team using a variety of technologies. The ideal candidate will possess strong technical, analytical, and interpersonal skills. In addition, the candidate will lead data engineers, data scientist on the team to achieve architecture and design objectives as agreed with stakeholders.
Position Description
- Lead engineers on the team to meet product deliverables.
- Work with data engineers, data scientist, cloud architect, Devops engineer on the team to meet product deliverables.
- Coach other developers on the team to develop scalable implementation.
- Work independently and collaboratively on a multi-disciplined project team in an Agile development environment.
- Contribute detailed design and architectural discussions as well as customer requirements sessions to support the implementation of code and procedures for the Big Data product.
- Work with product management team to understand the roadmap commitments and communicate design and implementation milestones effectively.
- Be familiar with one or more ( SAS, SPSS, R, Julia)
- Familiarity with cloud constructs and concepts.
- Ability to identify and solve for code/design optimization.
- Learn and integrate with a variety of systems, APIs, and platforms.
- Establish full-proof QA process for data validations and overall quality control on the product.
- Interact with a multi-disciplined team to clarify, analyze, and assess requirements.
- Be actively involved in the design, development, and testing activities in big data product.
Required Skills and Experience
- Minimum 4 years of proven experience working on generation of Big datasets using different source datasets.
- Expert in ETL implementation.
- Hands-on experience with Spark.
- Hands-on experience Python and Pyspark, Jupyter Notebooks
- Hands-on experience using Relational Databases, such as Oracle, SQL Server, MySQL,Postgres or similar.
- Familiarity with one or more (SAS , R,SPSS to Python).
- Familiarity with Databricks. Azure Databricks is a plus.
- Proven technical leadership on prior Big Data projects.
- Hands-on experience with a code versioning tool such as GitHub,Bitbucket, etc.
- Hands-on experience building pipelines in GitHub (or Azure Devops, Jenkins, etc.)
- Strong written and verbal communication skills.
- Self-motivated and ability to work well in a team.
Any mix of the following skills is valuable:
- Containers and their environments (Docker, Podman, Docker-Compose, Kubernetes, Minikube, Kind, etc.)
- Experience with Azure Cloud Services and Azure Data Factory.
Education
Bachelor of Science degree from an accredited university