Big Data Software Engineer
Applies intermediate level of subject matter knowledge to solve a variety of common business issues. Works on problems of moderately complex scope. Acts as an informed team member providing analysis of information and limited project direction input. Exercises independent judgment within defined practices and procedures to determine appropriate action. Follows established guidelines and interprets policies. Evaluates unique circumstances and makes recommendations.
- Designs enhancements, updates, and programming changes for portions and subsystems of data pipelines, repositories or models for structured/unstructured data.
- Analyses design and determines coding, programming, and integration activities required based on specific objectives and established project guidelines.
- Executes and writes portions of testing plans, protocols, and documentation for assigned portion of application; identifies and debugs issues with code and suggests changes or improvements.
- Participates as a member of a project team of other data science professionals to develop reliable, cost effective and high-quality solutions for assigned data system, model, or component.
- Collaborates and communicates with project team regarding project progress and issue resolution.
Knowledge & Skills
- Using data engineering tools, languages, frameworks to cleanse, mine and explore data.
- Understanding of big data systems along with complex, distributed and massively parallel systems.
- Ability to apply analytical and problem-solving skills.
- Ability to understand complex data structures.
- Understanding of database technologies, management systems, and cloud based systems/services.
- Database architecture testing methodology, including execution of test plans, debugging, and testing scripts and tools.
- Strong written and verbal communication skills; mastery in English and local language.
- Ability to effectively communicate design proposals and negotiate options.
Scope & Impact
- Collaborates with peers, junior engineers, data scientists and project team.
- Typically partners with high-level Individual Contributors and Managers.
- Supports projects requiring data engineering solutions expertise.
Education & Experience
- Bachelor's or Master's degree in Data Engineering, Computer Science, Data Science, Information Systems, Engineering, or equivalent.
- 2-4 years’ Data Engineering experience.
- Experience with AWS cloud services: EC2, EMR, RDS, Redshift, etc.
- Experience with big data tools: Databricks, Spark, Hadoop, Kafka, etc.
- Experience with object-oriented/object function scripting languages: Python, PySpark, Java, Scala, etc.
- Experience with relational databases: SQL, etc.
- Experience with data pipeline and workflow management tools: Airflow, etc.