Sr Data Engineer - Hadoop/Spark

  • The Walt Disney Company
  • Burbank, CA, United States
  • Feb 02, 2018
Full-time Data Analysis

Job Description

  • Build and optimize performance of Hadoop/Spark batch jobs.
  • Build and optimize performance of Spark, Kafka, Cassandra, ELK, and whatever else makes sense for realtime pipelines.
  • Design and architect high quality data-lake, data-warehouse, and data-marts data models.
  • Enable and implement Data Science workflows and advanced machine learning althorithms.
  • Build and optimize performance of ElasticSearch cluster and relevance
  • Build data pipelines orchestration.
  • Become and stay an expert in current and emerging technologies and tools
  • Contribute to Open Source solutions and communities wherever you can
  • Collaborate with other software engineers and cross­functional teams
  • Evangelize technologies, solutions, and best practices
  • Contribute new ideas to a larger community of high­caliber professionals
  • Balance resources, requirements, and complexity
  • Very passionate about coding (If you have a Github profile, that’s awesome! Would love to check it out! Otherwise we will ask you to code before or during the interview).
  • String understanding of distributed systems and distributed computation.
  • Strong working knowledge in at least 2 of: Scala, Java, Python, or Go-Lang
  • Demonstrated working knowledge with data Apache Hadoop / Spark ecosystem like Spark, Hive, Presto, Oozie, Pig, Hue, Zeppelin
  • Demonstrated working knowledge of data modeling
  • Unit, Integration, and Load testing
  • Developing REST APIs.
  • Git
  • Ant, Maven, SBT, and/or Gradle
  • Unix/Linux
  • Docker containers building and deployment.
  • Excellent communication and collaboration skills
  • GraphQL knowledge
  • Kubernetes knowledge
  • Apache Spark MLlib
  • Apache Spark GraphX
  • R
  • Amazon AWS or other cloud Services
  • Jenkins
  • BS in Computer Science or related field with 5+ years of experience
This position is a legal entity of The Walt Disney Studios, an equal opportunity employer. The Senior Data Engineer works in a small team of multi-disciplined engineers on creating the next generation video consumption platforms. We expect you to be up to date on the happenings in the data community, passionate about what you do and connected to the open source community. You will participate in overall system design, come up with web­tailored solutions that emphasize re­use and good design patternsThis position is a legal entity of The Walt Disney Studios, an equal opportunity employer. 477011

Job ID

477011BR