2021 — Rockstar Data Engineer Roadmap

This article presents a roadmap for those who want to become Data Engineers in 2021. It also serves as a reference to learn and improve the understanding of the different skills required for this position.

For each skill category, I cite the skill points that are recommended with links to video/text courses and some reference books at the end. I have no association with the author, and this is not a referral article. The goal is to provide an easy way for you to navigate and improve your skills :)!

Photo by Scott Webb on Unsplash

To become a skilled data engineer, you need to know computer science, dev, networking, and databases fundamentals.

CS Fundamentals

• Data Structures and Algorithms.
• Basic Computer Architecture.
• APIs & REST.
• Linux (Terminal, Shell Scripting, Cronjobs, Processes).

Reference Books
The Algorithm Design Manual
Computer Systems: A Programmer’s Perspective
How Linux Works: What Every Superuser Should Know

Dev Fundamentals

• Git.
• Python, Scala, Java, Go.
• Testing: Unit tests, Integration Tests, Functional Testing.
• VIM / VSCode.
• SCRUM / Agile Methodologies.
• SOLID / Refactoring.

Reference Books
Operating Systems: Three Easy Pieces
Refactoring: Improving the Design of Existing Code

Networking Fundamentals

• HTTP / HTTPs.
• TCP / IP / SSH / DNS.
• VPN, VPC.
• Firewalls.

Reference Books
Computer Networking: A Top-Down Approach

Database Fundamentals

• Relational Algebra / Normalisation.
• SQL.
• ACID / CAP.
• OLTP vs OLAP.
• Datawarehouses and Datamarts.
• Datalake / Cloud Data Platform.

Reference Books
Readings in Database Systems

Photo by Annie Spratt on Unsplash

After fundamentals, you need to grasp some technologies. Here is a non-exhaustive list of tools to store, process, and create data pipelines.

Learn Data Science in a Flash!? | Data Driven Investor

Data Storage

• Relational: MySql, PostgreSQL, MariaDB, Amazon Aurora.
• Document: MongoDB, Elasticsearch, Apache CouchDB.
• Wide column: Cassandra, Hbase, Google Bigtable.
• Graph: Neo4j, Amazon Neptune.
• Key-value: Redis, Memcached, Amazon DynamoDB.
• Messaging: RabbitMQ, Apache ActiveMQ.
• Data warehouses: Snowflake, Presto, Apache Hive, Amazon Redshift, Google BigQuery.
• Datalakes: S3, ADLS Gen2.

Reference Books
Designing Data-Intensive Applications

Data Processing:

• Cluster Computing: Hadoop, HDFS, MapReduce.
• Batch: Apache Pig, Apache Arrow, Apache Impala.
• Hybrid: Apache Spark, Apache Beam, Apache Flink.
• Streaming: Apache Kafka, Apache Storm, Apache Samza, Amazon Kinesis.
• Managed Solutions: Databricks, Amazon EMR, Google Dataproc, Azure HDInsight.

Workflow Scheduling / ETC

• Apache Airflow / Google Composer.
• Azure Data Factory.
• AWS Glue.
• Apache Oozie.

DevOps & Security

• Jenkins, Azure Devops, AWS CodePipeline, Github Actions.
• Docker, Kuberentes and Helm.
• Active Directory / Azure Active Directory.
• Encryption, Key Management.
• Data Governance, GDPR.

Photo by davisuko on Unsplash

It’s time to prove your skills. Here are some certifications that can help you get your dream job or just make you more confident.

Valuable Certifications

Databricks Certified Data Engineer Associate - Databricks

Databricks Certified Data Engineer Professional - Databricks

Databricks Certified Associate Developer for Apache Spark 3.0

Microsoft Certified: Azure Data Engineer Associate - Learn

Certification AWS Certified Big Data - Specialty

Professional Data Engineer Certification | Certifications

CCA Spark and Hadoop Developer Certification - Cloudera

Certified Kubernetes Application Developer (CKAD) | Cloud Native Computing Foundation

Conclusion

Keep this story bookmarked! It can help you learn efficiently and become a better data engineer. You are not born a guru, It is continuous trial and error that improves your skills.

Dunning–Kruger Effect — Wikimedia