Data Engineering
Build the pipelines and platforms that move, clean and serve data for analytics and AI.
Your progress
0/40
Tick items in Skills, Roadmap and Start today. Saved in this browser.
What is Data Engineering?
Data engineers build the 'plumbing' for data: pipelines that collect data from apps, databases and APIs, clean and transform it, and load it into warehouses or lakehouses where analysts and AI systems can use it. They care about reliability (pipelines that don't break at 3 a.m.), data quality, cost and speed. Every AI project depends on good data, which is why data engineering demand has grown alongside AI.
Why it's trending now
- AI and LLM projects need clean, well-governed data — retrieval systems, feature stores and training sets all rely on data pipelines.
- The WEF Future of Jobs Report 2025 ranks big data specialists as the fastest-growing job in percentage terms.
- Open table formats (Apache Iceberg, Delta Lake) and lakehouse platforms (Databricks, Snowflake) are replacing older Hadoop stacks, creating demand for engineers who know the modern tools.
- BLS projects database architects — the closest US category to senior data engineers — to grow faster than average (9%, 2025–35).
Job roles
Common job titles you'll see in listings, and what each one actually does.
Data Engineer
Builds batch and streaming pipelines that ingest, transform and load data into warehouses or lakehouses.
Analytics Engineer
Models warehouse data into clean, tested tables with dbt so analysts can trust and reuse them.
Big Data / Platform Engineer
Runs Spark, Kafka and lakehouse infrastructure and makes it reliable and cost-efficient.
Streaming Data Engineer
Builds real-time pipelines with Kafka, Flink or Spark Structured Streaming for fraud, IoT or live dashboards.
Data Architect
Designs the overall data platform: storage, modelling standards, governance and security.
AI Data Engineer
Prepares data for ML and LLM systems: feature pipelines, embeddings, vector stores and training datasets.
Skills from beginner to advanced
Tick skills as you learn them — your progress is saved in this browser.
1Beginner
Start here2Intermediate
Job-ready core3Advanced
Stand outMost-requested languages & tools
What employers mention most often in current job listings for this field.
Languages
Tools & platforms
SQL + Python + one cloud platform + Airflow/dbt covers most junior and mid-level listings. Spark and Kafka appear in larger-scale roles.
30/60/90-day roadmap
About 1–2 hours a day. Tick goals as you complete them.
Days 1–30
Databases and Python
Milestone: A Python script that ingests API data into a database on a schedule (cron).
Days 31–60
Pipelines and the warehouse
Milestone: An end-to-end ELT pipeline with orchestration, dbt models and data tests.
Days 61–90
Scale and streaming
Milestone: A capstone project with batch + streaming pipelines and a dashboard on top.
Free learning resources
All free to start. Some courses offer paid certificates — auditing is free.
- Data Engineering ZoomcampDataTalks.Club · Course · github.comFree
- dbt Learn (Fundamentals)dbt Labs · Course · learn.getdbt.comFree
- Apache Airflow documentation & tutorialsApache · Docs · airflow.apache.org
- Spark: Quick StartApache Spark · Docs · spark.apache.org
- PostgreSQL TutorialPostgreSQL Tutorial · Guide · postgresqltutorial.comFree
- Kafka 101Confluent Developer · Course · developer.confluent.ioFree
- BigQuery sandboxGoogle Cloud · Practice · cloud.google.comFree, no credit card
- Data Engineering Wikidataengineering.wiki · Guide · dataengineering.wikiCommunity
- roadmap.sh — Data Engineer roadmaproadmap.sh · Guide · roadmap.sh
- Databricks Academy free learningDatabricks · Course · databricks.comFree self-paced options
Portfolio project ideas
Build these in order. Put each one on GitHub with a clear README and a short demo.
Weather or currency API ingester
easyPull daily exchange rates (e.g. BDT/USD) or weather data from a public API into Postgres and chart the trend.
Star-schema sales warehouse
easyDesign a fact/dimension model for a retail dataset and answer business questions with SQL.
Airflow + dbt ELT pipeline
mediumOrchestrate extraction from an API, load into BigQuery, transform with tested dbt models and publish a dashboard.
Real-time ride or delivery events stream
mediumSimulate ride-sharing events, stream them through Kafka, aggregate in real time and show a live dashboard.
Lakehouse platform with data quality and lineage
hardBuild a Terraform-provisioned lakehouse with Iceberg tables, Spark jobs, data-quality checks, lineage tracking and cost reporting.
10 interview questions
Try answering out loud first, then open the short model answer.
1ETL vs ELT?
ETL transforms data before loading into the warehouse; ELT loads raw data first and transforms inside the warehouse. ELT is common today because cloud warehouses are powerful and keep raw data for reprocessing.
2What is a star schema?
A data model with a central fact table (events/measurements) linked to dimension tables (who, what, where, when), designed for fast, simple analytical queries.
3What makes a pipeline idempotent and why does it matter?
Running it multiple times produces the same result (e.g. using upserts or overwriting partitions). It makes retries and backfills safe.
4Batch vs streaming processing?
Batch processes data in chunks on a schedule (simpler, cheaper). Streaming processes events continuously with low latency (fraud detection, live metrics) but is more complex.
5Why use Parquet instead of CSV?
Parquet is columnar, compressed and typed, so analytical queries read only needed columns and run much faster and cheaper.
6What is partitioning?
Splitting a table by a key such as date so queries scan only relevant partitions, reducing cost and time.
7How do you ensure data quality?
Add tests for nulls, uniqueness, ranges and referential integrity (dbt tests, Great Expectations), monitor freshness and volume, and alert on failures.
8What is a slowly changing dimension (SCD Type 2)?
A way to keep history of dimension changes by adding a new row with valid-from/valid-to dates instead of overwriting the old value.
9What does Airflow do?
It schedules and orchestrates workflows defined as DAGs of tasks, handling dependencies, retries, backfills and monitoring.
10How would you handle late-arriving data in streaming?
Use event-time processing with watermarks that define how long to wait for late events, and design downstream tables to accept corrections.
Resume bullet templates
Replace the [brackets] with your real numbers. Quantified results stand out to recruiters.
Built [N] production pipelines in [Airflow/Dagster] ingesting [X] GB/day from [sources] with [99.X]% on-time delivery.
Migrated legacy reports to a [Snowflake/BigQuery] + dbt stack, reducing query cost by [X]% and report latency from [X] hours to [Y] minutes.
Implemented [N] automated data-quality tests that caught [N] incidents before reaching dashboards.
Designed a streaming pipeline with Kafka processing [N] events/sec for real-time [fraud/ops] monitoring.
Modelled [domain] data into a star schema used by [N] analysts, cutting average query time by [X]%.
Formula: action verb + what you built + tools + measurable result. Only claim numbers you can explain in an interview.
Salary & demand
Ranges vary by company, city, experience and negotiation. Always check the linked sources for the latest figures.
Bangladesh
Tk 16,417 – Tk 64,881
gross per month
Paylab has no separate data-engineer category for Bangladesh, so this uses its Backend Developer range (10th–90th percentile, gross monthly). Data engineers with Spark/cloud experience at offshore centres and telecoms often earn more.
Demand
High at telecoms, banks, MFS and offshore software firms building data platforms for foreign clients. Postings often ask for SQL, Python and Spark or a cloud warehouse.
Remote (from Bangladesh)
$30,000 – $80,000
per year
Estimate. Arc reports $41,569 average for Bangladesh-based remote back-end developers and $69,548 for seniors; cloud data platform skills command a premium.
Demand
High. Many remote contracts are pipeline builds or warehouse migrations — well suited to freelancers with a strong portfolio.
Global benchmark
$139,500
US median, Database Architects (BLS, 2025)
BLS has no 'data engineer' title; database architects are the closest match for senior data engineers. Database administrators earn a lower $104,620 median with flat growth. Projected growth: +9% (2025–35, faster than average).
Demand
Very high. Data engineering underpins every AI initiative and is consistently among the most-posted data roles.
How to read these numbers: Bangladesh ranges are Paylab's 10th–90th percentile gross monthly salaries (top product companies often pay more). Remote ranges are CareerPulse estimates informed by Arc's Bangladesh remote-developer data. Global figures are US medians from the Bureau of Labor Statistics (May 2025 data, 2025–35 projections) unless stated otherwise.
Start today checklist
Small, concrete actions you can finish today. Momentum beats perfect plans.