Technical notes
Concepts explained the way you need them on the job: what it is, why it matters, the code, and the trade-offs.
77 notes · 17 topics · each topic goes from beginner to advanced
Data engineering
From SQL to production pipelines
- Python01Idiomatic Python for data engineering: typing, generators, packaging and testing.
- Reading and Writing JSON, CSV and Parquet in Python
- Type Hints and Dataclasses for Cleaner Pipeline Code
- Python Generators for Memory-Safe Data Pipelines
- Testing Data Pipelines with pytest
- SQL02Window functions, joins, query plans and modelling patterns that hold up in production.
- CTEs vs Subqueries vs Temp Tables
- NULL in SQL
- Window Functions Every Data Engineer Uses
- SQL Logical Query Order
- Apache Spark03Execution model, memory, shuffles, partitioning and performance tuning.
- Lazy Evaluation
- Spark Partitions and File Sizes
- Spark Execution Model
- Adaptive Query Execution (AQE) in Spark
- PySpark04DataFrame API patterns, schemas, UDF trade-offs and testing Spark code.
- Always Use Explicit Schemas in PySpark
- PySpark DataFrame Basics in 10 Operations
- Broadcast Joins and Handling Data Skew
- Python UDFs vs Built-in Functions vs Pandas UDFs
- AWS05Glue, EMR, S3, Lambda, Step Functions and the data services around them.
- Designing an S3 Data Lake Layout
- Glue Data Catalog, Crawlers and Athena
- IAM for Data Pipelines
- AWS Glue JDBC Ingestion
- Snowflake06Loading, transactions, clustering, cost control and secure data sharing.
- Snowflake Virtual Warehouses and Cost Control
- Time Travel and Zero-Copy Cloning in Snowflake
- Loading Data into Snowflake with COPY INTO
- Micro-partitions, Pruning and Clustering Keys
- Databricks07Lakehouse, Delta Lake, Unity Catalog and workflows.
- Medallion Architecture
- Delta Lake
- Upserts with Delta Lake MERGE
- OPTIMIZE, Z-ORDER, Liquid Clustering and VACUUM
- Apache Airflow08DAG design, idempotent tasks, sensors and operating Airflow.
- Airflow Core Concepts
- Sensors, Reschedule Mode and Deferrable Operators
- Writing Idempotent Airflow DAGs
- Airflow Backfills and Catchup Without Surprises
- Data Warehousing09Dimensional modelling, facts, dimensions and slowly changing history.
- OLTP vs OLAP
- Data Quality Checks That Catch Real Incidents
- Kimball vs Inmon vs Data Vault
- Slowly Changing Dimensions Type 2, Explained Honestly
- ETL / ELT10Ingestion patterns, idempotency, replay and data quality.
- ETL vs ELT
- Incremental Loads and Watermarks
- Designing Idempotent, Replayable Pipelines
- Change Data Capture
- Data Modeling11Grains, keys, normalisation and modelling for analytics.
- Normalization in Plain English
- Star Schema Basics
- Surrogate Keys vs Natural Keys
- Three Kinds of Fact Tables
- Data Engineering System Design12Designing pipelines and platforms end to end, with trade-offs.
- Data Pipeline Observability
- System Design
- Batch vs Streaming
- Exactly-Once Is a Myth; Idempotency Is the Answer
AI engineering
From how LLMs work to agents in production
- Generative AI01How LLM applications are built, evaluated and shipped.
- How LLMs Generate Text, One Token at a Time
- Tokens, Context Windows and Sampling
- Prompting for Engineers
- Hallucinations and How to Ground an LLM
- RAG02Retrieval-augmented generation: chunking, embeddings, retrieval and evaluation.
- Embeddings and Vector Search, Explained for Engineers
- RAG Pipeline Fundamentals
- Chunking Strategies for Retrieval
- Hybrid Search and Reranking for Better RAG
- Agentic AI03Tool use, planning, memory and multi-agent systems.
- What Actually Makes an AI Agent
- Tool Calling Patterns for Reliable Agents
- Agent Memory
- Guardrails for AI Agents in Production
- LLMs04Model internals, fine-tuning, quantisation and inference.
- LLM Inference
- RAG vs Fine-Tuning vs Prompting
- Evaluating LLM Applications Before and After Launch
- LoRA and QLoRA
- AI Tools05Practical tooling for building with AI: frameworks, vector stores, observability.
- Using AI Coding Assistants Well (and Safely)
- LLM Frameworks
- Observability for LLM Applications
- Model Context Protocol (MCP) Explained
Beginner Intermediate AdvancedBrowse by tagMy reading