Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

Python

Idiomatic Python for data engineering: typing, generators, packaging and testing.

4 notes · all topicsData engineering

Start here · beginner3

  1. 1Reading and Writing JSON, CSV and Parquet in PythonWhich file format to use where, how to read each one without surprises, and why Parquet is the default for analytics.
  2. 2Type Hints and Dataclasses for Cleaner Pipeline CodeUse type hints and dataclasses to make records explicit, catch bugs before runtime, and keep pipeline functions small and testable.
  3. 3Python Generators for Memory-Safe Data PipelinesProcess files far bigger than RAM by streaming records through small generator stages instead of loading lists.

Next · intermediate1

  1. 4Testing Data Pipelines with pytestWhat to test in a pipeline, how to build tiny hand-checked fixtures, and the tests that catch real production bugs: replays, duplicates and bad rows.