11. Batch Processing
11.10 Terminology introduced here
Terminology introduced here
47- online vs offline systems
- batch processing job
- time travel
- human fault tolerance
- working set
- distributed filesystem (DFS)
- block device driver / page cache / VFS
- data node / DataNode / NameNode
- erasure coding (Reed–Solomon)
- FUSE / NFS
- object store / bucket / key
- prefix listing
- job orchestrator
- task executor / NodeManager / kubelet
- resource manager
- scheduler
- ApplicationMaster / operator
- cgroups
- gang scheduling
- starvation
- preemption
- spot / preemptible instances
- FIFO / dominant resource fairness (DRF) / bin-packing
- workflow / DAG
- workflow scheduler (Airflow, Dagster, Prefect)
- backpressure
- MapReduce
- mapper / reducer
- input format parser
- shuffle
- secondary sort
- sort-merge join
- dataflow engine
- operator fusion
- lineage
- checkpointing
- cost-based optimizer
- Apache Pig / Morel / Gremlin
- data lakehouse
- table format / catalog
- pre-aggregation vs ad hoc queries
- feature engineering / model training / batch inference
- bulk synchronous parallel (BSP) / Pregel model
- fine-tuning
- data mesh / data contract / data fabric
- DMZ
- bulk import / SST files