DataTypes List Tuple Dict Set String Mutable Immutable Hashing Memory

pandas, ETL pipeline, in-memory processing, data transformation, data validation, data cleaning, Landing Zone, Curated Zone, Parquet format, columnar storage, partition pruning, pandas vs PySpark, single machine, distributed processing, vectorisation, np.where, apply lambda

PySpark, RDD vs DataFrame, Transformations & Actions, Joins, GroupBy & Aggregation, Window Functions, UDFs, Broadcast Join, Catalyst Optimizer, Adaptive Query Execution, Data Skew, Partitioning, Caching & Checkpointing, Shuffle Optimization, Parquet vs CSV, Performance Tuning

ncremental Loading, Job Bookmarking, Cluster Sizing, Data Skew, Adaptive Query Execution (AQE)

SQL, Joins, Aggregation, Window Functions, Subqueries