From code to tasks: DAG, DAG Scheduler, Task Scheduler, jobs, stages, tasks and execution plans

Topics in PySpark

1. 1Why Learn Spark?
2. 2Data Filters
3. 3Realtime ETL Project
4. 4With Spark & Without Spark
5. 5Spark Architecture
6. 6The problems of processing data on many machines
7. 7Inside one EC2 machine
8. 8Why 128 MB partitions in Spark?
9. 9Number of EC2 Machines Calculator
10. 10Predicate pushdown, column pruning, partition pruning: read less from S3
11. 11Narrow vs Wide transformations in Spark
12. 12Transformations vs Actions in Spark
13. 13Broadcast join vs Sort-merge join
14. 14Handling data skew in Spark
15. 15How many output files does Spark write
16. 16From code to tasks: DAG, DAG Scheduler, Task Scheduler, jobs, stages, tasks and execution plans
17. 17PySpark Installation Steps In Local Machine
18. 18Dataframe Creation
19. 19Data Selection
20. 20Data Mathematical Operations
21. 21Data Transformations
22. 22Dataframe Joins