After learning PySpark, where can I run it? Platforms and resources

Facts as of October 2026. Cloud services rename and change often: check the official docs and pricing pages before you build or quote a number.

🗺️ Platform table

Notes: Azure HDInsight still exists but Microsoft has stopped investing in it and points customers to Fabric. Microsoft says it has no current plan to retire Synapse, but most new work goes to Fabric. Google renamed Dataproc + Serverless for Apache Spark to Managed Service for Apache Spark; job posts still say "Dataproc". Snowflake Snowpark has a PySpark-like DataFrame API but is not the Spark engine.

🎒 Resources checklist

🚀 Setup quick-start

# 1) check Java (need 17 or 21)
java -version

# 2) a virtual environment + PySpark
python3 -m venv .venv
source .venv/bin/activate          # Windows WSL2 / Linux / macOS
pip install pyspark

# 3) check it
python -c "import pyspark; print(pyspark.__version__)"

# Google Colab / Kaggle cell
!pip install pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").getOrCreate()
print(spark.version, spark.sparkContext.defaultParallelism)

# Jupyter + PySpark in one box (open the 127.0.0.1:8888 address it prints)
docker run -it --rm -p 8888:8888 -v "$PWD":/home/jovyan/work quay.io/jupyter/pyspark-notebook

# Official Spark image, PySpark shell (check tags on Docker Hub)
docker run -it --rm spark:python3 /opt/spark/bin/pyspark

# local test
spark-submit --master "local[*]" job.py

# on an EMR cluster (YARN): 3 worker machines of m5.xlarge, 2 executors each is too many,
# so: 1 executor x 3 cores per machine -> 3 executors
spark-submit --master yarn --deploy-mode cluster \
  --num-executors 3 --executor-cores 3 --executor-memory 9g \
  s3://my-bucket/jobs/job.py

🧪 AWS Glue and EMR key terms

Costs: Glue bills per DPU-hour, EMR bills per second for the EC2 machines plus an EMR fee, serverless products bill for what the job uses. We do not quote prices here: check the AWS, Azure and Google pricing pages for your region.

🧭 Which one to choose?

🎤 Interview Q&A

• Where have you run PySpark?

  Locally with local[*] for development and unit tests, then on AWS: Glue jobs for S3 ETL and EMR for heavier batch jobs submitted with spark-submit. I practised on Databricks Free Edition. Same code everywhere; only the master, paths and job settings changed.

• What is the difference between EMR and Glue?

  EMR gives you a cluster of EC2 machines you size and configure (instance types, machine counts, Spark settings, other tools like Hive or Presto); you pay while it runs. Glue is serverless Spark for ETL: you choose a worker type and number of workers (DPUs), AWS runs it, and it ties into the Glue Data Catalog. EMR = more control, Glue = less to manage.

• What is serverless Spark?

  You submit code and the platform creates, scales and removes the machines for you; you pay for what the job uses. Examples: AWS Glue, EMR Serverless, Athena for Spark, Google's serverless Spark, Databricks serverless compute.

• What do you need to run PySpark locally?

  A Java JDK (17 or 21 for Spark 4.x), Python 3.10+ (3.11+ safest), pip install pyspark, enough RAM (8–16 GB) and CPU cores. Then SparkSession.builder.master("local[*]"). On Windows, use WSL2 or Docker to avoid winutils/HADOOP_HOME problems.

🔬 Real local run (the script from step 4)

Ran once in the sandbox: Java 21.0.12.1, Python 3.13.16, PySpark 4.2.0, master local[*], 2 cores, CareWell file healthcare_dataset_1000.csv (1,000 rows), 41.0 s including start-up.

import subprocess, sys, os
os.environ["PYSPARK_PYTHON"] = sys.executable
j = subprocess.run(["java", "-version"], capture_output=True, text=True).stderr
print("java   :", [l for l in j.splitlines() if "version" in l][0])
print("python :", sys.version.split()[0])
import pyspark
from pyspark.sql import SparkSession, functions as F
print("pyspark:", pyspark.__version__)
spark = (SparkSession.builder.master("local[*]")   # laptop: use all cores
         .appName("where-to-run").getOrCreate())
sc = spark.sparkContext
print("master :", sc.master)
print("cores  :", sc.defaultParallelism)
df = spark.read.csv("healthcare_dataset_1000.csv",
                    header=True, inferSchema=True)
print("rows   :", df.count())
(df.groupBy("city").count()
   .orderBy(F.desc("count")).show(3))
spark.stop()

java   : openjdk version "21.0.12.1" 2026-08-18
python : 3.13.16
pyspark: 4.2.0
master : local[*]
cores  : 2
rows   : 1000
+---------+-----+
|     city|count|
+---------+-----+
|    Delhi|   78|
|Bangalore|   77|
|  Chennai|   77|
+---------+-----+
only showing top 3 rows

The steps in words

• 1. Where do I run it?

  Suresh has a PySpark script; it can run in five kinds of places, from his own laptop to big company clusters.

• 2. Backpack checklist

  Pack Java, Python, the pyspark package, enough RAM, CPU cores, some disk and an editor.

• 3. Laptop vs cluster

  On a laptop one machine does everything; on a cluster a driver EC2 machine plans, worker EC2 machines run tasks and data sits in S3.

• 4. Laptop: real run

  In the sandbox, Java 21 and PySpark 4.2.0 ran with local[*], used 2 cores and counted 1,000 CareWell rows.

• 5. Docker box

  A Docker image brings Java, Python, Spark and Jupyter ready-made, so the same setup works on every laptop.

• 6. Windows tip

  Plain Windows often fails on winutils.exe and HADOOP_HOME, so use WSL2 with Ubuntu or Docker Desktop instead.

• 7. Colab · Kaggle

  Colab and Kaggle give you one free cloud machine in the browser; pip install pyspark and run local mode there.

• 8. Databricks Free

  Community Edition was replaced by Databricks Free Edition in 2025: free serverless Spark with Python and SQL, for learning only.

• 9. AWS: EMR · Glue

  On AWS, EMR on EC2 lets you pick the machines, EMR Serverless and Glue pick them for you, and you pay for what runs.

• 10. Azure

  On Azure most teams use Azure Databricks or Microsoft Fabric Spark; Synapse Spark pools still run, and HDInsight is the old road.

• 11. GCP

  Google joined Dataproc and its Serverless Spark into Managed Service for Apache Spark: pick clusters or go serverless.

• 12. Databricks everywhere

  Databricks is one Spark platform that sits on top of AWS, Azure or GCP, so the same notebook skills work on all three.

• 13. Self-managed

  You can run Spark yourself on EC2 machines, on Kubernetes, or on an office Hadoop YARN cluster, but you fix everything.

• 14. Same script everywhere

  The PySpark logic stays the same; only where the master comes from and the data path change, from a local file to s3.

• 15. Which one to choose?

  Learn on a laptop, Colab or Databricks Free, build interview projects on Databricks Free plus a small AWS Glue job, and at work use what the company uses.

• 16. Suresh's journey

  Suresh starts on his laptop, practises online for free, and moves the same PySpark code to cloud platforms when the data grows.

Sources (checked October 2026)

• https://apache.googlesource.com/spark/+/master/docs/index.md (Java 17/21/25, Python 3.11+ on master)
• https://spark.apache.org/docs/4.2.0/api/python/getting_started/install.html (Python 3.10+, Java 17+, pip install pyspark)
• https://docs.databricks.com/aws/getting-started/free-edition-limitations
• https://docs.databricks.com/aws/en/getting-started/ce-migration (Community Edition to Free Edition)
• https://docs.aws.amazon.com/glue/latest/dg/worker-types.html (DPU, worker types)
• https://aws.amazon.com/blogs/big-data/introducing-aws-glue-5-1-for-apache-spark (Glue 5.1 / 5.0 Spark versions)
• https://aws-news.com/article/2026-09-22-amazon-emr-714-is-now-available (EMR 7.14 = Spark 3.5.8)
• https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark810-release.html (emr-spark-8.1.0 = Spark 4.1.1 LTS)
• https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-release-versions.html
• https://cloud.google.com/products/serverless-spark (Managed Service for Apache Spark, formerly Dataproc)
• https://blog.fabric.microsoft.com/blog/microsoft-fabric-explained-for-existing-synapse-users (no current plan to retire Synapse; Fabric is the focus)
• https://www.directionsonmicrosoft.com/reports/hdinsight-stagnates-time-to-migrate/ (HDInsight: no new investment)
• https://jupyter-docker-stacks.readthedocs.io/en/latest/using/specifics.html (pyspark-notebook image)