I know Python + pandas: how much effort for PySpark?

You already carry two thirds of the bag. Panda cooks alone in one kitchen; Sparky runs a restaurant chain on many EC2 machines. Real pandas vs PySpark code and outputs on the CareWell file.

📊 Effort: what you already know (teacher's estimate)

If you know Python and pandas well, about 60–70% of PySpark is already yours. The new 30–40% is mostly how Spark runs on many EC2 machines. These percentages are the teacher's estimate, not a measurement.

Topic	Already known	New effort
Python language	100%	0%
Data thinking (rows, columns, joins, aggregations)	~90%	~10%
DataFrame API syntax	~60%	~40%
Spark SQL (if you know SQL)	~70%	~30%
How Spark runs: lazy plans, partitions, shuffle, jobs / stages / tasks	0%	100%
Performance tuning: joins, skew, caching, AQE, file sizes	0%	100%
Cluster & cloud: EMR, Glue, Databricks, S3, spark-submit, Spark UI	0%	100%
Overall	~60–70%	~30–40%

🍳 pandas vs PySpark

	🐼 pandas	⚡ PySpark
Where it runs	one machine, data must fit in RAM (GBs)	driver + executors on many EC2 machines (TBs)
When work happens	eager: every line runs now	lazy: builds a plan; an action (count, show, write, collect) runs it
Changing data	mutable; `inplace=True` changes the same DataFrame	immutable; every step returns a new DataFrame
Index / row order	has an index (`[0, 1, 2]`), order is kept	no index (`hasattr(df, "index")` = False), order only after `orderBy`
Missing value	`NaN` (real: nan)	`null` (Python `None`)
Output	one file (`out_pandas.csv`)	a folder of part files + `_SUCCESS` (real: 3 part files)
Custom Python	`apply()` is fine	Python UDFs are slow (rows go to a Python worker); prefer `F.` functions
Small data speed	fast: 18 ms (read + groupby, 1,000 rows)	slower: 579 ms, plus 7,313 ms SparkSession start-up (sandbox)
Optimizer	none: runs what you wrote	Catalyst + AQE: pushdown, pruning, join choice

Timings are from one small run in a sandbox (1,000 rows, PySpark 4.2.0 local[3], pandas 2.3.3). On small data pandas wins; Spark pays off when data does not fit one machine.

🔁 Syntax translation (pandas → PySpark)

Task	pandas	PySpark (`from pyspark.sql import functions as F`)
look at rows	`pdf.head(3)`	`df.show(3)`
new column	`pdf["c"] = expr`	`df.withColumn("c", expr)`
filter	`pdf[pdf["age"] >= 60]`	`df.filter(F.col("age") >= 60)`
group + aggregate	`pdf.groupby("d")["x"].mean()`	`df.groupBy("d").agg(F.avg("x"))`
join	`pdf.merge(o, on="k")`	`df.join(o, on="k")`
missing	`pdf["c"].isna() / fillna({...})`	`F.col("c").isNull() / fillna({...})`
sort	`sort_values("x", ascending=False)`	`orderBy(F.desc("x"))`
row count	`len(pdf)`	`df.count()`
write	`pdf.to_csv("out.csv")`	`df.write.csv("out_folder")`
drop column	`pdf.drop(columns=["c"])`	`df.drop("c")`

⭐ Only in PySpark: 11 things to learn

💤Lazy evaluation: transformations build a plan; actions run it
🧩Partitions: data split into pieces; one task per partition
🔀Shuffle (narrow / wide): wide steps (groupBy, join, orderBy) move rows between EC2 machines
🧠Driver, executors, jobs, stages, tasks: one action = a job; stages split at shuffles; tasks run on executors
📡Join strategies: broadcast a small table, or sort-merge two big ones
⚖️Data skew: one huge key makes one slow task
🔍AQE & optimizer: explain(), predicate pushdown, column pruning, run-time plan changes
🧊cache / persist: keep a reused DataFrame in memory
🐢F functions, not UDFs: built-in functions stay in the JVM; Python UDFs are slow
📦File formats & schema: Parquet, partitionBy, explicit StructType schema
☁️Running on a cluster: spark-submit, Spark UI, EMR / Glue / Databricks, S3

🛑 Habits to break

🔁No Python loops, no collect() on big data: use column expressions and write results to S3
✏️No inplace: always reassign: df = df.withColumn(...)
🔢Do not rely on row order or an index: sort with orderBy and use a key column

🗺️ 4-week plan

When	What
Weeks 1–2	DataFrame API + Spark SQL: translate your pandas code (fast, mostly known)
Weeks 3–4	Internals + tuning: lazy plans, partitions, shuffle, joins, skew, cache, AQE, Spark UI (the real effort)
Then	A project: S3 raw → Glue or EMR PySpark job → Parquet curated layer, run with spark-submit

🎤 Interview Q&A

What is the difference between pandas and PySpark?
pandas runs eagerly on one machine and the data must fit in its RAM. PySpark is the Python API for Spark: it builds a lazy plan, optimizes it, and runs it in parallel on partitions across many EC2 machines. pandas DataFrames are mutable and have an index; Spark DataFrames are immutable, have no index, and write a folder of part files.
When would you use pandas over Spark?
When the data fits comfortably in one machine's memory: small files, quick analysis, plotting, unit-sized lookups. Spark has start-up and scheduling overhead; in our sandbox run pandas took 18 ms and Spark 579 ms on 1,000 rows (plus 7,313 ms to start the SparkSession).
Why is Spark lazy?
So the optimizer sees the whole plan before running: it pushes filters to the source, reads only needed columns, combines narrow steps into one stage, and picks join strategies. Nothing is read until an action such as count(), show() or write.
Why avoid Python UDFs?
Built-in F. functions run inside the JVM and the optimizer understands them. A Python UDF sends every row to a Python worker and back (serialization), and is a black box for the optimizer. Use F functions first, then pandas UDFs (Arrow) if you must write custom Python.

Real code and real outputs

Each pair ran in one Python process: pandas 2.3.3 and PySpark 4.2.0 (local[3], driver memory 2g), on the CareWell file healthcare_dataset_1000.csv (1,000 rows).

Setup · pandasreal run

import pandas as pd
PATH = "healthcare_dataset_1000.csv"          # CareWell, 1,000 rows
pdf = pd.read_csv(PATH)

Setup · PySparkreal run

import os, sys, time
os.environ["PYSPARK_PYTHON"] = sys.executable
from pyspark.sql import SparkSession, functions as F
t = time.time()
spark = (SparkSession.builder.master("local[3]").appName("pandas-to-pyspark")
         .config("spark.driver.memory", "2g").config("spark.ui.port", "4074")
         .config("spark.ui.showConsoleProgress", "false").getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
START_MS = (time.time() - t) * 1000
df = spark.read.csv(PATH, header=True, inferSchema=True)

head · pandasreal run

print(pdf[["patient_id", "age", "city"]]
      .head(3))

  patient_id   age           city
0      P2000  53.0        Chennai
1      P2001  69.0  Visakhapatnam
2      P2002  34.0        Kurnool

head · PySparkreal run

(df.select("patient_id", "age", "city")
   .show(3))

+----------+---+-------------+
|patient_id|age|         city|
+----------+---+-------------+
|     P2000| 53|      Chennai|
|     P2001| 69|Visakhapatnam|
|     P2002| 34|      Kurnool|
+----------+---+-------------+
only showing top 3 rows

newcol · pandasreal run

pdf["cost_lakh"] = (pdf["treatment_cost"]
                    / 1e5).round(2)
print(pdf["cost_lakh"].head(3).tolist())

[0.92, 0.08, 0.1]

newcol · PySparkreal run

df2 = df.withColumn("cost_lakh",
    F.round(F.col("treatment_cost") / 1e5, 2))
print([r[0] for r in
       df2.select("cost_lakh").take(3)])

[0.92, 0.08, 0.1]

filter · pandasreal run

old = pdf[pdf["age"] >= 60]
print("rows:", len(old))

rows: 219

filter · PySparkreal run

old = df.filter(F.col("age") >= 60)
print("rows:", old.count())

rows: 219

group · pandasreal run

g = (pdf.groupby("department")
       ["treatment_cost"].mean()
       .round(0).sort_values(ascending=False)
       .head(3))
print(g)

department
Orthopedics      73180.0
ENT              66957.0
Ophthalmology    63465.0
Name: treatment_cost, dtype: float64

group · PySparkreal run

(df.groupBy("department")
   .agg(F.round(F.avg("treatment_cost"), 0)
        .alias("avg"))
   .orderBy(F.desc("avg"))
   .show(3))

+-------------+-------+
|   department|    avg|
+-------------+-------+
|  Orthopedics|73180.0|
|          ENT|66957.0|
|Ophthalmology|63465.0|
+-------------+-------+
only showing top 3 rows

join · pandasreal run

floors = pd.DataFrame({
  "department": ["Cardiology", "Oncology"],
  "floor": [3, 5]})
j = pdf.merge(floors, on="department")
print(len(j), j["floor"].unique().tolist())

123 [3, 5]

join · PySparkreal run

floors = spark.createDataFrame(
  [("Cardiology", 3), ("Oncology", 5)],
  ["department", "floor"])
j = df.join(floors, on="department")
print(j.count(), sorted(r[0] for r in
      j.select("floor").distinct().collect()))

123 [3, 5]

fillna · pandasreal run

print("missing:", pdf["blood_group"]
      .isna().sum())
f = pdf.fillna({"blood_group": "Unknown"})
print("Unknown:", (f["blood_group"]
      == "Unknown").sum())

missing: 17
Unknown: 17

fillna · PySparkreal run

print("missing:", df.filter(
  F.col("blood_group").isNull()).count())
f = df.fillna({"blood_group": "Unknown"})
print("Unknown:", f.filter(
  F.col("blood_group") == "Unknown").count())

missing: 17
Unknown: 17

sort · pandasreal run

print(pdf.sort_values("treatment_cost",
      ascending=False)
      [["patient_id", "treatment_cost"]]
      .head(3).to_string(index=False))

patient_id  treatment_cost
     P2411        231757.0
     P2262        231384.0
     P2442        230752.0

sort · PySparkreal run

(df.orderBy(F.desc("treatment_cost"))
   .select("patient_id", "treatment_cost")
   .show(3))

+----------+--------------+
|patient_id|treatment_cost|
+----------+--------------+
|     P2411|        231757|
|     P2262|        231384|
|     P2442|        230752|
+----------+--------------+
only showing top 3 rows

write · pandasreal run

pdf.to_csv("out_pandas.csv", index=False)
print([f for f in os.listdir(".")
       if f.startswith("out_pandas")])

['out_pandas.csv']

write · PySparkreal run

df.repartition(3).write.mode("overwrite").csv(
    "out_spark", header=True)
for f in sorted(os.listdir("out_spark")):
    if not f.startswith("."):
        print(f[:24] + ("…" if len(f) > 24 else ""))

_SUCCESS
part-00000-a368f357-b32a…
part-00001-a368f357-b32a…
part-00002-a368f357-b32a…

lazy · pandasreal run

t = time.time()
old = pdf[pdf["age"] >= 60]       # runs NOW
print(type(old).__name__, len(old), "rows")
print(f"{(time.time()-t)*1000:.1f} ms")

DataFrame 219 rows
0.8 ms

lazy · PySparkreal run

t = time.time()
old = df.filter(F.col("age") >= 60)  # plan
print(type(old).__name__, "no rows yet")
print(f"{(time.time()-t)*1000:.1f} ms")
t = time.time(); n = old.count()     # action
print(n, f"rows {(time.time()-t)*1000:.0f} ms")

DataFrame no rows yet
25.2 ms
219 rows 204 ms

mutable · pandasreal run

pdf.drop(columns=["name"], inplace=True)
print("columns:", pdf.shape[1])

columns: 15

mutable · PySparkreal run

df3 = df.drop("name")             # NEW df
print("df :", len(df.columns), "columns")
print("df3:", len(df3.columns), "columns")

df : 16 columns
df3: 15 columns

nan · pandasreal run

print(pdf["blood_group"].iloc[1])
print(pdf.index[:3].tolist())

nan
[0, 1, 2]

nan · PySparkreal run

print(df.collect()[1]["blood_group"])
print("index? ", hasattr(df, "index"))

None
index?  False

timing · pandasreal run

t = time.time()
r = pd.read_csv(PATH).groupby("city")\
      ["treatment_cost"].sum()
print("pandas :", f"{(time.time()-t)*1000:.0f} ms")

pandas : 18 ms

timing · PySparkreal run

t = time.time()
r = (spark.read.csv(PATH, header=True,
      inferSchema=True).groupBy("city")
      .agg(F.sum("treatment_cost")).collect())
print("spark  :", f"{(time.time()-t)*1000:.0f} ms")
print("start-up:", f"{START_MS:,.0f} ms")

spark  : 579 ms
start-up: 7,313 ms

parts · pandasreal run

# pandas: one machine, one block of RAM
print("pandas: 1 piece, no shuffle")

pandas: 1 piece, no shuffle

parts · PySparkreal run

print("partitions:",
      df.rdd.getNumPartitions())
print("after repartition(3):",
      df.repartition(3).rdd.getNumPartitions())
plan = (df.groupBy("city").count()
        ._jdf.queryExecution()
        .executedPlan().toString())
print("shuffle (Exchange):", "Exchange" in plan)

partitions: 1
after repartition(3): 3
shuffle (Exchange): True

plan · pandasreal run

# pandas has no optimizer: it runs
# each line exactly as you wrote it
print("pandas: no plan, no explain()")

pandas: no plan, no explain()

plan · PySparkreal run

small = spark.createDataFrame(
  [("Cardiology", 3)], ["department", "floor"])
p = (df.join(small, "department")
       ._jdf.queryExecution()
       .executedPlan().toString())
print("BroadcastHashJoin:", "BroadcastHashJoin" in p)

BroadcastHashJoin: True

The full script

pandas_vs_pyspark_demo.pysetup + pairs above

# pandas_vs_pyspark_demo.py : setup + every pair below, in the order they ran
# (the pandas side re-reads the CSV before each pair so inplace changes do not leak)
import pandas as pd
PATH = "healthcare_dataset_1000.csv"          # CareWell, 1,000 rows
pdf = pd.read_csv(PATH)
import os, sys, time
os.environ["PYSPARK_PYTHON"] = sys.executable
from pyspark.sql import SparkSession, functions as F
t = time.time()
spark = (SparkSession.builder.master("local[3]").appName("pandas-to-pyspark")
         .config("spark.driver.memory", "2g").config("spark.ui.port", "4074")
         .config("spark.ui.showConsoleProgress", "false").getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
START_MS = (time.time() - t) * 1000
df = spark.read.csv(PATH, header=True, inferSchema=True)

# ---- head : pandas
pdf = pd.read_csv(PATH)
print(pdf[["patient_id", "age", "city"]]
      .head(3))
# ---- head : PySpark
(df.select("patient_id", "age", "city")
   .show(3))

# ---- newcol : pandas
pdf = pd.read_csv(PATH)
pdf["cost_lakh"] = (pdf["treatment_cost"]
                    / 1e5).round(2)
print(pdf["cost_lakh"].head(3).tolist())
# ---- newcol : PySpark
df2 = df.withColumn("cost_lakh",
    F.round(F.col("treatment_cost") / 1e5, 2))
print([r[0] for r in
       df2.select("cost_lakh").take(3)])

# ---- filter : pandas
pdf = pd.read_csv(PATH)
old = pdf[pdf["age"] >= 60]
print("rows:", len(old))
# ---- filter : PySpark
old = df.filter(F.col("age") >= 60)
print("rows:", old.count())

# ---- group : pandas
pdf = pd.read_csv(PATH)
g = (pdf.groupby("department")
       ["treatment_cost"].mean()
       .round(0).sort_values(ascending=False)
       .head(3))
print(g)
# ---- group : PySpark
(df.groupBy("department")
   .agg(F.round(F.avg("treatment_cost"), 0)
        .alias("avg"))
   .orderBy(F.desc("avg"))
   .show(3))

# ---- join : pandas
pdf = pd.read_csv(PATH)
floors = pd.DataFrame({
  "department": ["Cardiology", "Oncology"],
  "floor": [3, 5]})
j = pdf.merge(floors, on="department")
print(len(j), j["floor"].unique().tolist())
# ---- join : PySpark
floors = spark.createDataFrame(
  [("Cardiology", 3), ("Oncology", 5)],
  ["department", "floor"])
j = df.join(floors, on="department")
print(j.count(), sorted(r[0] for r in
      j.select("floor").distinct().collect()))

# ---- fillna : pandas
pdf = pd.read_csv(PATH)
print("missing:", pdf["blood_group"]
      .isna().sum())
f = pdf.fillna({"blood_group": "Unknown"})
print("Unknown:", (f["blood_group"]
      == "Unknown").sum())
# ---- fillna : PySpark
print("missing:", df.filter(
  F.col("blood_group").isNull()).count())
f = df.fillna({"blood_group": "Unknown"})
print("Unknown:", f.filter(
  F.col("blood_group") == "Unknown").count())

# ---- sort : pandas
pdf = pd.read_csv(PATH)
print(pdf.sort_values("treatment_cost",
      ascending=False)
      [["patient_id", "treatment_cost"]]
      .head(3).to_string(index=False))
# ---- sort : PySpark
(df.orderBy(F.desc("treatment_cost"))
   .select("patient_id", "treatment_cost")
   .show(3))

# ---- write : pandas
pdf = pd.read_csv(PATH)
pdf.to_csv("out_pandas.csv", index=False)
print([f for f in os.listdir(".")
       if f.startswith("out_pandas")])
# ---- write : PySpark
df.repartition(3).write.mode("overwrite").csv(
    "out_spark", header=True)
for f in sorted(os.listdir("out_spark")):
    if not f.startswith("."):
        print(f[:24] + ("…" if len(f) > 24 else ""))

# ---- lazy : pandas
pdf = pd.read_csv(PATH)
t = time.time()
old = pdf[pdf["age"] >= 60]       # runs NOW
print(type(old).__name__, len(old), "rows")
print(f"{(time.time()-t)*1000:.1f} ms")
# ---- lazy : PySpark
t = time.time()
old = df.filter(F.col("age") >= 60)  # plan
print(type(old).__name__, "no rows yet")
print(f"{(time.time()-t)*1000:.1f} ms")
t = time.time(); n = old.count()     # action
print(n, f"rows {(time.time()-t)*1000:.0f} ms")

# ---- mutable : pandas
pdf = pd.read_csv(PATH)
pdf.drop(columns=["name"], inplace=True)
print("columns:", pdf.shape[1])
# ---- mutable : PySpark
df3 = df.drop("name")             # NEW df
print("df :", len(df.columns), "columns")
print("df3:", len(df3.columns), "columns")

# ---- nan : pandas
pdf = pd.read_csv(PATH)
print(pdf["blood_group"].iloc[1])
print(pdf.index[:3].tolist())
# ---- nan : PySpark
print(df.collect()[1]["blood_group"])
print("index? ", hasattr(df, "index"))

# ---- timing : pandas
pdf = pd.read_csv(PATH)
t = time.time()
r = pd.read_csv(PATH).groupby("city")\
      ["treatment_cost"].sum()
print("pandas :", f"{(time.time()-t)*1000:.0f} ms")
# ---- timing : PySpark
t = time.time()
r = (spark.read.csv(PATH, header=True,
      inferSchema=True).groupBy("city")
      .agg(F.sum("treatment_cost")).collect())
print("spark  :", f"{(time.time()-t)*1000:.0f} ms")
print("start-up:", f"{START_MS:,.0f} ms")

# ---- parts : pandas
pdf = pd.read_csv(PATH)
# pandas: one machine, one block of RAM
print("pandas: 1 piece, no shuffle")
# ---- parts : PySpark
print("partitions:",
      df.rdd.getNumPartitions())
print("after repartition(3):",
      df.repartition(3).rdd.getNumPartitions())
plan = (df.groupBy("city").count()
        ._jdf.queryExecution()
        .executedPlan().toString())
print("shuffle (Exchange):", "Exchange" in plan)

# ---- plan : pandas
pdf = pd.read_csv(PATH)
# pandas has no optimizer: it runs
# each line exactly as you wrote it
print("pandas: no plan, no explain()")
# ---- plan : PySpark
small = spark.createDataFrame(
  [("Cardiology", 3)], ["department", "floor"])
p = (df.join(small, "department")
       ._jdf.queryExecution()
       .executedPlan().toString())
print("BroadcastHashJoin:", "BroadcastHashJoin" in p)

spark.stop()

The steps in words

1. The question
Suresh already carries two backpacks, Python and pandas, and asks Sparky how much more he must learn.
2. Effort meter
Python and pandas lift Suresh about two thirds up the mountain, so only about 30 to 40 percent is new.
3. Breakdown
The language, data thinking and most syntax are already yours; how Spark runs, tuning and the cluster are fully new.
4. Where the 35% goes
A little new syntax and SQL, but most of the new effort is how Spark runs, tuning, and the cluster in the cloud.
5. One cook vs chain
Panda cooks alone in one machine's RAM, fine for gigabytes; Sparky is the head chef who sends work to cooks on many EC2 machines for terabytes.
6. Eager vs lazy
Panda cooks every line immediately; Sparky only writes the order and cooks when an action like count() says serve.
7. Same plate vs new plate
In pandas inplace changes the same DataFrame; in Spark the old DataFrame never changes, every step returns a new one, with no index and null instead of NaN.
8. One file vs folder
Panda writes one CSV file; in Spark every partition writes its own part file into a folder, and _SUCCESS says the job finished.
9. head · new column
head becomes show, and a new column is made with withColumn; the real values come out the same.
10. filter · sort
A boolean mask becomes filter with F.col, and sort_values becomes orderBy with F.desc.
11. groupby · join
groupby becomes groupBy with agg and F functions, and merge becomes join with the same on= key.
12. missing values
isna becomes isNull, and fillna keeps its name in Spark, so 17 missing blood groups become 17 Unknown in both.
13. Spark engine basics
Spark plans lazily, splits data into partitions, moves rows between EC2 machines in a shuffle, and runs jobs, stages and tasks.
14. Joins, skew, optimizer
Spark picks a broadcast or sort-merge join, suffers when one key is huge, and rewrites your plan with the optimizer and AQE.
15. Cache, F, files, cluster
Cache what you reuse, prefer F functions over Python UDFs, write Parquet with a schema, and run it all on a cluster with spark-submit.
16. Habits to break
Stop looping and collecting big data, stop using inplace, and stop trusting row order or an index.
17. Learning plan
Weeks one and two for the API and Spark SQL, weeks three and four for internals and tuning, then an S3 plus Glue or EMR project.
18. Summary
You already know about 65 percent; the new 35 percent is how Spark runs on many EC2 machines, so focus there.