20 PySpark Commands Every Spark Developer Should Know

Learning PySpark, the Python API for Apache Spark? About twenty PySpark commands cover most of the work in real big data pipelines. Once you know what each one does and when to use it, the rest of the API is a lookup away.

This post walks through those operations. Each one comes with an explanation, a snippet, and a real use case. The last section combines them into a working log-analysis job.

Spark transformations vs actions: lazy evaluation explained

Spark operations come in two kinds:

• Transformations (map, filter, reduceByKey…) describe what to compute. They return a new RDD or DataFrame and run nothing.
• Actions (count, collect, save…) trigger the actual computation and return a result.

Spark builds a plan from your transformations and only executes it when an action asks for a result. That lets it optimize the whole pipeline at once. It is also why a bug in your map only shows up when you call collect().

Sample data for the PySpark examples

All examples use two datasets, so the snippets build on each other:

# Web server access logs:
# "2026-10-21 10:01:33 192.168.1.5 GET /products/42 200 532ms"
logs = sc.textFile('access.log')

# E-commerce orders: (user_id, city, amount)
orders = sc.parallelize([
    ('u1', 'Casa', 250), ('u2', 'Rabat', 90),
    ('u1', 'Casa', 400), ('u3', 'Fes', 120),
])

PySpark RDD transformations: map, filter, reduceByKey and more

1. map(f): one in, one out

Applies f to every element and returns exactly one result per input. It is the go-to tool for parsing and reshaping records.

parsed = logs.map(lambda line: line.split())

Real use case: turn raw log lines into structured fields (IP, URL, status, latency) before any analysis.

2. filter(f): keep what matters

Keeps only the elements for which f returns True.

errors = parsed.filter(lambda f: f[5].startswith('5'))   # HTTP 5xx

Real use case: isolate server errors, flagged transactions, or orders above a threshold. Filter as early as possible so every later step processes less data.

3. flatMap(f): one in, many out

Like map, but f returns a list, and Spark flattens all the lists into one RDD.

words = sc.textFile('reviews.txt').flatMap(lambda l: l.lower().split())

Real use case: split product reviews into words for keyword analysis, or explode an order into its line items.

4. distinct(): remove duplicates

unique_ips = parsed.map(lambda f: f[2]).distinct()

Real use case: count unique visitors, or deduplicate records that were ingested twice. It triggers a shuffle, so don't call it casually on huge data.

5. union(other): stack two datasets

all_logs = sc.textFile('server1.log').union(sc.textFile('server2.log'))

Real use case: merge logs from several servers, or this month's data with last month's. Duplicates are kept; chain distinct() if you need them gone.

6. mapValues(f): change the value, keep the key

Works on (key, value) pairs and transforms only the value.

prices = orders.map(lambda o: (o[0], o[2]))
with_vat = prices.mapValues(lambda amount: amount * 1.2)

Real use case: tax or currency conversion per key. It beats map here because Spark knows the keys did not change and keeps the existing partitioning.

7. reduceByKey(f): aggregate per key

Merges all values that share a key, using a function that combines two values into one.

revenue = orders.map(lambda o: (o[0], o[2])).reduceByKey(lambda a, b: a + b)
# [('u1', 650), ('u2', 90), ('u3', 120)]

Real use case: revenue per customer, hits per URL, errors per server. This is the workhorse of Spark. It pre-aggregates on each partition before shuffling, so very little data crosses the network.

A common beginner error is calling reduceByKey() on a plain list of words. It needs (key, value) pairs and a function:

words.map(lambda w: (w, 1)).reduceByKey(lambda a, b: a + b)

8. groupByKey(): gather all values per key

orders.map(lambda o: (o[0], o[2])).groupByKey().mapValues(list)
# [('u1', [250, 400]), ('u2', [90]), ('u3', [120])]

Real use case: when you need every value together, such as a user's full purchase history or the median latency per page. For sums and counts, use reduceByKey instead: groupByKey ships every single value across the network.

9. sortBy(f) and sortByKey(): rank results

top_pages = (parsed.map(lambda f: (f[4], 1))
                   .reduceByKey(lambda a, b: a + b)
                   .sortBy(lambda kv: kv[1], ascending=False))

Real use case: leaderboards such as most-visited pages, best-selling products, or top customers. sortByKey() sorts by key instead, which suits time-ordered reports.

10. join(other): combine datasets by key

users = sc.parallelize([('u1', 'Marouane'), ('u2', 'Sara')])
orders.map(lambda o: (o[0], o[2])).join(users)
# [('u1', (250, 'Marouane')), ('u1', (400, 'Marouane')), ('u2', (90, 'Sara'))]

Real use case: enrich transactions with customer details, or logs with geo-IP data. join is an inner join, so u3 disappears. Use leftOuterJoin to keep it, which is also how you find orders with no matching user.

PySpark RDD actions: collect, count, reduce and more

11. collect(): bring everything to the driver

Returns the whole dataset as a Python list.

Real use case: fetching a small final aggregate, or debugging on a sample. On a large dataset it pulls everything into the driver's memory and crashes it. It is the single most common way to break a Spark job.

12. take(n) and first(): peek safely

errors.take(5)

Real use case: checking that your parsing works while you develop, without loading the whole dataset.

13. count(): how many?

errors.count()

Real use case: KPIs such as total requests, error count, or orders today. Also a quick sanity check after a filter: if the count is 0, your condition is probably wrong.

14. reduce(f): collapse everything into one value

total_revenue = orders.map(lambda o: o[2]).reduce(lambda a, b: a + b)   # 860

Real use case: global totals, or the maximum or minimum across the whole dataset. Unlike reduceByKey, it produces one value, not one per key.

15. countByValue(): instant distribution

parsed.map(lambda f: f[5]).countByValue()
# {'200': 9500, '404': 320, '500': 12}

Real use case: a quick breakdown of HTTP status codes, categories, or ratings. The result is a Python dict on the driver, so use it only when there are few distinct values.

16. collectAsMap(): a lookup table in one line

Turns a pair RDD into a Python dict.

Real use case: build a small mapping such as product_id → name to use in the rest of your script.

17. stats(): summary statistics in one pass

latency = parsed.map(lambda f: int(f[6].rstrip('ms')))
latency.stats()   # count, mean, stdev, max, min

Real use case: latency monitoring or basket-size analysis. sum(), max() and mean() exist on their own too, but stats() gets all of them from a single scan.

18. saveAsTextFile(path): persist the output

top_pages.saveAsTextFile('hdfs:///reports/top_pages')

Real use case: write pipeline results to HDFS or S3 for other jobs and dashboards. It writes a directory containing one part file per partition, not a single file.

Spark performance tuning: cache, repartition and coalesce

19. cache() and persist(): don't compute twice

Keeps an RDD in memory after it is first computed, so later actions reuse it.

parsed = logs.map(lambda l: l.split()).cache()

Real use case: you run three reports on parsed (errors, top pages, latency). Without cache(), Spark re-reads and re-parses the log file for each of them, because every action replays the whole plan from the source. Use persist() when you want to choose the storage level, such as memory plus disk.

20. repartition(n) and coalesce(n): control parallelism

• repartition(n) reshuffles data into n partitions. It is a full shuffle, but it spreads the work evenly.
• coalesce(n) reduces the number of partitions by merging existing ones, with no full shuffle.

big = logs.repartition(200)        # more parallelism for heavy processing
report.coalesce(1).saveAsTextFile('out/report')   # one single output file

Real use case: repartition before an expensive step on a large cluster where a few partitions hold most of the data. coalesce(1) right before saving a small report that a person will open as one file.

PySpark DataFrame cheat sheet

RDDs teach you how Spark thinks, but most production code uses DataFrames. A DataFrame is a distributed table with a schema. Spark's Catalyst optimizer rewrites your query before running it, so DataFrame code is usually faster than hand-written RDD code.

Revenue per city, in DataFrame style:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
df = spark.read.csv('orders.csv', header=True, inferSchema=True)

(df.filter(F.col('amount') > 100)
   .groupBy('city')
   .agg(F.sum('amount').alias('revenue'),
        F.count('*').alias('orders'))
   .orderBy(F.desc('revenue'))
   .show())

And the classic word count, without a single lambda:

(spark.read.text('reviews.txt')
   .select(F.explode(F.split('value', r'\s+')).alias('word'))
   .groupBy('word').count()
   .orderBy(F.desc('count'))
   .show())

PySpark example: a complete log analysis job

This script answers five operations questions about a web server, and it uses most of the operations above:

parsed = sc.textFile('access.log').map(lambda l: l.split()).cache()

print("Total requests:", parsed.count())

print("Unique visitors:",
      parsed.map(lambda f: f[2]).distinct().count())

print("Status codes:",
      parsed.map(lambda f: f[5]).countByValue())

print("Top 5 pages:",
      parsed.map(lambda f: (f[4], 1))
            .reduceByKey(lambda a, b: a + b)
            .sortBy(lambda kv: -kv[1])
            .take(5))

print("Latency stats:",
      parsed.map(lambda f: int(f[6].rstrip('ms'))).stats())

How it works:

1. cache() parses the file once and shares the result across all five questions.
2. Each print holds one action, so each line triggers one Spark job.
3. The top-pages query is the canonical pattern: map to pairs → reduceByKey → sortBy → take. Learn it by heart; it solves a surprising number of problems.

Conclusion: PySpark best practices

Three habits separate working Spark code from fast Spark code:

• Use reduceByKey instead of groupByKey for sums and counts. It aggregates before the shuffle.
• Never collect() big data. Use take(), count(), or write the output to storage.
• cache() anything you reuse. Every action otherwise recomputes the plan from the source.

Master these 20 PySpark commands and the lazy-evaluation rule, and you can read and write most Spark pipelines you will meet in production.

PySpark FAQ

What is the difference between map and flatMap in PySpark?

map returns exactly one output per input element. flatMap returns zero or more outputs per input and flattens them into a single RDD, which is why it is used to split lines into words.

What is the difference between reduceByKey and groupByKey?

Both work on (key, value) pairs. reduceByKey combines values on each partition before the shuffle, so it is much faster for sums and counts. groupByKey sends every value across the network and is only worth it when you need all values per key.

Why does my PySpark code do nothing until I call collect()?

Transformations are lazy: Spark only builds an execution plan. The plan runs when an action such as collect(), count(), take() or saveAsTextFile() asks for a result.

Should I use RDDs or DataFrames?

Use DataFrames for most new code, because Spark's Catalyst optimizer makes them faster. Use RDDs to learn how Spark works or when you need low-level control over unstructured data.

When should I use cache() in Spark?

Cache an RDD or DataFrame when two or more actions reuse it. Otherwise each action recomputes it from the original source.