The previous post covered filtering and sorting. Here we use Pokémon data to calculate group statistics and explore conditional aggregations and window functions.
These examples were originally written for Polars 1.5. Their behavior with other versions has not been separately verified.
import polars as pl
df = pl.read_csv("Pokemon.csv")
1. Basic grouping
Group by one column
Use group_by to group rows and agg to apply aggregations to each group. For example, count Pokémon by their primary type:
type_counts = df.group_by("Type 1").agg(
pl.len().alias("count")
)
print(type_counts.sort("count", descending=True).head(5))
shape: (5, 2)
┌─────────┬───────┐
│ Type 1 ┆ count │
│ --- ┆ --- │
│ str ┆ u32 │
╞═════════╪═══════╡
│ Water ┆ 112 │
│ Normal ┆ 98 │
│ Grass ┆ 70 │
│ Bug ┆ 69 │
│ Psychic ┆ 57 │
└─────────┴───────┘
Group by multiple columns
Combining keys creates more specific groups, such as one for each generation and type:
gen_type_stats = df.group_by(["Generation", "Type 1"]).agg([
pl.col("HP").mean().alias("avg_HP"),
pl.col("Attack").mean().alias("avg_Attack"),
])
print(gen_type_stats.head())
Adding keys can create more groups with fewer records in each. Group output order is not guaranteed by default; sort the result when a consistent order is needed.
2. Aggregation functions
Basic aggregations
A single agg call can calculate several statistics, including counts, means, sums, standard deviations, and maximums:
type_stats = df.group_by("Type 1").agg([
pl.len().alias("count"),
pl.col("HP").mean().alias("avg_HP"),
pl.col("Attack").sum().alias("total_Attack"),
pl.col("Defense").std().alias("std_Defense"),
pl.col("Speed").max().alias("max_Speed"),
])
print(type_stats.head())
For numeric data, other options include mean(), sum(), std(), var(), min(), max(), median(), and quantile(). n_unique() counts distinct values, first() and last() return the first and last values in a group, and value_counts() counts occurrences of each value. Unlike pl.len(), which counts rows, count() counts non-null values.
Conditional aggregation
Sum a boolean expression to count matching rows, then calculate their percentage within each group:
type_hp_stats = df.group_by("Type 1").agg([
pl.col("HP").count().alias("total"),
(pl.col("HP") >= 100).sum().alias("high_hp_count"),
((pl.col("HP") >= 100).sum() / pl.col("HP").count() * 100)
.alias("high_hp_ratio"),
])
print(type_hp_stats.sort("high_hp_ratio", descending=True))
3. Window functions and multi-column aggregation
Use a window function
over performs a calculation within each group while retaining the original rows. This example ranks Attack within each type:
ranked_pokemon = df.with_columns([
pl.col("Attack").rank()
.over("Type 1")
.alias("attack_rank_in_type")
])
print(ranked_pokemon.select(
["Name", "Type 1", "Attack", "attack_rank_in_type"]
).head())
shape: (5, 4)
┌───────────────────────┬────────┬────────┬─────────────────────┐
│ Name ┆ Type 1 ┆ Attack ┆ attack_rank_in_type │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ i64 ┆ f64 │
╞═══════════════════════╪════════╪════════╪═════════════════════╡
│ Bulbasaur ┆ Grass ┆ 49 ┆ 13.5 │
│ Ivysaur ┆ Grass ┆ 62 ┆ 27.0 │
│ Venusaur ┆ Grass ┆ 82 ┆ 43.5 │
│ VenusaurMega Venusaur ┆ Grass ┆ 100 ┆ 58.5 │
│ Charmander ┆ Fire ┆ 52 ┆ 7.5 │
└───────────────────────┴────────┴────────┴─────────────────────┘
rank() uses ascending order by default. Use rank(descending=True) to give higher Attack values lower rank numbers.
Aggregate multiple columns
Select multiple columns to apply the same aggregations, then use name.suffix to distinguish the output names:
type_stats_all = df.group_by("Type 1").agg([
pl.col(["HP", "Attack", "Defense", "Speed"]).mean().name.suffix("_avg"),
pl.col(["HP", "Attack", "Defense", "Speed"]).std().name.suffix("_std"),
])
print(type_stats_all.head())
4. Grouping and aggregation tips
Select the columns you need
Selecting only the columns needed for an aggregation can reduce unnecessary data:
efficient_group_by = (
df.select(["Type 1", "HP", "Attack"])
.group_by("Type 1")
.agg([
pl.col("HP").mean(),
pl.col("Attack").mean(),
])
)
print(efficient_group_by)
Use lazy evaluation
Build a query with lazy() and execute it with collect(). This example starts with an already loaded DataFrame. Use pl.scan_csv() to defer reading the file as well.
lazy_group_by = (
df.lazy()
.group_by("Type 1")
.agg([
pl.col("HP").mean(),
pl.col("Attack").mean(),
])
.collect()
)
print(lazy_group_by)
Extract complex expressions into functions and use alias for descriptive output names to make aggregation pipelines easier to read.
Preserve group order
A default group_by does not guarantee group output order. Set maintain_order=True to keep groups in the order they first appear. Maintaining order can have a performance cost.
basic_group = df.group_by("Type 1").agg([
pl.col("HP").mean().alias("avg_HP")
])
print("Default grouping:")
print(basic_group.head())
ordered_group = df.group_by("Type 1", maintain_order=True).agg([
pl.col("HP").mean().alias("avg_HP")
])
print("Order-preserving grouping:")
print(ordered_group.head())
The next post covers joining and reshaping data.