Writing

Getting Started with Polars: Architecture, Pandas Differences, and Quickstart

Polars logo banner

Polars is a lightning-fast DataFrame library built from the ground up in Rust on top of the Apache Arrow columnar memory format. If you have hit memory exhaustion or single-core bottlenecks when crunching datasets in Pandas, Polars offers a modern, high-performance alternative.

Pandas vs. Polars: Architectural Differences

  1. Rust Core & SIMD Execution: Pandas relies heavily on NumPy arrays and legacy C extensions. Polars is written in Rust and utilizes columnar memory layouts with SIMD vectorization.
  2. Automatic Multi-Threading: Operations in Polars split workloads across available CPU cores by default without manual multiprocessing setups.
  3. Expression API: In Pandas, dataframe mutations often involve direct indexing (df['c'] = ...) or in-place state modifications. Polars treats transformations as purely declarative expressions (pl.col("c")), allowing query optimization.
  4. Lazy Evaluation (LazyFrame): While Pandas executes operations eagerly, Polars lets you construct execution plans using .lazy(). The optimizer pushes filters down to the file level (predicate pushdown) and only reads requested columns (projection pushdown) before .collect() runs.
import polars as pl

# In Polars, transformations are declarative:
lazy_df = pl.LazyFrame({
    "A": [1, 2, 3, 4, 5],
    "B": [10, 20, 30, 40, 50],
    "C": [100, 200, 300, 400, 500]
})

# Plan is compiled and optimized before execution
result = (
    lazy_df
    .filter(pl.col("A") > 2)
    .select(pl.col("B"), (pl.col("C") * 2).alias("C_doubled"))
    .collect()
)
print(result)

Installation

pip install polars

Quickstart: Fundamental Operations

Let's initialize a sample DataFrame:

import polars as pl

df = pl.DataFrame({
    "name": ["Pikachu", "Charmander", "Bulbasaur", "Squirtle", "Charizard", "Blastoise"],
    "type": ["Electric", "Fire", "Grass", "Water", "Fire", "Water"],
    "weight": [6.0, 8.5, 6.9, 9.0, 90.5, 85.5],  # kg
    "height": [0.4, 0.6, 0.7, 0.5, 1.5, 1.6],   # m
})

print(df)
shape: (6, 4)
┌────────────┬──────────┬────────┬────────┐
│ name       ┆ type     ┆ weight ┆ height │
│ ---        ┆ ---      ┆ ---    ┆ ---    │
│ str        ┆ str      ┆ f64    ┆ f64    │
╞════════════╪══════════╪════════╪════════╡
│ Pikachu    ┆ Electric ┆ 6.0    ┆ 0.4    │
│ Charmander ┆ Fire     ┆ 8.5    ┆ 0.6    │
│ Bulbasaur  ┆ Grass    ┆ 6.9    ┆ 0.7    │
│ Squirtle   ┆ Water    ┆ 9.0    ┆ 0.5    │
│ Charizard  ┆ Fire     ┆ 90.5   ┆ 1.5    │
│ Blastoise  ┆ Water    ┆ 85.5   ┆ 1.6    │
└────────────┴──────────┴────────┴────────┘

1. Projecting Columns with select()

Use select() to isolate columns and compute derived features:

result = df.select(
    pl.col("name"),
    pl.col("type"),
    (pl.col("weight") / (pl.col("height") ** 2)).round(2).alias("bmi"),
)
print(result)

2. Appending Computed Columns with with_columns()

Rather than assigning directly to the DataFrame, use with_columns() to maintain immutability:

result = df.with_columns(
    bmi=(pl.col("weight") / (pl.col("height") ** 2)).round(2),
)

3. Row Filtering with filter()

Filter rows using boolean expressions over pl.col():

fire_types = df.filter(pl.col("type") == "Fire")
print(fire_types)

4. Grouped Aggregations with group_by()

Group by categorical variables and compute multi-metric summaries:

[!NOTE] Polars API Update In modern Polars (1.0+), use group_by() (not the legacy groupby()), and count rows using pl.len() rather than pl.count().

result = df.group_by("type", maintain_order=True).agg(
    pl.len().alias("sample_size"),
    pl.col("weight").mean().round(2).alias("avg_weight"),
    pl.col("height").max().alias("tallest"),
)
print(result)

Next in the series: Loading and Manipulating Data with Polars (CSV and Parquet).