Writing

Loading and Manipulating Data with Polars: CSV, Parquet, and Column Selectors

Following Part 1: Getting Started with Polars, this guide explores how to ingest datasets efficiently from CSV and Parquet, inspect dataset distributions, and manipulate columns using both standard expressions and modern column selectors.

Download the example data: Pokemon.csv and Pokemon.parquet.

Fast Data Ingestion: CSV vs. Parquet

1. Reading CSV Files

Standard CSV ingestion parses text data dynamically:

import polars as pl

df = pl.read_csv("Pokemon.csv")

To optimize memory usage and avoid inference overhead on large files, pass explicit types using schema_overrides:

df = pl.read_csv(
    "Pokemon.csv",
    schema_overrides={
        "Generation": pl.UInt8,   # Max 6, small unsigned integer fits perfectly
        "Legendary": pl.Boolean,
        "HP": pl.UInt16,
        "Attack": pl.UInt16,
        "Defense": pl.UInt16,
        "Speed": pl.UInt16,
    }
)

[!TIP] When querying multi-gigabyte CSV files where only specific fields are needed, pass columns=["Name", "Type 1", "HP"] directly into read_csv() to prevent reading unwanted bytes into RAM.

2. Reading Columnar Parquet Files

Apache Parquet stores metadata per column chunk, making ingestion significantly faster than CSV:

df = pl.read_parquet("Pokemon.parquet")

For massive datasets, use pl.scan_parquet("Pokemon.parquet") to create a LazyFrame. Polars only loads the exact row groups and columns requested by downstream operators.

Inspecting Data Structure and Distributions

Metadata & Summaries

print(df.schema)     # Dict of column names to Polars DataTypes
print(df.shape)      # (rows, cols) tuple
print(df.head(5))    # First 5 rows
print(df.tail(5))    # Last 5 rows
print(df.sample(5))  # Random 5-row sample

Statistical Analysis

Generate statistical distributions across numeric columns:

print(df.describe())

# Compute specific statistics across multiple columns
stats = df.select(
    pl.col(["HP", "Attack", "Defense"]).mean().round(2)
)
print(stats)

Common aggregation methods on expressions include .mean(), .median(), .std(), .var(), .quantile(0.95), .min(), and .max().

Selecting and Transforming Columns

1. Basic Selection and Exclusion

# Select specific columns
df_subset = df.select(["Name", "Type 1", "HP"])

# Exclude unwanted columns using wildcard and exclude()
df_clean = df.select(pl.col("*").exclude(["Generation", "Legendary"]))

2. Ergonomic Selection with polars.selectors

The polars.selectors module (cs) allows querying columns based on data types or string patterns:

import polars.selectors as cs

# Select by data type
numeric_cols = df.select(cs.numeric())
string_cols = df.select(cs.string())

# Select by column name patterns or regex
type_cols = df.select(cs.contains("Type"))
stat_cols = df.select(pl.col("^Sp.*$"))  # Regex matching 'Sp. Atk', 'Sp. Def'

3. Adding and Modifying Columns with with_columns()

with_columns() evaluates one or more expressions and appends or replaces columns. Passing expressions in a list allows Polars to execute calculations in parallel across CPU threads:

df_enhanced = df.with_columns(
    (pl.col("Attack") + pl.col("Defense")).alias("CombatPower"),
    (pl.col("HP") * 2).alias("DoubleHP"),
)

# Overwriting an existing column in-place:
df_normalized = df.with_columns(
    pl.col("Name").str.to_uppercase()
)

Next in the series: Filtering and Sorting Data with Polars.