Following Part 1: Getting Started with Polars, this guide explores how to ingest datasets efficiently from CSV and Parquet, inspect dataset distributions, and manipulate columns using both standard expressions and modern column selectors.
Download the example data: Pokemon.csv and Pokemon.parquet.
Fast Data Ingestion: CSV vs. Parquet
1. Reading CSV Files
Standard CSV ingestion parses text data dynamically:
import polars as pl
df = pl.read_csv("Pokemon.csv")
To optimize memory usage and avoid inference overhead on large files, pass explicit types using schema_overrides:
df = pl.read_csv(
"Pokemon.csv",
schema_overrides={
"Generation": pl.UInt8, # Max 6, small unsigned integer fits perfectly
"Legendary": pl.Boolean,
"HP": pl.UInt16,
"Attack": pl.UInt16,
"Defense": pl.UInt16,
"Speed": pl.UInt16,
}
)
[!TIP] When querying multi-gigabyte CSV files where only specific fields are needed, pass
columns=["Name", "Type 1", "HP"]directly intoread_csv()to prevent reading unwanted bytes into RAM.
2. Reading Columnar Parquet Files
Apache Parquet stores metadata per column chunk, making ingestion significantly faster than CSV:
df = pl.read_parquet("Pokemon.parquet")
For massive datasets, use pl.scan_parquet("Pokemon.parquet") to create a LazyFrame. Polars only loads the exact row groups and columns requested by downstream operators.
Inspecting Data Structure and Distributions
Metadata & Summaries
print(df.schema) # Dict of column names to Polars DataTypes
print(df.shape) # (rows, cols) tuple
print(df.head(5)) # First 5 rows
print(df.tail(5)) # Last 5 rows
print(df.sample(5)) # Random 5-row sample
Statistical Analysis
Generate statistical distributions across numeric columns:
print(df.describe())
# Compute specific statistics across multiple columns
stats = df.select(
pl.col(["HP", "Attack", "Defense"]).mean().round(2)
)
print(stats)
Common aggregation methods on expressions include .mean(), .median(), .std(), .var(), .quantile(0.95), .min(), and .max().
Selecting and Transforming Columns
1. Basic Selection and Exclusion
# Select specific columns
df_subset = df.select(["Name", "Type 1", "HP"])
# Exclude unwanted columns using wildcard and exclude()
df_clean = df.select(pl.col("*").exclude(["Generation", "Legendary"]))
2. Ergonomic Selection with polars.selectors
The polars.selectors module (cs) allows querying columns based on data types or string patterns:
import polars.selectors as cs
# Select by data type
numeric_cols = df.select(cs.numeric())
string_cols = df.select(cs.string())
# Select by column name patterns or regex
type_cols = df.select(cs.contains("Type"))
stat_cols = df.select(pl.col("^Sp.*$")) # Regex matching 'Sp. Atk', 'Sp. Def'
3. Adding and Modifying Columns with with_columns()
with_columns() evaluates one or more expressions and appends or replaces columns. Passing expressions in a list allows Polars to execute calculations in parallel across CPU threads:
df_enhanced = df.with_columns(
(pl.col("Attack") + pl.col("Defense")).alias("CombatPower"),
(pl.col("HP") * 2).alias("DoubleHP"),
)
# Overwriting an existing column in-place:
df_normalized = df.with_columns(
pl.col("Name").str.to_uppercase()
)
Next in the series: Filtering and Sorting Data with Polars.