Writing

Pandas Data Operations: Masking, Filtering, and Grouping

This tutorial was written in 2020. The examples were checked in sequence with Pandas 2.2.3 and Faker 4.1.3.

This final post continues the DataFrame from parts one, two, and three.

df.head()
Pandas example output

Most examples return a result without assigning it back to df. Assign the result if you want to replace the original DataFrame.

Masking

mask replaces values where the condition is true. This masks rows with a Salary below 6000:

import numpy as np  # Import NumPy

df.mask(df["Salary"] < 6000, np.nan)
Pandas example output

where does the opposite: it replaces values where the condition is false.

df.where(df["Salary"] < 6000, np.nan)
Pandas example output

Filtering

Use loc to extract rows matching a condition. Here are rows with Salary at least 5000:

df.loc[df["Salary"] >= 5000]
Pandas example output

Combine conditions with & (and) or | (or), enclosing each condition in parentheses:

df.loc[(df["Salary"] >= 5000)&(df["Salary"] < 7000)]
Pandas example output ```python df.loc[(df["Salary"] >= 5000)|(df["WeeklyExercise"] < 4)] ``` Pandas example output

Sorting

sort_index sorts by index; ascending=False reverses the order:

df.sort_index()
Pandas example output ```python df.sort_index(ascending=False) ``` Pandas example output

sort_values sorts by one or more columns:

df.sort_values("Salary", ascending=False)
Pandas example output ```python df.sort_values(["WeeklyExercise", "WeeklyDrink"], ascending=False) ``` Pandas example output

Applying functions

apply applies a function to values or to rows. First check whether Pandas offers a built-in operation: vectorized methods are often faster than iterating through values.

This function extracts and joins the first character of each word in an address, then applies it to a Series:

def convert_address(address):
    splitted = address.split()
    first_characters = [each[0] for each in splitted]
    return "".join(first_characters)
df["NewAddress"]  = df["Address"].apply(convert_address)
df.head(10)
Pandas example output

For a DataFrame, apply passes rows with axis=1 or columns with axis=0. This illustrative function applies the author's exercise and drinking criteria to each row; it is not a medical assessment.

def is_healthy(row):
    if row["WeeklyDrink"] < 2:
        return True
    elif row["WeeklyDrink"] < 5 and row["WeeklyExercise"] > 2:
        return True
    elif row["WeeklyExercise"] > 4:
        return True
    else:
        return False
df["is_healthy"] = df.apply(is_healthy, axis=1)
df.head(10)
Pandas example output

For a short function, use a lambda expression:

df["Age"].apply(lambda x: (x // 10)*10)
Pandas example output

Grouping

Use groupby to calculate statistics for groups. For an index, specify the index level: a single index uses level 0, while a MultiIndex can use another level. Here the median Age and Salary are grouped by WeeklyDrink:

df.set_index("WeeklyDrink").groupby(level=0)[["Age", "Salary"]].median()
Pandas example output

Group by a column by passing its name:

df.groupby(["Gender"]).mean(numeric_only=True)
Pandas example output

You can also group by a function applied to each index label. This groups usernames by their first character:

def groupby_alphabet(index):
    return index[0]
df.groupby(groupby_alphabet)[["Salary", "WeeklyExercise", "Age"]].mean()
Pandas example output

This concludes the Pandas series. The Pandas documentation covers many more operations.