Pandas is a library for storing and manipulating tabular data, or data stored in rows and columns like a spreadsheet. Pandas is a huge library with many different functions and methods, so what follows is a brief introduction to the most important functions for data management.
DataFrames and Series¶
Instead of normal Python lists and dictionaries, Pandas stores data in its own specialized objects. The main one is a DataFrame, which is a lot like a spreadsheet with rows and columns.
You can create a DataFrame directly with the DataFrame() class in Pandas, but it’s more likely that you’ll read in a DataFrame from a CSV or spreadsheet file. First you must import the library, and it’s a good idea to import the numpy library as well.
import pandas as pd
import numpy as npNow you can use the read_csv() function to read in a comma-separated value (CSV) spreadsheet file. You must put the name of this file in quotes, and the file should be in the same directory as your Jupyter notebook (or else you should include a full path). The read_csv() function will also accept a URL that points to a CSV file online.
For this example, we’ll use the file mpg.csv which comes from R’s ggplot2 library.
mpg = pd.read_csv("../data/mpg.csv")
mpgYou can get basic information about your DataFrames columns using the .info() method.
mpg.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 234 entries, 0 to 233
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 manufacturer 234 non-null object
1 model 234 non-null object
2 displ 234 non-null float64
3 year 234 non-null int64
4 cyl 234 non-null int64
5 trans 234 non-null object
6 drv 234 non-null object
7 cty 234 non-null int64
8 hwy 234 non-null int64
9 fl 234 non-null object
10 class 234 non-null object
dtypes: float64(1), int64(4), object(6)
memory usage: 20.2+ KB
A Series is a lot like a Python list, and each column of a DataFrame is a Series. You can access the columns of a Dataframe with dot notation.
mpg.model0 a4
1 a4
2 a4
3 a4
4 a4
...
229 passat
230 passat
231 passat
232 passat
233 passat
Name: model, Length: 234, dtype: objectYou can also turn a list into a Series with the Series() class.
myseries = pd.Series([5, 6, 7, 8])
myseries0 5
1 6
2 7
3 8
dtype: int64Selecting Rows and Columns¶
Once you have a DataFrame, you’ll typically want to filter and select different rows or columns.
To filter specific rows, Pandas uses a bracket notation. It takes conditional statements that are similar to Python conditions.
# Get cars with fewer than 6 cylinders
four_cylinders = mpg[mpg.cyl < 6]
four_cylindersYou can also use the operators & (and), | (or), and ! (not) to combine conditional filters.
# Get Volkswagens and Fords
vw_ford = mpg[(mpg.manufacturer == 'volkswagen') | (mpg.manufacturer == 'ford')]
vw_fordYou can use a double bracket notation to select a subset of columns.
class_cty_hwy = mpg[["class", "cty", "hwy"]]
class_cty_hwyData Wrangling¶
In addtion to selecting rows and columns from DataFrames, you can also use Pandas to do a wide variety of data transformations.
Sorting¶
mpg.sort_values("year", ascending=False)Counting¶
mpg.value_counts("manufacturer")manufacturer
dodge 37
toyota 34
volkswagen 27
ford 25
chevrolet 19
audi 18
hyundai 14
subaru 14
nissan 13
honda 9
jeep 8
pontiac 5
land rover 4
mercury 4
lincoln 3
dtype: int64Renaming Columns¶
# Note the use of a Python dictionary as this method's argument
mpg = mpg.rename({"cty":"city", "hwy": "highway"})
mpgCreate new columns¶
You can use assign() to create new columns based on existing ones.
mpg = mpg.assign(displ_per_cyl = mpg.displ/mpg.cyl)
mpgGrouping and Summarizing¶
This combines a couple functions that exist within Pandas to create summary tables.
Pandas has a wide range of summary statistics that you can apply to individual columns.
# Average city fuel efficiency
mpg.cty.mean()16.858974358974358# Standard deviation of highway fuel efficiency
mpg.hwy.std()5.9546434411664455Pandas also has a .groupby() method (which returns a generator) that groups categorical variables.
mpg.groupby("manufacturer")<pandas.core.groupby.generic.DataFrameGroupBy object at 0x1170c53d0>By itself, .groupby() doesn’t show anything. It needs to be combined with a summary statistic to create a summary table.
# Averages by manufacturer
# set `numeric_only=True` to avoid a warning
mpg.groupby("manufacturer").mean(numeric_only=True)Dropping Null Values¶
For many statistical modeling tasks, you need to drop rows that contain null values. Pandas lets you do this easily with .dropna().
# Drop any row that contains a null value in any column
mpg = mpg.dropna()
mpgYou can also drop null values from only a subset of columns.
# Drop any rows that contain null values in a subset of columns
mpg = mpg.dropna(subset=["model", "displ"])
mpgSampling¶
Many statistical methods, especially hypothesis tests, require to take a random sample of your overall data. Again, Pandas provides an easy way to do this with the .sample() method.
You can take a sample of rows from an entire dataframe.
# Get 5 random rows from mpg
mpg.sample(5)You can also get a sample of a specific column.
# Get 5 sample engine displacement values, as a series
mpg.displ.sample(5)86 4.6
201 2.7
228 1.8
138 4.0
35 3.5
Name: displ, dtype: float64You can also sample with replacement. (This is also called “bootstrap sampling.”) This makes it possible to have the same value in your sample twice.
mpg.displ.sample(5, replace=True)119 2.7
86 4.6
83 4.2
28 5.3
199 5.7
Name: displ, dtype: float64Pandas will also let you get a fraction of values instead of a set number in your sample.
# Get a random sample of one twentieth the size of the dataset
mpg.sample(frac=.05)There’s one more trick you can do with sampling. Sometimes you don’t need to get a smaller random sample: instead, you just want to reshuffle every row of the dataset. You can do this by setting frac to 1. In a way, you’re taking a random sample that is 100% of the size of the dataset! (But make sure you do this without replacement.)
mpg.displ.sample(frac=1)51 3.9
205 4.0
83 4.2
73 5.9
122 3.0
...
228 1.8
85 4.6
123 3.7
106 1.8
155 3.8
Name: displ, Length: 234, dtype: float64Pandas will remember the indices in your new Series, which means if you use this reordered sample it might put things back in order for you! To avoid this, you can reset the index and drop the old labels.
mpg.displ.sample(frac=1).reset_index(drop=True)0 4.7
1 3.1
2 2.5
3 5.3
4 2.5
...
229 2.4
230 4.0
231 4.6
232 2.8
233 2.0
Name: displ, Length: 234, dtype: float64