Exploratory Data Analysis
CIS 241, Dr. Ladd
spacebar to go to the next slide, esc/menu to navigate
Look at the Data
John Tukey (1962)
John Tukey pioneered Exploratory Data Analysis starting in 1962 and again with a book in 1977.
Summary Statistics
Summary Statistics are focused on the location, variability, and distribution of data.
Location: What is the data’s “typical value”?
Let’s imagine a variable showing the heights of different dogs.
The dog’s heights (in mm) are: 600, 470, 170, 430, and 300
Mean is the sum of all values divided by the number of values.
AKA “average”
600+470+170+430+3005=394
We can calculate the mean easily in Python.
Median is the value such that half of the data lies above and below.
AKA “50th percentile”
600, 470, 170, 430, 300
Percentile is a value such that P percent of the data lies below.
AKA “quantile”
The 25th Percentile is the 1st Quartile.
600, 470, 170, 430, 300
The 75th Percentile is the 3rd Quartile.
600, 470, 170, 430, 300
The median is the 2nd Quartile!!
An outlier is a data value that’s different from most of the data.
AKA “extreme value”
A robust variable is not sensitive to extreme values.
Variability: Is the data tightly clustered or spread out?
The interquartile range is the difference between the 1st and 3rd quartiles.
A deviation is the difference between an actual value and an estimate of location (like the mean).
The variance is the sum of the squared deviations, divided by the number of values.
2062+762+(−224)2+362+(−94)25=21,704
The standard deviation is the square root of the variance.
It gives us a “standard” way of knowing what is normal, or what is extra large/extra small.
- Rottweilers are tall, and dachsunds are short—compared to the standard deviation from the mean.
- Outliers are often more than two standard deviations from the mean.
Now calculate the variance and standard deviations in Python.
Were these the results you expected?
Population vs. Sample
- Population: the full data that exists (i.e. all the dogs in the world)
- Sample: the actual data that was collected (i.e. our set of 5 dogs)
Population vs. Sample
When you have “N” data values:
The Entire Population: divide by N when calculating variance (like we did)
A Sample: divide by N-1 when calculating variance
Sample variance: 108,5204=27,130
Sample standard deviation: √27,130=164
Think of it as a “correction” when your data is only a sample. Pandas does this by default!
Neither the mean, variance, nor standard deviation are robust. They are all very sensitive to outliers!
Distributions: How many of each value are there?
Histograms show distributions based on frequency counts.
The histogram for miles per gallon highway in the mpg dataset.
The normal distribution has most values in the middle.
In a normal distribution, 95% of the values lie within 2 standard deviations of the mean.
Be careful: normal distributions are assumed for many statistical analyses!
Boxplots show distribution based on the median.
You can see how the box plot and the histogram are similar but different.
Correlation: Are two variables related?
Location, Variability, and Distribution are for one variable at a time (univariate analysis). Correlation is for two variables (bivariate analysis).
Scatter plots and correlation coefficients are used to determine correlation.
We’ll talk more about correlation in a couple weeks!
Resampling and Confidence Intervals
Resampling is simply drawing multiple random samples from observed data.
I can grab a sample of 5 observations. Then I can “resample” 5 more. Then 5 more, and so on and so on.
First proposed in the 1960s, resampling procedures weren’t practical until computing took off in the 1980s.
Resampling is an umbrella term. It can include:
- The Bootstrap, used to assess the reliability of an estimated statistic
- Permutation Tests, used as an alternative to parametric hypothesis tests
You can resample with or without replacement.
Replacement means an item is returned to the sample before the next draw (i.e. you could wind up with the same observation multiple times).
Using bootstrap sampling, we can estimate a confidence interval for any summary statistic.
- Remember: bootstrap sampling is sampling with replacement.
- The 90% confidence interval tells us where 90% of the data is likely to fall, if the data were random.
- Confidence intervals are used to gauge uncertainty.
Confidence intervals with Pandas
Work with a partner to calculate the confidence interval for the mean height in the dogs dataframe (remember to use your Pandas cheatsheet and past slideshows):
- Find the number of rows in the dataframe, assign to a variable
n_rows. - Create a
forloop through a list of 5000 numbers usingrange(5000).- In the loop, use the
sample()function wheren=your number of rows andreplace=True. - Get the
mean()of the height column of each sample, and assign it to a variable. - Use
append()to add your sample means to an empty list.
- In the loop, use the
- Turn the list in to a Pandas Series with
pd.Series(your_list). - Find the 95th percentile (quantile) and the 5th percentile (quantile) and assign each to a variable.
print()your results using f-strings.- The result should read: “The mean dog height in our data is _______, with a 90% confidence interval of _____ to _____.”
Confidence intervals with Pandas
n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")n_rows = dogs.shape[0] # First find the number of rows in the dataframe. # Then take a sample with replacement from your dataset, and # Calculate the mean each time. Do this 5000 times. bootstrap_samples = [] for i in range(5000): sample_mean = dogs.sample(n=n_rows, replace=True).height.mean() bootstrap_samples.append(sample_mean) # Put the results into a pandas Series object bootstrap_samples = pd.Series(bootstrap_samples) # Calculate the 95th percentile and the 5th percentile. top_percentile = bootstrap_samples.quantile(.95) bottom_percentile = bootstrap_samples.quantile(.05) # Print the results using nice f-strings. print(f"The mean dog height in our data is {dogs.height.mean():.3f}, with a 90% confidence interval of {bottom_percentile:.3f} to {top_percentile:.3f}.")
Exploratory Data Analysis CIS 241, Dr. Ladd spacebar to go to the next slide, esc /menu to navigate