Descriptive Statistics

Looking back: Exploratory Data Analysis¶

• First, we dove deep into data visualization.
• Then, we’ve learned how to wrangle and clean our data.

Now we’ll turn to describing our data - as well as the foundations of statistics.

Goals of this lecture¶

There are many ways to describe a distribution.

Here we will discuss:

• Measures of central tendency: what is the typical value in this distribution?
• Measures of variability: how much do the values differ from each other?
• Measures of skewness: how strong is the asymmetry of the distribution?
• Measures of curvature: what is the intensity of extreme values?
• Measurement of the relationship between distributions using linear, rank correlations.
• Measurement of the relationship between qualitative variables using contingency.

Importing relevant libraries¶

Measurement scales in statistics¶

Measurement scales determine what mathematical and statistical operations can be performed on data. There are four basic types of scales:

1. Nominal scale

• Data is used only for naming or categorizing.
• The order between values cannot be determined.
• Possible operations: count, mode, frequency analysis.

Examples:

• Pokémon type (type_1): “fire”, ‘water’, “grass”, etc.
• Species, gender, colors.

1. Ordinal scale

• Data can be ordered, but the distances between them are not known.
• Possible operations: median, quantiles, rank tests (e.g. Spearman).

Examples:

• Strength level: “low,” “medium,” “high.”
• Quality ratings: “weak,” “good,” “very good.”

3. Interval scale

• Data are numerical, with equal intervals, but no absolute zero.
• Differences, mean, standard deviation can be calculated.
• No sense for quotients (e.g., “twice as much”).

Examples:

• Temperature in °C (but not in Kelvin!). Why? No absolute zero - zero does not mean the absence of a characteristic, it is just a conventional reference point. 0°C does not mean no temperature; 20°C is not 2 x 10°C.
• Year in the calendar (e.g., 1990). Why. Year 0 does not mean the beginning of time; 2000 is not 2 x 1000.
• Time in the hour system (e.g., 1 p.m.). Why. 0:00 is not the absence of time, but a fixed reference point.

4. Ratio scale

• Numerical data, have absolute zero.
• All mathematical operations can be performed, including division.

Not all numerical (numerical) data are on the quotient scale! Example: temperature in degrees Celsius is not on the quotient scale, because 0°C does not mean no temperature. In contrast, temperature in Kelvin (K) already does, because 0 K is the absolute absence of heat energy.

Examples:

• Height, weight, number of Pokémon’s attack points (attack), HP, speed.

Table: Measurement Scales in Statistics¶

Conclusion: The type of scale affects the choice of statistical methods - for example, the Pearson correlation test requires quotient or interval data, while the Chi² test requires nominal data.

Central Tendency¶

Central Tendency refers to the “typical value” in a distribution.

Many ways to measure what’s “typical”.

• The mean.
• The median.
• The mode.

Why is this useful?¶

• A dataset can contain lots of observations.
  • E.g., $N = 5000$ survey responses about height.
• One way to “describe” this distribution is to visualize it.
• But it’s also helpful to reduce that distribution to a single number.

This is necessarily a simplification of our dataset!

The mean¶

The arithmetic mean is defined as the sum of all values in a distribution, divided by the number of observations in that distribution.

numpy.mean¶

The numpy package has a function to calculate the mean on a list or numpy.ndarray.

Calculating the mean of a pandas column¶

If we’re working with a DataFrame, we can calculate the mean of specific columns.

Check-in¶

How would you calculate the mean of the gdp_cap column?

Solution¶

This tells us the average gdp_cap of countries in this dataset across all years measured.

The mean and skew¶

Skew means there are values elongating one of the “tails” of a distribution.

Of the measures of central tendency, the mean is most affected by the direction of skew.

• How would you describe the skew below?
• Do you think the mean would be higher or lower than the median?

Check-in¶

Could you calculate the mean of the continent column? Why or why not?

Solution¶

• You cannot calculate the mean of continent, which is a categorical variable.
• The mean can only be calculated for continuous variables.

Check-in¶

• Subtract each observation in numbers from the mean of this list.
• Then, calculate the sum of these deviations from the mean.

What is their sum?

Solution¶

• The sum of deviations from the mean is equal to 0.
• This will be relevant when we discuss standard deviation.

Interim summary¶

• The mean is one of the most common measures of central tendency.
• It can only be used for continuous interval/ratio data.
• The sum of deviations from the mean is equal to 0.
• The mean is most affected by skew and outliers.

The median¶

The median is calculated by sorting all values from least to greatest, then finding the value in the middle.

If there is an even number of values, you take the mean of the middle two values.

Comparing median and mean¶

The median is less affected by the direction of skew.

Check-in¶

Could you calculate the median of the continent column? Why or why not?

Solution¶

• You cannot calculate the median of continent, which is a categorical variable.
• The median can only be calculated for ordinal (ranked) or interval/ratio variables.

The mode¶

The mode is the most frequent value in a dataset.

Unlike the median or mean, the mode can be used with categorical data.

mode() returns multiple values?¶

• If multiple values tie for the most frequent, mode() will return all of them.
• This is because technically, a distribution can have multiple modes!

Central tendency: a summary¶

Quantiles¶

Quantiles are descriptive statistics that divide an ordered dataset into equal parts. The most common quantiles are:

• Median (quantile of order 0.5),
• Quartiles (divide data into 4 parts),
• Deciles (divide data into 10 parts),
• Percentiles (divide data into 100 parts).

Definition¶

A quantile of order $q \in (0,1)$ is a value $x_q$ such that:

In other words: $q \cdot 100\%$ of the values in the dataset are less than or equal to $x_q$.

Formula (for an ordered dataset)¶

For a dataset $x_1, x_2, \ldots, x_n$ sorted in ascending order, the quantile of order $q$ is calculated as:

1. Compute the positional index:

2. If $i$ is an integer, the quantile is $x_i$.

3. If $i$ is not an integer, interpolate linearly between the neighboring values:

Note: In practice, different methods for calculating quantiles are used — libraries like NumPy or Pandas offer various modes (e.g., method='linear', method='midpoint').

Example - Manual Calculation¶

For the dataset:
$[3, 7, 8, 5, 12, 14, 21, 13, 18]$

1. Sort the data in ascending order:

$[3, 5, 7, 8, 12, 13, 14, 18, 21]$

2. Median (quantile of order 0.5):

The number of elements $n = 9$, the middle element is the 5th value:

$\text{Median} = x_5 = 12$

3. First quartile (Q1, quantile of order 0.25):

$i = 0.25 \cdot (9 + 1) = 2.5$

Interpolate between $x_2 = 5$ and $x_3 = 7$:

$Q_1 = 5 + 0.5 \cdot (7 - 5) = 6$

4. Third quartile (Q3, quantile of order 0.75):

$i = 0.75 \cdot 10 = 7.5$

Interpolate between $x_7 = 14$ and $x_8 = 18$:

$Q_3 = 14 + 0.5 \cdot (18 - 14) = 16$

Deciles¶

Deciles divide data into 10 equal parts. For example:

• D1 is the 10th percentile (quantile of order 0.1),
• D5 is the median (0.5),
• D9 is the 90th percentile (0.9).

The formula is the same as for general quantiles, using the appropriate $q$. For example, for D3:

$q = \frac{3}{10} = 0.3$

Percentiles¶

Percentiles divide data into 100 equal parts. For example:

• P25 = Q1,
• P50 = median,
• P75 = Q3,
• P90 is the value below which 90% of the data lies.

Percentiles help us better understand the data distribution — for example, in standardized tests, results are often expressed as percentiles (e.g., “85th percentile” means someone scored better than 85% of the population).

Quantiles - Summary¶

Example - we compute quantiles¶

Your turn!¶

Looking at the aforementioned quantile results and the box plot, try to interpret these measures.

Variability¶

Variability (or dispersion) refers to the degree to which values in a distribution are spread out, i.e., different from each other.

The mean hides variance¶

Both distributions have the same mean, but different standard deviations.

Capturing variability¶

There are at least three major approaches to quantifying variability:

• Range: the difference between the maximum and minimum value.
• Interquartile range (IQR): The range of the middle 50% of data.
• Standard deviation: the typical amount that scores deviate from the mean.

Range¶

Range is the difference between the maximum and minimum value.

Intuitive, but only considers two values in the entire distribution.

IQR¶

Interquartile range (IQR) is the difference between the value at the 75% percentile and the value at the 25% percentile.

Focuses on middle 50%, but still only considers two values.

Standard deviation¶

Standard deviation (SD) measures the typical amount that scores in a distribution deviate from the mean.

Things to keep in mind:

• SD is the square root of the variance.
• There are actually two measures of SD:
  • Population SD: when you’re measuring the entire population of interest (very rare).
  • Sample SD: when you’re measuring a sample (the typical case); we’ll focus on this one.

Sample SD¶

The formula for sample standard deviation of $X$ is as follows:

$\Large s = \sqrt{\frac{\sum{(X_i - \bar{X})^2}}{n - 1}}$

• $n$: number of observations.
• $\bar{X}$: mean of $X$.
• $X_i - \bar{X}$: difference between a particular value of $X$ and mean.
• $\sum$: sum of all these squared deviations.

Check-in¶

The formula involves summing the squared deviations of each value in $X$ with the mean of $X$. Why do you think we square these deviations first?

$\Large\sum{(X_i - \bar{X})^2}$

Solution¶

If you simply summed the raw deviations from the mean, you’d get 0 (this is part of the definition of the mean).

SD, explained¶

$\Large s = \sqrt{\frac{\sum{(X_i - \bar{X})^2}}{n - 1}}$

• First, calculate sum of squared deviations.
  • What is total squared deviation from the mean?
• Then, divide by n - 1: normalize to number of observations.
  • What is average squared deviation from the mean?
• Finally, take the square root:
  • What is average deviation from the mean?

Standard deviation represents the typical or “average” deviation from the mean.

Calculating SD in pandas¶

Watching out for np.std¶

• By default, numpy.std will calculate the population standard deviation!
• Must modify the ddof parameter to calculate sample standard deviation.

This is a very common mistake.

Coefficient of variation (CV).¶

• The coefficient of variation (CV) is equal to the standard deviation divided by the mean.
• It is also known as “relative standard deviation.”

Measures of the shape of the distribution¶

Now we will look at measures of the shape of the distribution. There are two statistical measures that can tell us about the shape of a distribution. These are skewness and kurtosis. These measures can be used to tell us about the shape of the distribution of a data set.

Skewness¶

• Skewness is a measure of the symmetry of a distribution, or more precisely, the lack of symmetry.
• It is used to determine the lack of symmetry with respect to the mean of a data set.
• It is a characteristic of deviation from the mean.
• It is used to indicate the shape of a data distribution.

Skewness is a measure of the asymmetry of the distribution of data relative to the mean. It tells us whether the data are more “stretched” to one side.

Interpretation:

• Skewness > 0 - right-sided (positive): long tail on the right (larger values are more dispersed)
• Skewness < 0 - left (negative): long tail on the left (smaller values are more dispersed)
• Skewness ≈ 0 - symmetric distribution (e.g., normal distribution)

Formula (for the sample):

where: - $n$ - number of observations - $\bar{x}$ - sample mean - $s$ - standard deviation of the sample

Negative skewness¶

• In this case, the data is skewed or shifted to the left.
• By skewed to the left, we mean that the left tail is long relative to the right tail.
• The data values may extend further to the left, but are concentrated on the right side.
• So we are dealing with a long tail, and the distortion is caused by very small values that pull the mean down and it is smaller than the median.
• In this case, we have Mean < Median < Mode.

Zero skewness¶

• This means that the data set is symmetric.
• A dataset is symmetric if it looks the same to the left and right of the center point.
• A dataset is bell-shaped or symmetric.
• A perfectly symmetrical data set will have a skewness of zero.
• So a normal distribution that is perfectly symmetric has a skewness of 0.
• In this case, we have Mean = Median = Mode.

Positive skewness¶

• The data set is skewed or shifted to the right.
• By skewed to the right, we mean that the right tail is long relative to the left tail.
• The data values are concentrated on the right side.
• There is a long tail on the right side, which is caused by very large values that pull the mean up and it is larger than the median.
• So we have Mean > Median > Mode.

Your turn.¶

Try to interpret the above mentioned measures and calculate sample statistics for several Types of Pokémons.

Kurtosis¶

Contrary to what some textbooks claim, kurtosis does not measure the “flatness” or “peakedness” of a distribution.

Kurtosis depends on the intensity of extreme values, so it measures what happens in the “tails” of the distribution, and the shape of the “peak” is irrelevant!

Sample Kurtosis:¶

Reference Range for Kurtosis¶

• The reference standard is the normal distribution, which has a kurtosis of 3.
• Often, instead of kurtosis, excess kurtosis is presented, where excess kurtosis is simply kurtosis - 3.

Mesokurtic Curve¶

• A normal distribution has a kurtosis of exactly 3 (excess kurtosis of exactly 0).
• Any distribution with kurtosis ≈3 (excess kurtosis ≈ 0) is called mesokurtic.

Platykurtic Curve¶

• A distribution with kurtosis < 3 (excess kurtosis < 0) is called platykurtic.
• Compared to the normal distribution, its central peak is lower and wider, and its tails are shorter and thinner.

Leptokurtic Curve¶

• A distribution with kurtosis > 3 (excess kurtosis > 0) is called leptokurtic.
• Compared to the normal distribution, its central peak is higher and sharper, and its tails are longer and thicker.

Summary statistics¶

A great tool for creating elegant summaries of descriptive statistics in Markdown format (ideal for Jupyter Notebooks) is pandas, especially in combination with the .describe() function and tabulate.

Example with pandas + tabulate (nice table in Markdown):

To make a summary table cross-sectionally (that is, by group), you need to use the groupby() method on DataFrame, and then, for example, describe() or your own aggregation function.

Suppose you want to group data by “Type 1” column (i.e., for example, Pokémon type: Fire, Water, etc.), and then summarize quantitative variables (mean, variance, min, max, etc.).

Your turn!¶

Try to interpret some of the mentioned statistics.

Detecting (potential) outliers with z-scores¶

Defining and detecting outliers is notoriously difficult.

• Sometimes, an observation in a histogram clearly looks like an outlier.
• But how do we quantify this?

What is a z-score?¶

A z-score is a standardized measure of how many any given point deviates from the mean:

$Z = \frac{X - \mu}{\sigma}$

This is useful because it allows us to quantify the standardized distance between some observation and the mean.

Calculating z-scores¶

Check-in¶

Can anyone deduce why a z-score would be useful for defining outliers?

Z-scores and outliers¶

Outliers are data points that differ significantly from other points in a distribution.

• A z-score gives us a standardized measure of how different a given value is from the rest of the distribution.
• We can define thresholds, e.g.:

$\Large z ≥ |3|$

Testing our operationalization¶

Z-scoring our distribution¶

What is the z-score of the values way off to the right?

Removing outliers¶

Check-in¶

Can anyone think of challenges or problems with this approach to detecting outliers?

Caveats, complexity, and context¶

• There is not a single unifying definition of what an outlier is.
• Depending on the shape of the distribution, removing observations where $z > |3|$ might be removing important data.
  • E.g., in a skewed distribution, we might just be “cutting off the tail”.
• Even if the values do seem like outliers, there’s a philosophical question of why and what that means.
  • Are those values “random”? What’s the underlying *data-generating process?

This is why statistics is not a mere matter of applying formulas. Context always matters!

Describing bivariate data with correlations¶

• So far, we’ve been focusing on univariate data: a single distribution.
• What if we want to describe how two distributions relate to each other?
  • For today, we’ll focus on continuous distributions.

Bivariate relationships: height¶

• A classic example of continuous bivariate data is the height of a parent and child.
• These data were famously collected by Karl Pearson.

Plotting Pearson’s height data¶

Introducing linear correlations¶

A correlation coefficient is a number between $[–1, 1]$ that describes the relationship between a pair of variables.

Specifically, Pearson’s correlation coefficient (or Pearson’s $r$) describes a (presumed) linear relationship.

Two key properties:

• Sign: whether a relationship is positive (+) or negative (–).
• Magnitude: the strength of the linear relationship.

Where:

• $r$ - Pearson correlation coefficient
• $x_i$, $y_i$ - values of the variables
• $\bar{x}$, $\bar{y}$ - arithmetic means
• $n$ - number of observations

Pearson’s correlation coefficient measures the strength and direction of the linear relationship between two continuous variables. Its value ranges from -1 to 1:

• 1 → perfect positive linear correlation
• 0 → no linear correlation
• -1 → perfect negative linear correlation

This coefficient does not tell about nonlinear correlations and is sensitive to outliers.

Calculating Pearson’s $r$ with scipy¶

scipy.stats has a function called pearsonr, which will calculate this relationship for you.

Returns two numbers:

• $r$: the correlation coefficent.
• $p$: the p-value of this correlation coefficient, i.e., whether it’s significantly different from 0.

Check-in¶

Using scipy.stats.pearsonr (here, ss.pearsonr), calculate Pearson’s $r$ for the relationship between the Attack and Defense of Pokemon.

• Is this relationship positive or negative?
• How strong is this relationship?

Solution¶

Check-in¶

Pearson’r $r$ measures the linear correlation between two variables. Can anyone think of potential limitations to this approach?

Limitations of Pearson’s $r$¶

• Pearson’s $r$ presumes a linear relationship and tries to quantify its strength and direction.
• But many relationships are non-linear!
• Unless we visualize our data, relying only on Pearson’r $r$ could mislead us.

Non-linear data where $r = 0$¶

When $r$ is invariant to the real relationship¶

All these datasets have roughly the same correlation coefficient.

Rank Correlations¶

Rank correlations are measures of the strength and direction of a monotonic (increasing or decreasing) relationship between two variables. Instead of numerical values, they use ranks, i.e., positions in an ordered set.

They are less sensitive to outliers and do not require linearity (unlike Pearson’s correlation).

Types of Rank Correlations¶

1. $ρ$ (rho) Spearman’s

• Based on the ranks of the data.
• Value: from –1 to 1.
• Works well for monotonic but non-linear relationships.

Where:

• $d_i$ – differences between the ranks of observations,
• $n$ – number of observations.

2. $τ$ (tau) Kendall’s

• Measures the number of concordant vs. discordant pairs.
• More conservative than Spearman’s – often yields smaller values.
• Also ranges from –1 to 1.

Where:

• $τ$ — Kendall’s correlation coefficient,
• $C$ — number of concordant pairs,
• $D$ — number of discordant pairs,
• $n$ — number of observations,
• $\frac{1}{2}n(n - 1)$ — total number of possible pairs of observations.

What are concordant and discordant pairs?

• Concordant pair: if $x_i$ < $x_j$ and $y_i$ < $y_j$, or $x_i$ > $x_j$ and $y_i$ > $y_j$.
• Discordant pair: if $x_i$ < $x_j$ and $y_i$ > $y_j$, or $x_i$ > $x_j$ and $y_i$ < $y_j$.

When to use rank correlations?¶

• When the data are not normally distributed.
• When you suspect a non-linear but monotonic relationship.
• When you have rank correlations, such as grades, ranking, preference level.

Interpretation of correlation values¶

Comparison of Correlation Coefficients¶

Brief summary:

• Pearson - best when the data are normal and the relationship is linear.
• Spearman - works better for non-linear monotonic relationships.
• Kendall - more conservative, often used in social research, less sensitive to small changes in data.

Your Turn¶

For the Pokemon dataset, find the pairs of variables that are most appropriate for using one of the quantitative correlation measures. Calculate them, then visualize them.

Correlation of Qualitative Variables¶

A categorical variable is one that takes descriptive values that represent categories—e.g. Pokémon type (Fire, Water, Grass), gender, status (Legendary vs. Normal), etc.

Such variables cannot be analyzed directly using correlation methods for numbers (Pearson, Spearman, Kendall). Other techniques are used instead.

Contingency Table¶

A contingency table is a special cross-tabulation table that shows the frequency (i.e., the number of cases) for all possible combinations of two categorical variables.

It is a fundamental tool for analyzing relationships between qualitative features.

Chi-Square Test of Independence¶

The Chi-Square test checks whether there is a statistically significant relationship between two categorical variables.

Concept:

We compare:

• observed values (from the contingency table),
• with expected values, assuming the variables are independent.

Where:

• $O_{ij}$ – observed count in cell ($i$, $j$),
• $E_{ij}$ – expected count in cell ($i$, $j$), assuming independence.

Example: Calculating Expected Values and Chi-Square Statistic in Python¶

Here’s how you can calculate the expected values and Chi-Square statistic (χ²) step by step using Python.

Step 1: Create the Observed Contingency Table¶

We will use the Pokémon example:

Step 2: Calculate Expected Values The expected values are calculated using the formula:

You can calculate this manually or use scipy.stats.chi2_contingency, which automatically computes the expected values.

Step 3: Calculate the Chi-Square Statistic The Chi-Square statistic is calculated using the formula:

This is done automatically by scipy.stats.chi2_contingency, but you can also calculate it manually:

Step 4: Interpret the Results The chi2_contingency function also returns:

p-value: The probability of observing the data if the null hypothesis (independence) is true. Degrees of Freedom (dof): Calculated as (rows - 1) * (columns - 1).

Interpretation of the Chi-Square Test Result:

Qualitative Correlations¶

Cramér’s V¶

Cramér’s V is a measure of the strength of association between two categorical variables. It is based on the Chi-Square test but scaled to a range of 0–1, making it easier to interpret the strength of the relationship.

Where:

• $\chi^2$ – Chi-Square test statistic,
• $n$ – number of observations,
• $k$ – the smaller number of categories (rows/columns) in the contingency table.

Phi Coefficient ($φ$)¶

Application:

• Both variables must be dichotomous (e.g., Yes/No, 0/1), meaning the table must have the smallest size of 2×2.
• Ideal for analyzing relationships like gender vs purchase, type vs legendary.

Where:

• $\chi^2$ – Chi-Square test statistic for a 2×2 table,
• $n$ – number of observations.

Tschuprow’s T¶

Tschuprow’s T is a measure of association similar to Cramér’s V, but it has a different scale. It is mainly used when the number of categories in the two variables differs. This is a more advanced measure applicable to a broader range of contingency tables.

Where:

• $\chi^2$ – Chi-Square test statistic,
• $n$ – number of observations,
• $k$ – the smaller number of categories (rows or columns) in the contingency table.

Application: Tschuprow’s T is useful when dealing with contingency tables with varying numbers of categories in rows and columns.

Summary - Qualitative Correlations¶

Example¶

Let’s investigate whether the Pokémon’s type (type_1) is affected by whether the Pokémon is legendary.

We’ll use the scipy library.

This library already has built-in functions for calculating various qualitative correlation measures.

Your turn¶

What visualization would be most appropriate for presenting a quantitative, ranked, and qualitative relationship?

Try to think about which pairs of variables could have which type of analysis based on the Pokemon data.

Heatmaps for qualitative correlations¶

Summary¶

There are many ways to describe our data:

• Measure central tendency.

• Measure its variability; skewness and kurtosis.

• Measure what correlations our data have.

All of these are useful and all of them are also exploratory data analysis (EDA).