Joint Distributions · Visual Guide

Definition

From one variable to a pair

Instead of studying $X$ and $Y$ apart, a joint distribution treats the pair $(X,Y)$ as a single object, assigning a probability or density to every combination of values it can take.

$(X,Y): \Omega \to \mathbb{R}^2$

This matters whenever the two variables are not chosen at random with respect to each other. Height and weight, study hours and exam score, or the goals scored by two teams in the same match all carry a relationship that a joint distribution can capture.

Distinction

Discrete pairs vs continuous pairs

As with single random variables, a pair can be discrete, continuous, or one of each. This visual guide focuses on the two matching cases.

Type	Values	Examples
Discrete pair	countable combinations	goals scored by two teams in a match, likes and comments on a social media post
Continuous pair	any point in a region of the plane	height and weight, study hours and exam score

Symbol	Meaning
$X, Y$	two random variables defined on the same sample space
$(x,y)$	a specific pair of values that $X$ and $Y$ could take together
support of $(X,Y)$	the set of pairs $(x,y)$ the pair can actually take

Definition

Probability of a pair of values

For a discrete pair, the joint PMF gives the probability that $X$ equals $x$ and $Y$ equals $y$ at the same time.

$p_{X,Y}(x,y) = P(X=x,\ Y=y)$

It must satisfy the same two conditions as a single PMF, now over every pair of values.

1$p_{X,Y}(x,y) \ge 0$ for every pair $(x,y)$ in the support

2$\sum_x \sum_y p_{X,Y}(x,y) = 1$, all the probability mass adds up to one

Example

Goals scored by Barça and Madrid

Let $X$ be the number of goals Barça scores in a match (0, 1 or 2) and $Y$ whether Madrid scores at least once (0 for no, 1 for yes). Tracking many matches gives this joint PMF.

$X \backslash Y$	$y=0$	$y=1$
$x=0$	0.15	0.05
$x=1$	0.20	0.15
$x=2$	0.25	0.20

To be clear, this is not a table that reflects reality. In real life Barça would score more than 2 goals with fairly high probability.

Example

Reading off probabilities from the table

Probability of a 2-0 score for Barça:
$p_{X,Y}(2,0) = 0.25$

Probability of Barça scoring:
$P(X \ge 1) = p_{X,Y}(1,0) + p_{X,Y}(1,1) + p_{X,Y}(2,1) + p_{X,Y}(2,0) = 0.20 + 0.15 + 0.20 + 0.25 = 0.80$

Probability of Madrid not scoring:
$P(Y=0) = p_{X,Y}(0,0) + p_{X,Y}(1,0) + p_{X,Y}(2,0) = 0.15 + 0.20 + 0.25 = 0.60$

Adding up all six cells in the table gives exactly $1$, confirming this is a valid joint PMF.

Definition

Density over a region of the plane

For a continuous pair, probability comes from the volume under a surface $f_{X,Y}(x,y)$ over a region $A$ of the plane, not from the value of $f_{X,Y}$ at any single point.

$P\big((X,Y) \in A\big) = \iint_A f_{X,Y}(x,y)\,dx\,dy$

It must satisfy the same two conditions as a joint PMF, now stated for a density.

1$f_{X,Y}(x,y) \ge 0$ everywhere on the support

2the total volume under the surface equals 1, $\iint_{\mathbb{R}^2} f_{X,Y}(x,y)\,dx\,dy = 1$

Example

Uniform density on a triangle

Let $f_{X,Y}(x,y) = 2$ on the triangle where $0 < x < y < 1$, and 0 elsewhere.

Example

Verifying the joint PDF integrates to 1

We check that this triangle example is a valid joint PDF by integrating $f_{X,Y}$ over its whole support and confirming the result is exactly 1.

Set up the double integral: for each fixed $x$, $y$ ranges from $x$ up to 1.
$\displaystyle\iint_{0 < x < y < 1} 2\,dx\,dy = \int_0^1 \int_x^1 2\,dy\,dx$

Inner integral, over $y$:
$\displaystyle\int_x^1 2\,dy = 2(1-x)$

Outer integral, over $x$:
$\displaystyle\int_0^1 2(1-x)\,dx = \big[2x - x^2\big]_0^1 = 2 - 1 = 1$

The quick shortcut is the same thing geometrically: a constant height of 2 times the triangle's area, $2 \times \tfrac{1}{2} = 1$.

Symbol	Meaning
$p_{X,Y}(x,y)$	joint PMF, probability that $X=x$ and $Y=y$ together (discrete case)
$f_{X,Y}(x,y)$	joint PDF, density of $(X,Y)$ around the point $(x,y)$ (continuous case)

Definition

Recovering one variable from the pair

The marginal PMF of $X$ ignores $Y$ entirely. It is found by fixing a value of $x$ and adding up the joint probability across every possible value of $Y$.

$p_X(x) = \sum_y p_{X,Y}(x,y)$

$p_Y(y) = \sum_x p_{X,Y}(x,y)$

Geometrically, this means summing across a row of the joint table to get $p_X$, or down a column to get $p_Y$.

Example

Row and column totals

Adding row totals and column totals to the match table gives both marginal distributions at once.

$X \backslash Y$	$y=0$	$y=1$	$p_X(x)$
$x=0$	0.15	0.05	0.20
$x=1$	0.20	0.15	0.35
$x=2$	0.25	0.20	0.45
$p_Y(y)$	0.60	0.40	1.00

Madrid scores at least once about 40% of the time, and Barça scores 2 goals about 45% of the time, regardless of what the other variable does.

Definition

Integrating out the other variable

For a continuous pair, the sums become integrals over the other variable.

$f_X(x) = \displaystyle\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy, \quad f_Y(y) = \displaystyle\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx$

Example

Marginal densities on the triangle

Integrating the joint density from section 02 over the other variable gives the marginal density, the continuous analog of summing a row or column.

$f_X(x) = \displaystyle\int_x^1 2\,dy = 2(1-x), \quad f_Y(y) = \displaystyle\int_0^y 2\,dx = 2y$

Symbol	Meaning
$p_X(x)$, $p_Y(y)$	marginal PMFs, the distribution of one variable alone (discrete case)
$f_X(x)$, $f_Y(y)$	marginal PDFs, obtained by integrating out the other variable (continuous case)

Definition

Summing over a corner of the table

The joint CDF answers "what is the chance that $X$ is at most $x$ and $Y$ is at most $y$, both at once?"

$F_{X,Y}(x,y) = P(X \le x,\ Y \le y)$

For a discrete pair, this means summing the joint PMF over the lower left block of the table, every cell with a row and column at or below $(x,y)$.

$F_{X,Y}(x,y) = \sum_{x_i \le x} \sum_{y_j \le y} p_{X,Y}(x_i, y_j)$

Example

Reading a corner of the match table

Probability that Barça scores at most 1 goal while Madrid fails to score:
$F_{X,Y}(1,0) = P(X \le 1,\ Y \le 0) = \displaystyle\sum_{x_i \le 1} p_{X,Y}(x_i, 0)$

Adding the two cells with $y=0$:
$F_{X,Y}(1,0) = p_{X,Y}(0,0) + p_{X,Y}(1,0) = 0.15 + 0.20 = 0.35$

That's a 35% chance across all matches in the table.

Definition

Integrating over a corner of the plane

For a continuous pair, the sum becomes a double integral over the same lower-left region of the plane.

$F_{X,Y}(x,y) = \displaystyle\int_{-\infty}^{x}\int_{-\infty}^{y} f_{X,Y}(s,t)\,dt\,ds$

Example

Reading a corner of the triangle density

We compute $F_{X,Y}(0.5, 0.8) = P(X \le 0.5,\ Y \le 0.8)$, the probability that $(X,Y)$ from the triangle density falls in the shaded corner shown below.

Example

Computing the corner probability

Set up the double integral: since $0.5 \le 0.8$, integrate $x$ from 0 to 0.5 and, for each $x$, $y$ from $x$ to 0.8.
$F_{X,Y}(0.5,0.8) = \displaystyle\int_0^{0.5}\int_x^{0.8} 2\,dy\,dx$

Inner integral, over $y$:
$\displaystyle\int_x^{0.8} 2\,dy = 2(0.8-x)$

Outer integral, over $x$:
$\displaystyle\int_0^{0.5} 2(0.8-x)\,dx = \big[1.6x - x^2\big]_0^{0.5} = 0.8 - 0.25 = 0.55$

There is a 55% chance that $(X,Y)$ falls in this lower-left corner of the plane.

Symbol	Meaning
$F_{X,Y}(x,y)$	joint CDF, probability that $X \le x$ and $Y \le y$ together

Definition

When the joint splits into a product

$X$ and $Y$ are independent when knowing one gives no information about the other. This holds exactly when the joint distribution factors into the two marginals.

Discrete case

$p_{X,Y}(x,y) = p_X(x)\,p_Y(y) \quad \text{for every } x,y$

The continuous version replaces the PMFs with PDFs, and the same factoring works for the joint CDF.

Continuous case

$f_{X,Y}(x,y) = f_X(x)\,f_Y(y), \quad F_{X,Y}(x,y) = F_X(x)\,F_Y(y)$

Example

Checking the match table

Independence requires $p_{X,Y}(0,0) = p_X(0)\,p_Y(0)$ for every cell, so it is enough to check one cell that fails.

$p_X(0)\,p_Y(0) = 0.20 \times 0.60 = 0.12$.

But the table shows $p_{X,Y}(0,0) = 0.15 \neq 0.12$, so $X$ and $Y$ are not independent. Knowing whether Madrid scores changes what we expect about how many goals Barça gets.

Symbol	Meaning
$X \perp Y$	shorthand for "$X$ and $Y$ are independent"

Definition

The distribution of Y once X is known

Once we learn that $X = x$, the distribution of $Y$ can be updated. The conditional PMF rescales the row for $x$ so it adds up to 1 again.

Discrete case

$p_{Y|X}(y \mid x) = \dfrac{p_{X,Y}(x,y)}{p_X(x)}, \quad p_X(x) > 0$

For a continuous pair, the same idea rescales the joint density along the slice $X=x$.

Continuous case

$f_{Y|X}(y \mid x) = \dfrac{f_{X,Y}(x,y)}{f_X(x)}, \quad f_X(x) > 0$

When $X$ and $Y$ are independent, this simplifies to $p_{Y|X}(y \mid x) = p_Y(y)$: learning $X$ changes nothing about $Y$.

Interactive

Pick a goal count for Barça, see Madrid's odds shift

Consider the table from section 2 where $x$ = Barça goals and $y$ = Madrid goals.

$X \backslash Y$	$y=0$	$y=1$
$x=0$	0.15	0.05
$x=1$	0.20	0.15
$x=2$	0.25	0.20

Barça goals =

Symbol	Meaning
$p_{Y\|X}(y \mid x)$	conditional PMF of $Y$ given that $X = x$

Definition

Do X and Y move together?

Covariance measures whether $X$ and $Y$ tend to be above their means at the same time, or in opposite directions.

Discrete and continuous cases

$\text{Cov}(X,Y) = E\big[(X-\mu_X)(Y-\mu_Y)\big] = E[XY] - E[X]E[Y]$

The formula is the same either way; only how $E[\cdot]$ is computed changes, a sum over the joint PMF or an integral over the joint PDF.

Depending on the sign of the covariance:

+positive: $X$ above its mean tends to come with $Y$ above its mean

−negative: they tend to move in opposite directions

0no linear relationship. Independence forces this, but a zero covariance does not force independence

Example

Covariance of Barça's goals and Madrid scoring

From the marginals, $E[X] = 0(0.20) + 1(0.35) + 2(0.45) = 1.25$ and $E[Y] = 0(0.60) + 1(0.40) = 0.40$.

$E[XY]$ only picks up cells where both $x$ and $y$ are nonzero: $1(1)(0.15) + 2(1)(0.20) = 0.15 + 0.40 = 0.55$.

$\text{Cov}(X,Y) = 0.55 - (1.25)(0.40) = 0.55 - 0.50 = 0.05$.

A positive covariance confirms what the table already suggested: matches where Barça scores more also tend to be matches where Madrid gets on the scoresheet.

Definition

Variance of a Sum

The variance of a sum is not just the sum of the variances. It also includes twice the covariance, which is the whole reason covariance matters in practice.

Discrete and continuous cases

$\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y) + 2\,\text{Cov}(X,Y)$

If $X$ and $Y$ are independent, $\text{Cov}(X,Y) = 0$ and the formula reduces to simple addition of variances.

Symbol	Meaning
$\text{Cov}(X,Y)$	covariance, the average product of deviations from the two means

Definition

A scale-free version of covariance

Covariance depends on the units of $X$ and $Y$, which makes its size hard to interpret. Dividing by both standard deviations fixes that.

Discrete and continuous cases

$\rho_{X,Y} = \dfrac{\text{Cov}(X,Y)}{\sigma_X\,\sigma_Y}$

The correlation coefficient always falls between $-1$ and $1$, and it does not change if either variable is rescaled or shifted.

$-1 \le \rho_{X,Y} \le 1$

Interactive

Watch the scatter as correlation changes

Drag the slider to change $\rho$ and see how the point cloud tilts, from a negative slope through no pattern to a tight positive line.

ρ 0.50

Simulated points

Symbol	Meaning
$\rho_{X,Y}$	correlation coefficient, covariance rescaled to fall between $-1$ and $1$
$\sigma_X$, $\sigma_Y$	standard deviations of $X$ and $Y$

Joint Distributions of Two Random Variables

Joint Distributions: Overview

Joint PMF and PDF

Marginal Distributions

Joint Cumulative Distribution Function (CDF)

Independence of Two Random Variables

Conditional Distributions

Covariance

Correlation Coefficient