So last time we talked about joint probabilities.
The probability of X and Y,
and we talked about joint information.
If there are different labels X_i and Y_j for these different possibilities,
we have the joint information with i and XY is just the ordinary information defined over
this joint quantity, X and Y.
And I defined something something I call mutual information.
So mutual information is just the information in X on its own,
the amount of information you get if you look at this variable X,
if it's a binary variable, like a coin flip, it could be up to a bit,
but could be anywhere between 0 and 1 bit of information.
So it's the sum of the individual informations of X and Y on their own, minus the joint information,
and this is defined as the mutual information, which we designate by having a little colon.
Now I argued, but not very convincingly I believe,
that this quantity measures the amount of information that I get about X if I look at Y,
it also measures the amount of information that I get about Y if I look at X.
And it also measures the information that X and Y in some sense hold in common.
So let's explore that notion a little further.
And we're going to do it in the following fashion.
So we're going to define a conditional probability.
The probability of Y given X.
So this is the probability that it's raining given that it's sunny.
Small, in most places, but actually not that small in Santa Fe.
So reasonably small.
Now, what is the probability that it's raining given that it's sunny?
A nice way of looking at that is we look at our little diagram before.
So here is the probability the whole set of events where it's sunny, here's the whole set
of events where it's raining.
And this is the set of events where it's raining and sunny.
So if we want to actually get the conditional probability that it's raining given that it's sunny,
then what we have to do is we have to look at the probability that it is raining and sunny,
divided by the probability that it's sunny.
So this is just an identity, it says that the probability that it's raining given that it's sunny
is the probability that it's raining and it's sunny divided by the probability that it's sunny.
This is conditional probability.
And we can also define condition information.
So w can look at the information in Y given that we know that the value of X is X_j.
Remember that we added this extra index to say X is now about sunniness or not sunniness,
j = 0 means not sunny, j = 1 means that it is sunny.
And what we can do is we can just take our ordinary formula for information,
we sum over our value j for the possible values