Information Theory, Inference and Learning Algorithms

Probabilities can be used in two ways. 1. Probabilities can describe frequencies of outcomes in random experiments 2. Probabilities can also be used, more generally, to describe degrees of belief in propositions that do not involve random variables This more general use of probability to quantify beliefs is known as the Bayesian viewpoint. It is also known as the subjective interpretation of probability, since the probabilities depend on assumptions. Advocates of a Bayesian approach to data modelling and pattern recognition do not view this subjectivity as a defect, since in their view, you cannot do inference without making assumptions.

The details of the other possible outcomes and their probabilities are irrelevant. All that matters is the probability of the outcome that actually happened (here, that the ball drawn was black) given the dierent hypotheses. We need only to know the likelihood, i.e., how the probability of the data that happened varies with the hypothesis. This simple rule about inference is known as the likelihood principle. The likelihood principle: given a generative model for data d given parameters [;\theta;], [;P(d|\theta);], and having observed a particular outcome [;d_1;], all inferences and predictions should depend only on the function [;P(d_1|\theta);]. In spite of the simplicity of this principle, many classical statistical methods violate it.