Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongsi...
Information theory and inference, taught together in this exciting textbook, lie at the heart of many important areas of modern technology - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics and cryptography. The book introduces theory in tandem with applications. Information theory is taught alongside practical communication systems such as arithmetic coding for data compression and sparse-graph codes for error-correction. Inference techniques, including message-passing algorithms, Monte Carlo methods and variational approximations, are developed alongside applications to clustering, convolutional codes, independent component analysis, and neural networks. Uniquely, the book covers state-of-the-art error-correcting codes, including low-density-parity-check codes, turbo codes, and digital fountain codes - the twenty-first-century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, the book is ideal for self-learning, and for undergraduate or graduate courses. It also provides an unparalleled entry point for professionals in areas as diverse as computational biology, financial engineering and machine learning.
Probabilities can be used in two ways.
1. Probabilities can describe frequencies of outcomes in random experiments
2. Probabilities can also be used, more generally, to describe degrees of belief in propositions that do not involve random variables
This more general use of probability to quantify beliefs is known as the Bayesian viewpoint. It is also known as the subjective interpretation of probability, since the probabilities depend on assumptions.
Advocates of a Bayesian approach to data modelling and pattern recognition do not view this subjectivity as a defect, since in their view,
you cannot do inference without making assumptions. (查看原文)
The details of the other possible outcomes and their probabilities are irrelevant.
All that matters is the probability of the outcome that actually
happened (here, that the ball drawn was black) given the dierent hypotheses.
We need only to know the likelihood, i.e., how the probability of the data
that happened varies with the hypothesis. This simple rule about inference is
known as the likelihood principle.
The likelihood principle: given a generative model for data d given
parameters [;\theta;], [;P(d|\theta);], and having observed a particular outcome
[;d_1;], all inferences and predictions should depend only on the function
[;P(d_1|\theta);].
In spite of the simplicity of this principle, many classical statistical methods
violate it. (查看原文)
1.刚从图书馆借到这本书，顺着书中的支持网站，发现作者把公开课视频也免费放到网上了，还可以直接下到英文原版电子版，这是什么精神~ ”A series of sixteen lectures covering the core of the book "Information Theory, Inference, and Learning Algorithms (Cambridge Un...
(展开)
这一段内容是讲概率论的两种解释：贝叶斯学派和非贝叶斯学派。 概率的常规解释为随机试验中的频率，但是贝叶斯学派的观点拓广了概率的应用范围，概率还可以表示对命题的信服（belief）程度，因为这样的概率基于一些假设（assumption），所以，对概率的解释增加了主观性。 Probabilities can be used in two ways. 1. Probabilities can describe frequencies of outcomes in random experiments 2. Probabilities can also be u...
Probabilities can be used in two ways.
1. Probabilities can describe frequencies of outcomes in random experiments
2. Probabilities can also be used, more generally, to describe degrees of belief in propositions that do not involve random variables
This more general use of probability to quantify beliefs is known as the Bayesian viewpoint. It is also known as the subjective interpretation of probability, since the probabilities depend on assumptions.
Advocates of a Bayesian approach to data modelling and pattern recognition do not view this subjectivity as a defect, since in their view,
you cannot do inference without making assumptions.引自 2.2 The meaning of probability
Shannon 1948: fundamental problem "reliable communication over an unreliable channel" received signal ~= transmitted signal + noise we would like "received message = transmitted message" solutions: 1. physical solutions; 2. system solutions source message->ENCODER->coded transmission->CHANNEL->received message->DECODER->predicted s
2013-10-25 01:56
Shannon 1948: fundamental problem "reliable communication over an unreliable channel"
received signal ~= transmitted signal + noise
we would like "received message = transmitted message"
solutions: 1. physical solutions; 2. system solutions
source message->ENCODER->coded transmission->CHANNEL->received message->DECODER->predicted s
这一段内容是讲概率论的两种解释：贝叶斯学派和非贝叶斯学派。 概率的常规解释为随机试验中的频率，但是贝叶斯学派的观点拓广了概率的应用范围，概率还可以表示对命题的信服（belief）程度，因为这样的概率基于一些假设（assumption），所以，对概率的解释增加了主观性。 Probabilities can be used in two ways. 1. Probabilities can describe frequencies of outcomes in random experiments 2. Probabilities can also be u...
Probabilities can be used in two ways.
1. Probabilities can describe frequencies of outcomes in random experiments
2. Probabilities can also be used, more generally, to describe degrees of belief in propositions that do not involve random variables
This more general use of probability to quantify beliefs is known as the Bayesian viewpoint. It is also known as the subjective interpretation of probability, since the probabilities depend on assumptions.
Advocates of a Bayesian approach to data modelling and pattern recognition do not view this subjectivity as a defect, since in their view,
you cannot do inference without making assumptions.引自 2.2 The meaning of probability
Shannon 1948: fundamental problem "reliable communication over an unreliable channel" received signal ~= transmitted signal + noise we would like "received message = transmitted message" solutions: 1. physical solutions; 2. system solutions source message->ENCODER->coded transmission->CHANNEL->received message->DECODER->predicted s
2013-10-25 01:56
Shannon 1948: fundamental problem "reliable communication over an unreliable channel"
received signal ~= transmitted signal + noise
we would like "received message = transmitted message"
solutions: 1. physical solutions; 2. system solutions
source message->ENCODER->coded transmission->CHANNEL->received message->DECODER->predicted s
两个计算熵比较有用的式子 1. Entropy is maximized if p is uniform: [;H(X) \leq log(|A_X|);] with equality iff [;p_i = 1/|A_X|;] for all i. 取等号的情形，经常能遇到，[;H(X) = log(|A_X|);]，[; |A_X|;]表示其中元素数量。 2. 这个公式有些长，直接上图片(P34) 这个思想就是逐步地揭示“真相”：先区分出是属于前一个{P1, ... , Pm}这个集合，还是属于后一个集合；然后对每个集合分别计算熵，当然，此时属于某一个...
2013-08-23 22:07
两个计算熵比较有用的式子
1.
Entropy is maximized if p is uniform:
[;H(X) \leq log(|A_X|);] with equality iff [;p_i = 1/|A_X|;] for all i.引自 2.5: Decomposability of the entropy
旅叶
高等教育出版社2006版
饮冰散人
高等教育出版社2006版
这一段内容是讲概率论的两种解释：贝叶斯学派和非贝叶斯学派。 概率的常规解释为随机试验中的频率，但是贝叶斯学派的观点拓广了概率的应用范围，概率还可以表示对命题的信服（belief）程度，因为这样的概率基于一些假设（assumption），所以，对概率的解释增加了主观性。 Probabilities can be used in two ways. 1. Probabilities can describe frequencies of outcomes in random experiments 2. Probabilities can also be u...
0 有用 NumineViget 2020-03-27
Shannon真的是我男神。很美妙的一套体系，日常查阅必备。有空可以深入读读，会对一些看似莫名其妙的 log 们有更深的体会。
0 有用 太常引 2009-11-18
有谁一起学习这本书吗？一起讨论吧QQ:63583981
0 有用 英子 2010-07-07
有点难，但是我觉得写的挺好的。
4 有用 pluskid 2011-04-25
读了一点，组会解散了，于是没有继续下去了，感觉这书讲得好 detail 啊。 组会在读的书之一。 水木 AI 版有人推荐，有电子版，有时间看一下。看章节标题似乎很不错的样子。
0 有用 ruby_eyes 2013-07-12
好书！只看了信息论部分，深入浅出
0 有用 NumineViget 2020-03-27
Shannon真的是我男神。很美妙的一套体系，日常查阅必备。有空可以深入读读，会对一些看似莫名其妙的 log 们有更深的体会。
0 有用 雪尘 2020-01-06
感觉有时间慢慢啃的话肯定能打开很多新世界大门
1 有用 水如歌 2019-06-16
机器学习领域中的 Feynman。
0 有用 MagnusXu 2019-03-07
作者赛高！由于水平有限，这本书的阅读体验并不轻松，甚至在某些部分很痛苦。它的讨论并不局限特定领域，而是融会贯通。断断续续花了好久才看完，收获颇丰。上一次有这种头脑爆炸的感觉，还是看《歌德 埃舍尔 巴赫》
0 有用 monica1302 2018-07-20
好书👍圈粉