Algebraic Geometry and Statistical Learning Theory

Sure to be influential, this book lays the foundations for the use of algebraic geometry in statistical learning theory. Many widely used statistical models and learning machines applied to information science have a parameter space that is singular: mixture models, neural networks, HMMs, Bayesian networks, and stochastic context-free grammars are major examples. Algebraic geom...

(展开全部)

Sure to be influential, this book lays the foundations for the use of algebraic geometry in statistical learning theory. Many widely used statistical models and learning machines applied to information science have a parameter space that is singular: mixture models, neural networks, HMMs, Bayesian networks, and stochastic context-free grammars are major examples. Algebraic geometry and singularity theory provide the necessary tools for studying such non-smooth models. Four main formulas are established: 1. the log likelihood function can be given a common standard form using resolution of singularities, even applied to more complex models; 2. the asymptotic behaviour of the marginal likelihood or 'the evidence' is derived based on zeta function theory; 3. new methods are derived to estimate the generalization errors in Bayes and Gibbs estimations from training errors; 4. the generalization errors of maximum likelihood and a posteriori methods are clarified by empirical process theory on algebraic varieties.

Half-title......Page 3
Series-title......Page 4
Title......Page 5
Copyright......Page 6
Contents......Page 7
Preface......Page 9
· · · · · · (更多)

Half-title......Page 3
Series-title......Page 4
Title......Page 5
Copyright......Page 6
Contents......Page 7
Preface......Page 9
1.1.1 Random samples......Page 11
1.1.2 Kullback–Leibler distance......Page 13
1.1.3 Fisher information matrix......Page 17
1.2.1 Singular models......Page 20
1.2.2 Density estimation......Page 23
1.2.3 Conditional probability density......Page 26
1.3 Statistical estimation methods......Page 28
1.3.1 Evidence......Page 29
1.3.2 Bayes and Gibbs estimations......Page 32
1.3.3 Maximum likelihood and a posteriori......Page 34
1.4.1 Standard form of log likelihood ratio function......Page 36
1.4.2 Evidence of singular model......Page 41
1.4.3 Bayes and Gibbs theory......Page 44
1.4.4 ML and MAP theory......Page 49
1.5 Overview of this book......Page 51
1.6 Probability theory......Page 52
2.1 Polynomials and analytic functions......Page 58
2.2 Algebraic set and analytic set......Page 60
2.3 Singularity......Page 63
2.4 Resolution of singularities......Page 68
2.5 Normal crossing singularities......Page 76
2.6 Manifold......Page 82
3.1 Ring and ideal......Page 87
3.2 Real algebraic set......Page 90
3.3 Singularities and dimension......Page 96
3.4 Real projective space......Page 97
3.5 Blow-up......Page 101
3.6.1 Simple cases......Page 109
3.6.2 A sample of a statistical model......Page 111
4.1 Schwartz distribution......Page 115
4.2 State density function......Page 121
4.3 Mellin transform......Page 126
4.4 Evaluation of singular integral......Page 128
4.5 Asymptotic expansion and b-function......Page 138
5.1 Convergence in law......Page 143
5.2 Function-valued analytic functions......Page 150
5.3 Empirical process......Page 154
5.4 Fluctuation of Gaussian processes......Page 164
6 Singular learning theory......Page 168
6.1 Standard form of likelihood ratio function......Page 170
6.2 Evidence and stochastic complexity......Page 178
6.3.1 Equations of states......Page 187
6.3.2 Basic lemmas......Page 193
6.3.3 Proof of the theorems......Page 206
6.4 Maximum likelihood and a posteriori......Page 213
7.1 Learning coefficient......Page 227
7.2 Three-layered neural networks......Page 237
7.3 Mixture models......Page 240
7.4 Bayesian network......Page 243
7.5 Hidden Markov model......Page 244
7.6 Singular learning process......Page 245
7.7 Bias and variance......Page 249
7.8 Non-analytic learning machines......Page 255
8.1 Universally optimal learning......Page 259
8.2 Generalized Bayes information criterion......Page 262
8.3 Widely applicable information criteria......Page 263
8.3.1 Experiments......Page 267
8.4.1 Optimal hypothesis test......Page 268
8.4.2 Example of singular hypothesis test......Page 271
8.5.1 Markov chain Monte Carlo......Page 274
Metropolis Algorithm......Page 275
8.5.2 Variational Bayes approximation......Page 277
8.6 From regular to singular......Page 284
Bibliography......Page 287
Index......Page 294
· · · · · · (收起)