NEW COURSE
CS/CNS/EE 156 B
Learning Systems
Spring 2000
http://www.vision.caltech.edu/welling/class/LearningSystemsB.html
Instructor: Max Welling
welling@vision.caltech.edu
TA: Silvio Savarese
savarese@vision.caltech.edu
Place and time: 070 Moore, Wednesday/Friday, 9.00-10.30 AM
prerequisites: Ma2 or equivalent
Units: 9 (3-0-6)
Description:
This class will introduce some widely used latent variable models
in engineering. Among others we will discuss
PCA, ICA, Factor Analysis, K-means, Mixture of Gaussians, Generative
Topographic Mapping, Cluster Weighted Models, Mixture of Experts,
Kalman Filter, HMM, Helmholtz Machine, Boltzman Machine. It will
be shown that most of the learning schemes for these models can be understood
as versions of the Expectation Maximization (EM) algorithm, thus providing
a unified view.
Students are expected to implement a rudimentary MATLAB version for
some of the models and discuss this in class.
The final project will consist of a study of a recent paper and
the implementation a more involved model.
Homeworks and Policy:
- 4 homework problem sets (15% each)
- One final project (40%)
- Homeworks will be published on this web page as soon as available.
- No midterm, no final
- Two or more students can collaborate on any homework, but every
student is expected to hand in his/her own answers.
Downloads & Links (last
update March 31 - 4pm)
Syllabus (last update March
31 - 4pm)
- maximum likelihood
estimation ( demo1.m
, ginput2.m
, plotGauss.m
)
- Baysian learning
- complexity measures:
MLD.
-
Week 2 : EM learning and generalizations
classnotes
- EM update rules
- variational EM
- GEM
- example: learning gaussian
parameters from incomplete data
- principal component analysis
(PCA)
- probabilistic PCA
- factor analysis (FA)
- independent component
analysis (ICA)
- vector quantization through
K-means
- mixture of Gaussians
- mixture of factor
analysers
-
Week 5 : Supervised Learning: Function
Approximation
- generative topographic
mapping
- cluster-weighted modelling
- mixture of experts
-
Week 6 : Dynamical Models
- hidden markov models (HMM)
- Kalman filter
-
Week 7 : Approximate Inference
- Laplace approximation
- variationa approximation
- sampling methods:
importance sampling, rejection sampling, condensation,
Markov chain
Monte Carlo (MCMC), Gibbs sampling.
-
Week 8 : Belief Nets, Graphical Models
- Helmholtz machine
- Boltzman machine
- belief propagation
-
Week 9: Summary and Advanced Topics
- leftovers & summary
Literature:
-
Classnotes will be provided before every class.
-
C.M. Bishop, Neural Networks for Pattern Recognition.
-
B.D. Ripley, Pattern Recognition and Neural Networks.
-
K.V. Mardia, J.T. Kent and J.M. Bibby, Multivariate Analysis.
-
M.I. Jordan, Learning in Graphical Models.
-
T.M. Cover and J.A. Thomas, Elements of Information Theory.
-
A. Papoulis, Probability, Random Variables and Stochastic Processes
-
G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions.
California Institute of Technology,
Pasadena, CA 91125.