This class will introduce some widely used latent variable models in engineering. Among others we will discuss
PCA, ICA, Factor Analysis, K-means, Mixture of Gaussians, Generative Topographic Mapping, Cluster Weighted Models,  Mixture of Experts, Kalman Filter,  HMM, Helmholtz Machine, Boltzman Machine. It will be shown that most of the learning schemes for these models can be understood as versions of the Expectation Maximization (EM) algorithm, thus providing a unified view.

Students are expected to implement a rudimentary MATLAB version for some of the models and discuss this in class.
The final project will consist of a study of a recent paper and the implementation a more involved model.

- 4 homework problem sets (15% each)
- One final project (40%)
- Homeworks will be published on this web page as soon as available.
- No midterm, no final
- Two or more students can collaborate on any homework, but every student is expected to hand in his/her own answers.

• Matlab tutorial
• Matlab Data Visualization Tool: PhiVis
•  ICA software package - link 1
•  ICA software package - link 2
•  Kalman filtering book by Peter Maybeck
• Data sets
• Handwritten digits
• Prof. Michael I. Jordan's webpage
• Gatsby Computational Neuroscience Unit
• Laboratory of Computer and Information Science
• NCRG - Aston University
• Microsoft Research (Bishop)
• demo_gibbs.m 
• demo_mcmc.m

• Week 1  : General Statistical Modeling classnotes

• Week 2  : EM learning and generalizations classnotes

• Week 3  :  Clustering classnotes

• Week 4  : Linear Models classnotes  -  EMICA

• Week 5  :  Supervised Learning: Function Approximation - classnotes

• Week 6  : Dynamical Models -  classnotes(KF) - classnotes(HMM)

• Week 7  :  Approximate Inference - classnotes

• Week 8 :  Belief Nets, Graphical Models  - classnotes

• Week 9:  Summary and Advanced Topics

• homework1  -   MoGdata.mat
• homework2  -   FAdata.mat -  FAparams.mat
• homework3  -   CWMdata.mat  -  CWMexample.m
• homework4  -  KFdata.mat -  KFexample.m
• final project - due by June 2nd

Below follow some suggestions for possible projects. Only  the first two contain detailed  derivations.
For the other ones, you may want to do a search in the web, or visit the labs (between brackets)
to look for papers. The links to most labs are given above (under Links & Download).
You can also find some interesting datasets there. You are very welcome to suggest your own
datasets or method. The general idea is that you study a statistical model from the literature,
implement it and test it on a real world dataset.
 

• Mixture of Factor Analysers (MoFA)   (BISHOP)
• Generative Topographic Mapping (GTM)  (BISHOP)
• Nonlinear PCA (CIS)
• HMM
• Extended Kalman Filter, Nonlinear Kalman Filter
• Product of Experts (PoE) (GATSBY)
• Deterministic Boltzmann Machine
• Helmholtz Machine (GATSBY)
• Community of Experts (GATSBY)
• Mixture of Experts IRLS implementation (JORDAN)
• Independent Factor Analysis (GATSBY)

•  Classnotes will be provided before every class.
•  C.M. Bishop, Neural Networks for Pattern Recognition.
•  B.D. Ripley, Pattern Recognition and Neural Networks.
•  K.V. Mardia, J.T. Kent and J.M. Bibby, Multivariate Analysis.
•  M.I. Jordan, Learning in Graphical Models.
•  T.M. Cover and J.A. Thomas, Elements of Information Theory.
•  A. Papoulis, Probability, Random Variables and Stochastic Processes
•  G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions.

California Institute of Technology, Pasadena, CA 91125.