ComS 573: Machine Learning
Department of Computer Science
Iowa State University
Spring 2008

The material to be covered each week and the assigned readings (along with online lecture notes, if available) are included on this page. The links to lecture notes will not be in place usually until a week after the lecture. The assigned readings are divided into required and recommended readings. You will be responsible for the material covered in the lectures and the assigned required readings. You are strongly encouraged to explore the recommended readings.

Survival Tips

• Keep up with the assigned readings.
• If you don't understand something in class, or the assigned problems/labs/readings, ask questions! 
• Do not postpone working on assignments.
• Before you come to class, review the materials from the previous lecture and come prepared to ask any questions that you might have.

Week 1 (starting January 14, 2008)

Overview of the course

Review of probability theory, Information theory.

Required Readings

• Course information and policies
• Overview of Machine Learning, Nils Nilsson.
• Chapter 1 up to 1.2.3, and 1.6 of C. Bishop (2006), Pattern Recognition and Machine Learning.
• Lecture slides
• Lecture slides
• Lecture slides

Assignments

• Problem Set 1, Due Friday, January 25, 2008, in class

Recommended Readings for those unfamiliar with probability theory

• Chapters 13, Artificial Intelligence: A Modern Approach, Russell and Norvig.
• Chapters 1, 4, 6 from Introduction to Probability -- Charles Grinstead and Laurie Snell.
• Chapters 2, 4 from Information Theory, Inference, and Learning Algorithms. David MacKay

Week 4 (starting February 4, 2008)

Evaluation of classifiers. Performance measures: Accuracy, Precision, Recall, ROC curves.

Evaluation of classifiers -- estimation of performance measures; confidence interval calculation for estimates; cross-validation; comparing two hypotheses; hypothesis testing; comparing two learning algorithms.
 

Required Readings

• Lecture slides
• Fawcett, T. (2003) ROC Graphs: Notes and Practical Considerations for Researchers HP Labs Tech. Report HPL2003-4.
• Chapters 5 from: Mitchell, T. 1997. Machine Learning. New York: McGraw Hill.
• Ron Kohavi, A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. IJCAI-95
• Confidence Interval and Hypothesis Testing.
• WEKA Machine Learning Algorithms in Java

Assignments

• Laboratory Assignment 1, Due Friday February 15, 12pm noon, extended to Wednesday February 20, 12 pm noon.

Recommended Readings

• F Provost, T Fawcett, R Kohavi (1998). A case against accuracy estimation of machine learning algorithms. In: proceedings of the fifteenth International Conference on Machine Learning.
• Stark, Philip B. Statistics Tools for Internet and Classroom Instruction with a Graphical User Interface, Chapter 19.
• Dietterich, T. (1998). Approximate Statistical Tests for Comparing Supervised Classification Algorithms Neural Computation. 10(7):1895-1923.
• Salzberg, S. (1999). On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach Data Mining and Knowledge Discovery, Vol. 1, pp. 317-328.

Week 5 (starting February 11, 2008)

The decision tree classifier: decision tree learning algorithm (Quinlan's ID3 algorithm and extensions C4.5); The problem of overfitting, missing data.

Linear models for classification: Linear Discriminant Functions, Perceptrons
 

Required Readings

• Lecture slides
• Chapters 3 from: Mitchell, T. 1997. Machine Learning. New York: McGraw Hill. Or Decision Trees, Nils Nilsson.
• Decision Trees, by H. Hamilton, E. Gurak, L. Findlater, and W. Olive

• Lecture slides
• Chapter 4.1 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Chapter 4.1 from Nils Nilsson. Neural Networks.

Recommended Readings

• Fayyad, U. and Irani, K.B. (1992). On the handling of continuous valued attributes in decision tree generation. Machine Learning vol. 8. pp. 87-102.
• Domingos, P. (1999). The Role of Occam's Razor in Knowledge Discovery. Vol. 3, no. 4., pp. 409-425.
• Brodley, C. and Utgoff, P. (1995). Multi-variate Decision Trees. Machine Learning 19: 45-77.
• Caragea, D., Silvescu, A., and Honavar, V. (2004). A Framework for Learning from Distributed Data Using Sufficient Statistics and its Application to Learning Decision Trees. International Journal of Hybrid Intelligent Systems. Invited Paper. Vol. 1. pp. 80-89.
• Johannes E. Gehrke, Raghu Ramakrishnan, and Venkatesh Ganti. RAINFOREST - A Framework for Fast Decision Tree Construction of Large Datasets. Data Mining and Knowledge Discovery, Volume 4, Issue 2/3, July 2000, pp 127-162.
• Quinlan, R.. Induction of Decision Trees, Machine Learning, Vol. 1, pp. 81-106, 1986.
• Quinlan, R.. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, Inc., 1993

Week 6 (starting February 18, 2008)

Linear models for classification: Linear Discriminant Functions, Perceptrons, Perceptron Learning algorithm, Multi-category classification, Winner-Take-All (WTA) networks (linear machines).

Regression: Linear regression, Least Mean Squared (LMS) Error Criterion,  Gradient descent algorithm, LMS solution for classification.

Fisher Linear discriminant
 

Required Readings

• Lecture slides
• Lecture slides
• Chapter 4.1, 1.1, 1.2.5, 1.5.5, 3.1 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Chapter 4.1, 4.2 from Nils Nilsson. Neural Networks.

Assignments

• Problem Set 3, Due Wednesday, February 27, 2008, in class
• Laboratory Assignment 2, Due Friday February 29, 12pm noon.

Recommended Readings

• Chapter 5 from: Pattern Classification (2nd ed., 2001) by Richard O. Duda, Peter E. Hart and David G. Stork.
• Elizondo, D., The linear separability problem: some testing methods. IEEE Transactions on Neural Networks, Volume: 17, Issue: 2, pages: 330- 344.
   

Week 7 (starting February 25, 2008)

Probabilistic generative models and linear classifiers

Discriminative models for classification, logistic regression models,  training logistic regression, regularized logistic regression

Instance-Based Learning, k-Nearest neighbor learning
 

Required Readings

• Lecture slides
• Chapter 4.2, 2.4, 4.3 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Mitchell, T. (2005). Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression, Draft book chapter.
• Rubinstein, D and Hastie, T. Discriminative vs Informative Learning. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1997.

• Lecture slides
• Chapter 2.5 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Chapters 8 from: Mitchell, T. 1997. Machine Learning. New York: McGraw Hill.

Recommended Readings

• Minka, T.P. (2004) A comparison of Numerical Optimizers for Logistic Regreassion.
• Raina, R., Shen, Y., Ng, A., and McCallum, A. (2003). Classification with Hybrid Generative/Discriminative Models. In Proceedings of the IEEE Conference on Neural Information Systems (NIPS 2003).
• Lasserre, J., C. M. Bishop, and T. Minka (2006). Principled hybrids of generative and discriminative models. In: Proceedings 2006 IEEE Conference on Computer Vision and Pattern Recognition, New York.
• Ng, A. and Jordan, M. (2002) On Discriminative vs. Generative Classifiers: A comparison of logistic regression and Naive Bayes, Proceedings of the IEEE Conference on Neural Information Systems (NIPS 2002).

Week 8 (starting March 3, 2008)

Multilayer neural networks: feedforward operations, expressive power of multilayer networks, Backpropagation algorithm, Regularization, stopping criterion, Practical techniques for improving backpropagation, , error functions for classification. Function optimization algorithms---conjugate gradient and quasi-Newton algorithms

Required Readings

• Lecture slides
• Lecture slides
• Chapter 5.1-5.5 from C. Bishop (2006), Pattern Recognition and Machine Learning.

Assignments

• Laboratory Assignment 3, Due Wednesday March 26, 12pm noon.

Recommended Readings

• Chapter 7 from C. Bishop (1995), Neural Networks for Pattern Recognition.
• Chapter 10  from ``Numerical Recipes in C'', William H. Press et al.
• Chapter 6 from: Pattern Classification (2nd ed., 2001) by Richard O. Duda, Peter E. Hart and David G. Stork
• R. Setiono, W.K. Leow and J.M. Zurada. Extraction of rules from artificial neural networks for nonlinear regression, IEEE Transactions on Neural Networks, 2002.
• Craven, M. and Shavlik, J. Using Neural Networks for Data Mining. Future Generation Computer Systems 13:211-229.
1997.

Week 9 (starting March 10, 2008)

Support Vector Machines: Dual Representation of the Perceptron Algorithm, Learning in feature spaces, Kernel Functions, Kernel-Induced Feature Spaces, Maximum Margin Classifiers, constrained optimization problem, Learning as optimization, Support Vectors, Non-Separable Case, The Soft-Margin Classifier

Required Readings

• Lecture slides
• Chapter 6.1, 6.2, Appendix E, 7.1 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• A Tutorial on Support Vector Machines. Nello Christianini, International Conference on Machine Learning (ICML 2001).
• M. A. Hearst, B. Schvlkopf, S. Dumais, E. Osuna, and J. Platt. Trends and controversies - support vector machines. IEEE Intelligent Systems, 13(4):18-28, 1998.
• Brown, M. P., Grundy, W. N., Lin, D., Cristianini, N., Sugnet, C. W., Furey, T. S., Ares, M. Jr, and Haussler, D. Knowledge-based analysis of microarray gene expression data by using support vector machines. Proc. Natl. Acad. Sci. USA 97: 262-267: 2000.
• T. Joachims. Text categorization with support vector machines: Learning with many relevant features. In European Conference on Machine Learning (ECML-98), 1998.

Assignments

• Problem Set 4, Due Monday, March 31, 2008, in class

Recommended Readings

• J.C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):121--167, 1998.
• Chapters 1-7 from Mathematical methods for economic theory: a tutorial by Martin J. Osborne
• Platt, J. Fast training of support vector machines using sequential minimal optimization. In B. Scholkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods --- Support Vector Learning, pages 185-208, Cambridge, MA, 1999. MIT Press.
• Laskov, P., Gehl, C., Kruger, S., Muller, K-R. (2006) Incremental Support Vector Learning: Analysis, Implementation and Applications , Journal of machine learning research, Vol. 7. pp. 1909-1936.
• Hsu, C-W., and Lin, C-J. (2002). A Comparison of methods for multi-class Support Vector Machines, IEEE Transactions on Neural Networks, Vol. 13, pp. 415-425.
• Duan, K-B., and Keerthi, S. (2005). Which is the best multiclass SVM Method? - An Empirical Study, Springer-Verlag Lecture Notes in Computer Science Vol. 3541, pp. 278-285.

Additional Information

• Cristianini, N. and Shawe-Taylor, J. (2000). Support Vector Machines. London: Cambridge University Press.
• Shawe-Taylor, J. and Cristianini, N. (2004). Kernel Methods for Pattern Classification. London: Cambridge University Press.

Spring Break

Week 10 (starting March 24, 2008)

Bayesian Networks: Syntax and Semantics, D-separation.

Required Readings

• Lecture slides
• Chapter 8 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Tutorial on Bayesian networks by Kevin Murphy

Additional Information

• Judea Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. (1988).
• Cowell, R. G. Lauritzen, S. L., and Spiegelhalter, D. J. Probabilistic Networks and Expert Systems Berlin: Springer (1999).
• Korb, K.B., and Nicholson, A.E., Bayesian Artificial Intelligence, Chapman and Hall (2004).
• Richard E. Neapolitan, Learning Bayesian Networks, Prentice Hall, 2004.

Week 11 (starting March 31, 2008)

Bayesian Networks: modeling

Required Readings

• Lecture slides
• Chapter 8 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Tutorial on Bayesian networks by Kevin Murphy

Assignments

• Problem Set 5, Due Friday, April 11, 2008, in class

Additional Information

• Judea Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. (1988).
• Cowell, R. G. Lauritzen, S. L., and Spiegelhalter, D. J. Probabilistic Networks and Expert Systems Berlin: Springer (1999).
• Korb, K.B., and Nicholson, A.E., Bayesian Artificial Intelligence, Chapman and Hall (2004).
• Richard E. Neapolitan, Learning Bayesian Networks, Prentice Hall, 2004.

Week 12 (starting April 7, 2008)

Bayesian Networks: inference, learning

Required Readings

• Lecture slides
• NIPS 2001 Tutorial on learning Bayesian networks from Data by Nir Friedman and Daphne Koller
• Nir Friedman and Moises Goldzsmidt, AAAI-98 Tutorial on learning Bayesian networks from Data.
• Chapter 8 from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Tutorial on Bayesian networks by Kevin Murphy

Recommended Readings

• R. Dechter, "Bucket Elimination: A Unifying Framework for Probabilistic Inference" . In Uncertainty in Artificial Intelligence (UAI) 1996.
• Cecil Huang and Adnan Darwiche. Inference in belief networks: A procedural guide. In International Journal of Approximate Reasoning, 15(3):225-263, October, 1996
• Heckerman, D., Geiger, D., and Chickering, D. (1995). Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3):197--243.
• Friedman and Koller, Being Bayesian about Network Structure: A Bayesian Approach to Structure Discovery in Bayesian Networks, Machine Learning, 50:95-126, 2003
•  D. Chickering 2003, Optimal structure identification with greedy search, the Journal of Machine Learning Research
• Tractable Learning of Large Bayes Net Structures from Sparse Data, Goldernberg, A. and Moore, A. (2004). In Proceedings of the International Conference on Machine Learning, 2004.
• Exact Bayesian Structure Discovery in Bayesian Networks, Mikko Koivisto, Kismat Sood, the Journal of Machine Learning Research, 2004.
• Nir Friedman, D. Geiger, and M. Goldszmidt, Bayesian network classifiers. In Machine Learning 29:131--163, 1997.
• Using Bayesian Networks to Analyze Expression Data N. Friedman,  M. Linial, I. Nachman, and D. Pe'er. Journal of Computational Biology, 7:601--620, 2000.
• Inferring cellular networks using probabilistic graphical models. N. Friedman, Science. 303:799-805, 2004. 
• S. L. Lauritzen and N. A. Sheehan. Graphical models for genetic analyses. Statistical Science, 18, 489-514, 2003

Additional Information

• Judea Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. (1988).
• Cowell, R. G. Lauritzen, S. L., and Spiegelhalter, D. J. Probabilistic Networks and Expert Systems Berlin: Springer (1999).
• Korb, K.B., and Nicholson, A.E., Bayesian Artificial Intelligence, Chapman and Hall (2004).
• Richard E. Neapolitan, Learning Bayesian Networks, Prentice Hall, 2004.

Week 13 (starting April 14, 2008)

Bayesian Networks: learning

Ensemble Classifiers: Bagging, The Adaboost Algorithm

Required Readings

• Lecture slides
• Chapter 14.1-3, from C. Bishop (2006), Pattern Recognition and Machine Learning.
• Thomas G. Dietterich, Ensemble methods in machine learning, 2000
• Breiman, L. (1994). Bagging Predictors. Tech. Rep. 421, Department of Statistics, University of California, Berkeley, CA.
• Freund, R. and Schapire, R. (1999) A Short Introduction to Boosting Journal of the Japanese Society for Artificial Intelligence, Vol 14, pp. 771-780.
• Baur, E., and Kohavi, R. (1999) An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants Machine Learning. Vol. 36. pp. 105-142.
• Dietterich, T. G., (2000). An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40 (2) 139-158

Recommended Readings

• Robert E. Schapire. The boosting approach to machine learning: An overview. In D. D. Denison, M. H. Hansen, C. Holmes, B. Mallick, B. Yu, editors, Nonlinear Estimation and Classification. Springer, 2003.
• Friedman, J., Hastie, T., and Tibshirani, R. (2000). A Statistical View of Boosting, Annals of Statistics, Vol. 35, pp. 337-407.
• Meir, R. and Ratsch, G. (2002). An Introduction to Boosting and Leveraging. Advanced Lectures on Machine Learning. Lecture Notes in Computer Science, pp. 118-183, Berlin: Springer-Verlag.

Week 14 (starting April 21, 2008)

Ensemble Classifiers: Error correcting output coding

Mixture Models: Clustering, the EM algorithm, the K-means clustering algorithm

Required Readings

• Lecture slides
• Thomas G. Dietterich and G. Bakiri (1995), Solving Multiclass Learning Problems via Error-Correcting Output Codes, Journal of Artificial Intelligence Research.

• Lecture slides
• Chapter 9, from C. Bishop (2006), Pattern Recognition and Machine Learning.

Recommended Readings

• Robert E. Schapire (1997) Using output codes to boost multiclass learning problems. Proc. of 14th International Conference on Machine Learning.
• Erin L. Allwein, Robert E. Schapire, and Yoram Singer (2000), Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers, Journal of Artificial Intelligence Research (JMLR)
• Pavel Berkhin. A Survey of Clustering Algorithms 2002.