Artificial Intelligence Research Laboratory Department of Computer Science Iowa State University

Constructive Neural Network Learning Algorithms for Pattern Classification
   Personnel   Project Summary   Funding   Publications   Additional Information   Projects   AI Lab  

• Dr. Vasant Honavar, Principal Investigator.
• Dr. Karthik Balakrishnan, Former Ph.D. Student.
• Dr. Rajesh Parekh, Former Ph.D. Student.
• Dr. Jihoon Yang, Former Ph.D. Student.
• Dr. Chun-Hsien Chen , Former Ph.D. Student.

Project Summary

Induction of pattern classifiers from data is an important area of research in machine learning which finds applications in diverse areas including automated diagnosis, bioinformatics, design of customizable information assistants, intrusion detection in computer systems, among others. Artificial neural networks, because of their potential for massive parallelism and fault and noise tolerance, offer an attractive approach to the design of trainable pattern classifiers.

Constructive learning algorithms, which avoid the guesswork involved in deciding a suitable network architectures for different pattern classification problems by growing a network by recruiting neurons as needed can be effectively trained to solve complex pattern classification problems. Furthermore, it is possible to generalize (and provide convergence guarantees for) a large family of such algorithms designed for 2-class binary pattern classification problems to handle classification problemsinvolving real-valued patterns and an arbitrary number of classes.

Different constructive neural network algorithms as well as the algorithms used to train the neuron weights have different inductive and representational biases making their performance sensitive to specific characteristics of the datasets. Thus, it is useful to experiment with a toolkit of multiple algorithms for practical applications. It is also possible to design hybrid algorithms that exploit the synergy between different algorithms. Visualization of decision boundaries constructed by the different algorithms can often provide useful insights into their behavior. This argues for systems that combine machine learning algorithms with visualization techniques in high dimensions to facilitate interactive knowledge discovery.

A simple inter-pattern distance based, provably convergent, polynomial time constructive neural network algorithm compares very favorably with computationally far more expensive algorithms in terms of generalization accuracy. The generalization accuracy of this algorithm (as expected), is sensitive to the choice of attributes used to represent the patterns and can be improved by using feature subset selection.

Fairly simple algorithms for incorporating prior knowledge can be used to enhance the performance of constructive algorithms for designing pattern classifiers. This raises the possibility of using constructive algorithms for knowledge transfer across similar tasks to facilitate multitask learning in complex enviroments.

A constructive learning procedure, augmented with a Kalman filter mechanism for estimation, can be effectively used for place learning and localization in a-priori unknown environments in the presence of sensor and motion uncertainties. The resulting computational model successfully accounts for a large body of behavioral and neurobiological data from animal experiments and offers several testable predictions.

Preliminary results indicate that constructive learning algorithms and related approaches to machine learning can be adapted for data-driven knowledge discoveryfrom distributed, dynamic data sources in a number of domains including bioinformatics, monitoring and control of complex distributed systems (computer systems, communication networks, power systems) and decision support systems.

Funding

• Constructive Neural Network Learning Algorithms for Pattern Classification, National Science Foundation, (1994-1999). Vasant Honavar. $111,537 (plus $10,000 in matching funds).
• Integrated Intelligent Diagnosis and Advisory Systems - Analysis, John Deere Foundation, 1995-1996. Vasant Honavar. $15,000.
• Integrated Intelligent Diagnosis Systems, John Deere Foundation, 1996-1998. Vasant Honavar. $15,000.
• Data Mining and Knowledge Discovery, IBM Corporation, 1997-1998.

Publications

• Balakrishnan, K., Bousquet, O. and Honavar, V. (1999). Spatial Learning and Localization in Animals: A Computational Model and Its Implications for Mobile Robots, Adaptive Behavior. In press.
• Balakrishnan, K. & Honavar, V. (2000). Experiments in Evolutionary Robotics. In: Advances in Evolutionary Synthesis of Neural Systems. Honavar, V., Patel, M. and Balakrishnan, K. (Ed). Cambridge, MA: MIT Press. To appear.
• Chen, C-H. & Honavar, V. (1999). A Neural Architecture for Syntax Analysis. IEEE Transactions on Neural Networks. Vol. 10 pp. 94-114.
• Chen, C-H. & Honavar, V. (1999). A Neural Architecture for Information Retrieval and Query Processing. Invited chapter. In: Handbook of Natural Language Processing. Dale, Moisl, and Somers (Ed.) Somers (Ed). New York: Marcel Dekker. In press.
• Honavar, V., Parekh, R. and Yang, J. (1999). Structural Learning. Invited article. In: Encyclopedia of Electrical and Electronics Engineering, Webster, J. (Ed.), New York: Wiley. In press.
• Honavar, V., Parekh, R. and Yang, J. (1999). Machine Learning. Invited article. In: Encyclopedia of Electrical and Electronics Engineering, Webster, J. (Ed.), New York: Wiley. In press.
• Parekh, R., Yang, J., and Honavar, V. (1999). Comparison of Performance of Variants of Single-Layer Perceptron Algorithms on Non-Separable Data. Neural, Parallel, and Scientific Computations. In press.
• Parekh, R., Yang, J., and Honavar, V. (1999). Constructive Neural Network Learning Algorithms for Multi-Category Pattern Classification. IEEE Transactions on Neural Networks. In press.
• Balakrishnan, K. and Honavar, V. (1998). Intelligent Diagnosis Systems. Journal of Intelligent Systems.
• Yang, J. and Honavar, V. (1998). Feature Subset Selection Using a Genetic Algorithm. Invited chapter. In: Feature Extraction, Construction, and Subset Selection: A Data Mining Perspective. Motoda, H. and Liu, H. (Ed.) New York: Kluwer. 1998.
• Yang, J. and Honavar, V. (1998). DistAl: An Inter-Pattern Distance Based Constructive Neural Network Learning Algorithm.. Intelligent Data Analysis. In press.
• Yang, J. and Honavar, V. (1998). Feature Subset Selection Using a Genetic Algorithm. IEEE Intelligent Systems (Special Issue on Feature Transformation and Subset Selection). vol. 13. pp. 44-49.
• Chen, C-H. and Honavar, V. (1996). A Neural Network Architecture for High-Speed Database Query Processing. Microcomputer Applications. vol. 15, no. 1. pp. 7-13.
• Chen, C-H. and Honavar, V. (1995). A Neural Memory Architecture for Content as well as Address-Based Storage and Recall: Theory and Applications Connection Science. vol. 7. pp. 293-312.
• Honavar, V. and Uhr, L. (1995). Integrating Symbol Processing and Connectionist Networks. Invited chapter. In: Intelligent Hybrid Systems. pp. 177-208. Goonatilake, S. and Khebbal, S. (Ed.) London: Wiley.
• Honavar, V. (1994). Toward Learning Systems That Use Multiple Strategies and Representations. In: Artificial Intelligence and Neural Networks: Steps Toward Principled Integration. pp. 615-644. Honavar, V. and Uhr, L. (Ed.) New York: Academic Press.
• Honavar, V. (1994). Symbolic Artificial Intelligence and Numeric Artificial Neural Networks: Toward a Resolution of the Dichotomy. Invited chapter. In: Computational Architectures Integrating Symbolic and Neural Processes. pp. 351-388. Sun, R. and Bookman, L. (Ed.) New York: Kluwer.
• Uhr, L., and Honavar, V. (1994). Artificial Intelligence and Neural Networks: Steps Toward Principled Integration. In: Artificial Intelligence and Neural Networks: Steps Toward Principled Integration. pp. xvii-xxxii. Honavar, V. and Uhr, L. (Ed). New York: Academic Press.
• Bhatt, R., Balakrishnan, K., and Honavar, V. (1999). A Constructive Neural Network Algorithm for Place Learning. In: Proceedings of the International Joint Conference on Neural Networks. Washington, D.C.
• Yang, J., Parekh, R., Honavar, V., and Dobbs, D. (1999). Data-Driven Theory Refinement Using KBDistAl. In: Proceedings of the Conference on Intelligent Data Analysis. Amsterdam, Holland.
• Yang, J., Parekh, R., Honavar, V., and Dobbs, D. (1999). Data-Driven Theory Refienement Algorithms for Bioinformatics. In: Proceedings of the International Joint Conference on Neural Networks. Washington, D.C.
• Balakrishnan, K., Bhatt, R., and Honavar, V. (1998). A Computational Model of Rodent Spatial Learning and Some Behavioral Experiments. In: Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society. Madison, WI.
• Bousquet, O., Balakrishnan, K. and Honavar, V. (1998). Is the Hippocampus a Kalman Filter?. In: Proceedings of the Pacific Symposium on Biocomputing. Singapore: World Scientific. pp. 655-666.
• Parekh, R. and Honavar, V. (1998). Constructive theory refinement in knowledge based neural networks. In: Proceedings of the International Joint Conference on Neural Networks. Anchorage, Alaska. pp. 2318-2323.
• Yang, J., Parekh, R., and Honavar, V. (1998). Distal: An Inter-pattern distance-based constructive learning algorithm. In: Proceedings of the International Joint Conference on Neural Networks. Anchorage, Alaska. pp. 2208-2213.
• Yang, J., Pai, P., Honavar, V., and Miller, L. (1998). Mobile Intelligent Agents for Document Classification and Retrieval: A Machine Learning Approach. In: Proceedings of the European Symposium on Cybernetics and Systems Research.
• Yang, J., Honavar, V., Miller, L. and Wong, J. (1998). Intelligent Mobile Agents for Information Retrieval and Knowledge Discovery from Distributed Data and Knowledge Sources. In: Proceedings of the IEEE Information Technology Conference. Syracuse, NY.
• Balakrishnan, K. and Honavar, V. (1997). Spatial Learning for Robot Localization. In: Proceedings of International Conference on Genetic Programming. Stanford, CA. pp. 389-397.
• Parekh, R.G., Yang, J., and Honavar, V. (1997). MUPStart - A Constructive Neural Network Learning Algorithm for Multi-Category Pattern Classification. In: Proceedings of IEEE International Conference on Neural Networks (ICNN'97). Houston, TX. pp. 1924-1929.
• Parekh, R.G., Yang, J., and Honavar, V. (1997). Pruning Strategies for Constructive Neural Network Learning Algorithms. In: Proceedings of IEEE International Conference on Neural Networks (ICNN'97). Houston, TX. pp. 1960-1965. June 9-12, 1997.
• Yang, J. and Honavar, V. (1997). Feature Subset Selection Using a Genetic Algorithm. In: Proceedings of International Conference on Genetic Programming. Stanford, CA. pp. 380-385.
• Zhou, G., McCalley, J. D. and Honavar, V. (1997). Power System Security Margin Prediction Using Radial Basis Function Networks. In: Proceedings of the 29th Annual North American Power Symposium. Laramie, Wyoming. October 13-14, 1997.
• Balakrishnan, K. and Honavar, V. (1996). Analysis of Neurocontrollers Designed by Simulated Evolution. Proceedings of the International Conference on Neural Networks. Washington, D.C.
• Balakrishnan, K. and Honavar, V. (1996). On Sensor Evolution in Robotics. Proceedings of the First International Conference on Genetic Programming, Stanford University, CA. pp. 455-460.
• Balakrishnan, K. and Honavar, V. (1996). Some Experiments in Evolutionary Synthesis of Robotic Neurocontrollers In: Proceedings of the World Congress on Neural Networks. (WCNN'96) San Diego, CA. September 15-20, 96. pp. 1035-1040.
• Yang, J., Parekh, R. and Honavar, V. (1996). MTiling: A Constructive Neural Network Learning Algorithm for Multi-Category Pattern Classification. In: Proceedings of the World Congress on Neural Networks. (WCNN'96), San Diego, CA. September 15-20, 96. pp. 182-187.
• Yang, J. and Honavar, V. (1996). A Simple Randomized Quantization Algorithm for Neural Network Pattern Classifiers. In: Proceedings of the World Congress on Neural Networks. San Diego, CA. September 15-20, 96. pp. 223-228.
• Chen, C-H., Parekh, R., Yang, J., Balakrishnan, K. and Honavar, V. (1995). Analysis of Decision Boundaries Generated by Constructive Neural Network Learning Algorithms. In: Proceedings of the World Congress on Neural Networks (WCNN'95). Washington, D.C. July 17-21, 1995. pp. 628-635.
• Chen, C-H., and Honavar, V. (1994). Neural Network Automata. In: Proceedings of the World Congress on Neural Networks. pp. 470-477. San Diego, CA.
• Yang, J. (1999). Adaptive Agents For Information Retrieval and Data-Driven Knowledge Acquisition. Doctoral Dissertation. Department of Computer Science. Iowa State University.
• Balakrishnan, K. (1998). Biologically inspired computational structures and processes for intelligent agents and robots.. Doctoral Dissertation. Department of Computer Science, Iowa State University.
• Parekh, R.G. (1998). Constructive Learning: Inducing Grammars and Neural Networks. Doctoral Dissertation. Department of Computer Science, Iowa State University.
• Chen, C. (1997). Neural Architectures for Associative Memory, Syntax Analysis, Knowledge Representation and Inference. Doctoral Dissertation. Department of Computer Science, Iowa State University.

Additional Information

To appear.

© Vasant Honavar, 1999.