HKUST Library Institutional Repository Banner

HKUST Institutional Repository >
Computer Science and Engineering >
CSE Journal/Magazine Articles >

Please use this identifier to cite or link to this item:
Title: Pruning hidden Markov models with optimal brain surgeon
Authors: Mak, Brian Kan-Wing
Chan, Kin-Wah
Keywords: Model pruning
Hidden Markov model
Optimal brain surgeon
Quadratic programming
Issue Date: Sep-2005
Citation: IEEE transactions on Speech and Audio Processing, v. 13, no. 5, September 2005, p. 993-1003
Abstract: A method of pruning hidden Markov models (HMMs) is presented. The main purpose is to find a good HMM topology for a given task with improved generalization capability. As a side effect, the resulting model will also save memory and computation costs. The first goal falls into the active research area of model selection. From the model-theoretic research community, various measures such as Bayesian information criterion, minimum description length, minimum message length have been proposed and used with some success. In this paper, we are considering another approach in which a well-performed HMM, though perhaps oversized, is optimally pruned so that the loss in the model training cost function is minimal. The method is known as Optimal Brain Surgeon (OBS) that has been applied to pruning neural networks in the past. In this paper, the OBS algorithm is modified to prune HMMs. While the application of OBS to neural networks is a constrained optimization problem with only equality constraints that can be solved by Lagrange multipliers, its application to HMMs requires significant modifications, resulting in a quadratic programming problem with both equality and inequality constraints.The detailed formulation of pruning an HMM with OBS is presented. It was evaluated by two experiments: one simulation using a discrete HMM, and another with continuous density HMMs trained for the TIDIGITS task. It is found that our novel OBS algorithm was able to 're-discover' the true topology of the discrete HMM in the first simulation experiment; in the second speech recognition experiment, up to about 30% of HMM transitions were successfully pruned, and yet the reduced models gave better generalization performance on unseen test data.
Rights: © 2005 Year IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.
Appears in Collections:CSE Journal/Magazine Articles

Files in This Item:

File Description SizeFormat
sapobshmmx.pdfpre-published version229KbAdobe PDFView/Open

Find published version via OpenURL Link Resolver

All items in this Repository are protected by copyright, with all rights reserved.