Approaches to Iterative Speech Feature Enhancement and Recognition - Windmann, Stefan; Haeb-Umbach, Reinhold (IEEE Transactions on Audio, Speech, and Language Processing)
In automatic speech recognition, hidden Markov models (HMMs) are commonly used for speech decoding, while switching linear dynamic models (SLDMs) can be employed for a preceding model-based speech feature enhancement. In this paper, these model types are combined in order to obtain a novel iterative speech feature enhancement and recognition architecture. It is shown that speech feature enhancement with SLDMs can be improved by feeding back information from the HMM to the enhancement stage. Two different feedback structures are derived. In the first, the posteriors of the HMM states are used to control the model probabilities of the SLDMs, while in the second they are employed to directly influence the estimate of the speech feature distribution. Both approaches lead to improvements in recognition accuracy both on the AURORA2 and AURORA4 databases compared to non-iterative speech feature enhancement with SLDMs. It is also shown that a combination with uncertainty decoding further enhances performance.
Keywords: AURORA2 databases; AURORA4 databases; automatic speech recognition; feedback structures; hidden Markov models; HMM; iterative methods; iterative speech feature enhancement; model probabilities; speech decoding; speech enhancement; speech feature distribution; speech recognition; switching linear dynamic models

Parameter Estimation of a State-Space Model of Noise for Robust Speech Recognition - Windmann, Stefan; Haeb-Umbach, Reinhold (IEEE Transactions on Audio, Speech, and Language Processing)
In this paper, parameter estimation of a state-space model of noise or noisy speech cepstra is investigated. A blockwise EM algorithm is derived for the estimation of the state and observation noise covariance from noise-only input data. It is supposed to be used during the offline training mode of a speech recognizer. Further a sequential online EM algorithm is developed to adapt the observation noise covariance on noisy speech cepstra at its input. The estimated parameters are then used in model-based speech feature enhancement for noise-robust automatic speech recognition. Experiments on the AURORA4 database lead to improved recognition results with a linear state model compared to the assumption of stationary noise.
Keywords: AURORA4 database; blockwise EM algorithm; covariance analysis; linear state model; noise covariance; noise-robust automatic speech recognition; noisy speech cepstra; offline training mode; parameter estimation; speech recognition; speech recognition equipment; speech recognizer; state-space methods; state-space model

Audio-Visual Data Processing for Ambient Communication - Schmalenstroeer, Joerg; Leutnant, Volker; Haeb-Umbach, Reinhold (1st International Workshop on Distributed Computing in Ambient Environments within 32nd Annual Conference on Artificial Intelligence)

A hierarchical approach to unsupervised shape calibration of microphone array networks - Hennecke, Marius; Ploetz, Thomas; Fink, Gernot A.; Schmalenstroeer, Joerg; Haeb-Umbach, Reinhold (IEEE/SP 15th Workshop on Statistical Signal Processing (SSP 2009))
Microphone arrays represent the basis for many challenging acoustic sensing tasks. The accuracy of techniques like beamforming directly depends on a precise knowledge of the relative positions of the sensors used. Unfortunately, for certain use cases manually measuring the geometry of an array is not feasible due to practical constraints. In this paper we present an approach to unsupervised shape calibration of microphone array networks. We developed a hierarchical procedure that first performs local shape calibration based on coherence analysis and then employs SRP-PHAT in a network calibration method. Practical experiments demonstrate the effectiveness of our approach especially for highly reverberant acoustic environments.
Keywords: acoustic sensing tasks; array geometry; calibration; coherence analysis; hierarchical procedure; local shape calibration; microphone array networks; microphone arrays; network calibration method; sensor arrays; SRP-PHAT; unsupervised shape calibration

Fusing Audio and Video Information for Online Speaker Diarization - Schmalenstroeer, Joerg; Kelling, Martin; Leutnant, Volker; Haeb-Umbach, Reinhold (Interspeech 2009)
In this paper we present a system for identifying and localizingspeakers using distant microphone arrays and a steerablepan-tilt-zoom camera. Audio and video streams are processedin real-time to obtain the diarization information grqqwho speakswhen and where'' with low latency to be used in advanced videoconferencing systems or user-adaptive interfaces. A key featureof the proposed system is to first glean information about thespeaker{\rq}s location and identity from the audio and visual datastreams separately and then to fuse these data in a probabilisticframework employing the Viterbi algorithm. Here, visual evidenceof a person is utilized through a priori state probabilities,while location and speaker change information are employedvia time-variant transition probablities. Experiments show thatvideo information yields a substantial improvement comparedto pure audio-based diarization.

Uncertainty Decoding in Automatic Speech Recognition - Haeb-Umbach, Reinhold (2008 ITG Conference on Voice Communication (SprachKommunikation))
The term uncertainty decoding has been phrased for a class of robustness enhancing algorithms in automatic speech recognition that replace point estimates and plug-in rules by posterior densities and optimal decision rules. While uncertainty can be incorporated in the model domain, in the feature domain, or even in both, we concentrate here on feature domain approaches as they tend to be computationally less demanding. We derive optimal decision rules in the presence of uncertain observations and discuss simplifications which result in computationally efficient realizations. The usefulness of the presented statistical framework is then exemplified for two types of realworld problems: The first is improving the robustness of speech recognition towards incomplete or corrupted feature vectors due to a lossy communication link between the speech capturing front end and the backend recognition engine. And the second is the well-known and extensively studied issue of improving the robustness of the recognizer towards environmental noise.

Error Concealement - Haeb-Umbach, Reinhold; Ion, Valentin (Automatic Speech Recognition on Mobile Devices and over Communication Networks)
In distributed and network speech recognition the actual recognition task is not carried out on the user{\rq}s terminal but rather on a remote server in the network. While there are good reasons for doing so, a disadvantage of this client-server architecture is clearly that the communication medium may introduce errors, which then impairs speech recognition accuracy. Even sophisticated channel coding cannot completely prevent the occurrence of residual bit errors in the case of temporarily adverse channel conditions, and in packet-oriented transmission packets of data may arrive too late for the given real-time constraints and have to be declared lost. The goal of error concealment is to reduce the detrimental effect that such errors may induce on the recipient of the transmitted speech signal by exploiting residual redundancy in the bit stream at the source coder output. In classical speech transmission a human is the recipient, and erroneous data are reconstructed so as to reduce the subjectively annoying effect of corrupted bits or lost packets. Here, however, a statistical classifier is at the receiving end, which can benefit from knowledge about the quality of the reconstruction. In this book chapter we show how the classical Bayesian decision rule needs to be modified to account for uncertain features, and illustrate how the required feature posterior density can be estimated in the case of distributed speech recognition. Some other techniques for error concealment can be related to this approach. Experimental results are given for both a small and a medium vocabulary recognition task and both for a channel exhibiting bit errors and a packet erasure channel.

A Novel Uncertainty Decoding Rule With Applications to Transmission Error Robust Speech Recognition - Ion, Valentin; Haeb-Umbach, Reinhold (IEEE Transactions on Audio, Speech, and Language Processing)
In this paper, we derive an uncertainty decoding rule for automatic speech recognition (ASR), which accounts for both corrupted observations and inter-frame correlation. The conditional independence assumption, prevalent in hidden Markov model-based ASR, is relaxed to obtain a clean speech posterior that is conditioned on the complete observed feature vector sequence. This is a more informative posterior than one conditioned only on the current observation. The novel decoding is used to obtain a transmission-error robust remote ASR system, where the speech capturing unit is connected to the decoder via an error-prone communication network. We show how the clean speech posterior can be computed for communication links being characterized by either bit errors or packet loss. Recognition results are presented for both distributed and network speech recognition, where in the latter case common voice-over-IP codecs are employed.
Keywords: automatic speech recognition; bit errors; codecs; communication links; corrupted observations; decoding; distributed speech recognition; error-prone communication network; feature vector sequence; hidden Markov model-based ASR; hidden Markov models; inter-frame correlation; Internet telephony; network speech recognition; packet loss; speech posterior; speech recognition; transmission error robust speech recognition; uncertainty decoding; voice-over-IP codecs

Investigations into Uncertainty Decoding Employing a Discrete Feature Space for Noise Robust Automatic Speech Recognition - Ion, Valentin; Haeb-Umbach, Reinhold (2008 ITG Conference on Voice Communication (SprachKommunikation))
This paper addresses the robustness of automatic speech recognition to environmental noise. In order to account for reliability of the clean feature estimate we employ the feature posterior density conditioned on observed noisy features to perform uncertainty decoding. We investigate two approaches to estimate the posterior using a discrete feature space, first conditioning only on the current observation, and second on the whole feature sequence of an utterance. Experiments with Aurora 2 showed that the latter provides slightly better performance, as it allows for exploiting the temporal correlations between consecutive features.

Blinde Akustische Strahlformung fuer Anwendungen im KFZ - Krueger, Alexander; Warsitz, Ernst; Haeb-Umbach, Reinhold (34. Deutsche Jahrestagung fuer Akustik (DAGA 2008))
In diesem Beitrag werden zwei neuartige akustische Strahlformungsalgorithmen fuer einen Einsatz im KFZ diskutiert. In beiden Verfahren ist fuer jede Frequenzkomponente der Eigenvektor zum groessten Eigenwert eines verallgemeinerten Eigenwertproblems zu bestimmen: bei der einen Varianate stellen die Filterkoeffizienten gerade obigen Eigenvektor dar. Sprachverzerrungen werden mit einem einkanaligen Nachfilter kompensiert. In der anderen Variante, welche die Struktur eines ''Generalized Sidelobe Cancellers'' (GSC) hat, basiert die Blockiermatrix auf obiger Eigenvektorzerlegung.