TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings - Boeddeker, Christoph; Subramanian, Aswin Shanmugam; Wichern, Gordon; Haeb-Umbach, Reinhold; Le Roux, Jonathan (IEEE/ACM Transactions on Audio, Speech, and Language Processing)
Keywords: Electrical and Electronic Engineering; Acoustics and Ultrasonics; Computer Science (miscellaneous); Computational Mathematics

Geodesic Interpolation of Frame-Wise Speaker Embeddings for the Diarization of Meeting Scenarios - Cord-Landwehr, Tobias; Boeddeker, Christoph; Zorilă, Cătălin; Doddipatla, Rama; Haeb-Umbach, Reinhold (ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

Frame-Wise and Overlap-Robust Speaker Embeddings for Meeting Diarization - Cord-Landwehr, Tobias; Boeddeker, Christoph; Zorilă, Cătălin; Doddipatla, Rama; Haeb-Umbach, Reinhold (ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

A Teacher-Student Approach for Extracting Informative Speaker Embeddings From Speech Mixtures - Cord-Landwehr, Tobias; Boeddeker, Christoph; Zorilă, Cătălin; Doddipatla, Rama; Haeb-Umbach, Reinhold (INTERSPEECH 2023)

Reverberation as Supervision For Speech Separation - Aralikatti, Rohith; Boeddeker, Christoph; Wichern, Gordon; Subramanian, Aswin; Le Roux, Jonathan (ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

Mixture Encoder for Joint Speech Separation and Recognition - Berger, Simon; Vieting, Peter; Boeddeker, Christoph; Schlüter, Ralf; Haeb-Umbach, Reinhold (INTERSPEECH 2023)

Segment-Less Continuous Speech Separation of Meetings: Training and Evaluation Criteria - von Neumann, Thilo; Kinoshita, Keisuke; Boeddeker, Christoph; Delcroix, Marc; Haeb-Umbach, Reinhold (IEEE/ACM Transactions on Audio, Speech, and Language Processing)
Continuous Speech Separation (CSS) has been proposed to address speech overlaps during the analysis of realistic meeting-like conversations by eliminating any overlaps before further processing. CSS separates a recording of arbitrarily many speakers into a small number of overlap-free output channels, where each output channel may contain speech of multiple speakers. This is often done by applying a conventional separation model trained with Utterance-level Permutation Invariant Training (uPIT), which exclusively maps a speaker to an output channel, in sliding window approach called stitching. Recently, we introduced an alternative training scheme called Graph-PIT that teaches the separation network to directly produce output streams in the required format without stitching. It can handle an arbitrary number of speakers as long as never more of them overlap at the same time than the separator has output channels. In this contribution, we further investigate the Graph-PIT training scheme. We show in extended experiments that models trained with Graph-PIT also work in challenging reverberant conditions. Models trained in this way are able to perform segment-less CSS, i.e., without stitching, and achieve comparable and often better separation quality than the conventional CSS with uPIT and stitching. We simplify the training schedule for Graph-PIT with the recently proposed Source Aggregated Signal-to-Distortion Ratio (SA-SDR) loss. It eliminates unfavorable properties of the previously used A-SDR loss and thus enables training with Graph-PIT from scratch. Graph-PIT training relaxes the constraints w.r.t. the allowed numbers of speakers and speaking patterns which allows using a larger variety of training data. Furthermore, we introduce novel signal-level evaluation metrics for meeting scenarios, namely the source-aggregated scale- and convolution-invariant Signal-to-Distortion Ratio (SA-SI-SDR and SA-CI-SDR), which are generalizations of the commonly used SDR-based metrics for the CSS case.
Keywords: Continuous Speech Separation; Source Separation; Graph-PIT; Dynamic Programming; Permutation Invariant Training

On Word Error Rate Definitions and Their Efficient Computation for Multi-Speaker Speech Recognition Systems - von Neumann, Thilo; Boeddeker, Christoph; Kinoshita, Keisuke; Delcroix, Marc; Haeb-Umbach, Reinhold (ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))
We propose a general framework to compute the word error rate (WER) of ASR systems that process recordings containing multiple speakers at their input and that produce multiple output word sequences (MIMO). Such ASR systems are typically required, e.g., for meeting transcription. We provide an efficient implementation based on a dynamic programming search in a multi-dimensional Levenshtein distance tensor under the constraint that a reference utterance must be matched consistently with one hypothesis output. This also results in an efficient implementation of the ORC WER which previously suffered from exponential complexity. We give an overview of commonly used WER definitions for multi-speaker scenarios and show that they are specializations of the above MIMO WER tuned to particular application scenarios. We conclude with a discussion of the pros and cons of the various WER definitions and a recommendation when to use which.
Keywords: Word Error Rate; Meeting Recognition; Levenshtein Distance

MeetEval: A Toolkit for Computation of Word Error Rates for Meeting Transcription Systems - von Neumann, Thilo; Boeddeker, Christoph; Delcroix, Marc; Haeb-Umbach, Reinhold (Proc. CHiME 2023 Workshop on Speech Processing in Everyday Environments)
MeetEval is an open-source toolkit to evaluate all kinds of meeting transcription systems. It provides a unified interface for the computation of commonly used Word Error Rates (WERs), specifically cpWER, ORC WER and MIMO WER along other WER definitions. We extend the cpWER computation by a temporal constraint to ensure that only words are identified as correct when the temporal alignment is plausible. This leads to a better quality of the matching of the hypothesis string to the reference string that more closely resembles the actual transcription quality, and a system is penalized if it provides poor time annotations. Since word-level timing information is often not available, we present a way to approximate exact word-level timings from segment-level timings (e.g., a sentence) and show that the approximation leads to a similar WER as a matching with exact word-level annotations. At the same time, the time constraint leads to a speedup of the matching algorithm, which outweighs the additional overhead caused by processing the time stamps.
Keywords: Speech Recognition; Word Error Rate; Meeting Transcription

End-to-End Dereverberation, Beamforming, and Speech Recognition in A Cocktail Party - Zhang, Wangyou; Chang, Xuankai; Boeddeker, Christoph; Nakatani, Tomohiro; Watanabe, Shinji; Qian, Yanmin (IEEE/ACM Transactions on Audio, Speech, and Language Processing)
Far-field multi-speaker automatic speech recognition (ASR) has drawn increasing attention in recent years. Most existing methods feature a signal processing frontend and an ASR backend. In realistic scenarios, these modules are usually trained separately or progressively, which suffers from either inter-module mismatch or a complicated training process. In this paper, we propose an end-to-end multi-channel model that jointly optimizes the speech enhancement (including speech dereverberation, denoising, and separation) frontend and the ASR backend as a single system. To the best of our knowledge, this is the first work that proposes to optimize dereverberation, beamforming, and multi-speaker ASR in a fully end-to-end manner. The frontend module consists of a weighted prediction error (WPE) based submodule for dereverberation and a neural beamformer for denoising and speech separation. For the backend, we adopt a widely used end-to-end (E2E) ASR architecture. It is worth noting that the entire model is differentiable and can be optimized in a fully end-to-end manner using only the ASR criterion, without the need of parallel signal-level labels. We evaluate the proposed model on several multi-speaker benchmark datasets, and experimental results show that the fully E2E ASR model can achieve competitive performance on both noisy and reverberant conditions, with over 30% relative word error rate (WER) reduction over the single-channel baseline systems.

An Initialization Scheme for Meeting Separation with Spatial Mixture Models - Boeddeker, Christoph; Cord-Landwehr, Tobias; von Neumann, Thilo; Haeb-Umbach, Reinhold (Interspeech 2022)

Utterance-by-utterance overlap-aware neural diarization with Graph-PIT - Kinoshita, Keisuke; von Neumann, Thilo; Delcroix, Marc; Boeddeker, Christoph; Haeb-Umbach, Reinhold (Proc. Interspeech 2022)
Recent speaker diarization studies showed that integration of end-to-end neural diarization (EEND) and clustering-based diarization is a promising approach for achieving state-of-the-art performance on various tasks. Such an approach first divides an observed signal into fixed-length segments, then performs {\it segment-level} local diarization based on an EEND module, and merges the segment-level results via clustering to form a final global diarization result. The segmentation is done to limit the number of speakers in each segment since the current EEND cannot handle a large number of speakers. In this paper, we argue that such an approach involving the segmentation has several issues; for example, it inevitably faces a dilemma that larger segment sizes increase both the context available for enhancing the performance and the number of speakers for the local EEND module to handle. To resolve such a problem, this paper proposes a novel framework that performs diarization without segmentation. However, it can still handle challenging data containing many speakers and a significant amount of overlapping speech. The proposed method can take an entire meeting for inference and perform {\it utterance-by-utterance} diarization that clusters utterance activities in terms of speakers. To this end, we leverage a neural network training scheme called Graph-PIT proposed recently for neural source separation. Experiments with simulated active-meeting-like data and CALLHOME data show the superiority of the proposed approach over the conventional methods.

SA-SDR: A Novel Loss Function for Separation of Meeting Style Data - von Neumann, Thilo; Kinoshita, Keisuke; Boeddeker, Christoph; Delcroix, Marc; Haeb-Umbach, Reinhold (ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

MMS-MSG: A Multi-purpose Multi-Speaker Mixture Signal Generator - Cord-Landwehr, Tobias; von Neumann, Thilo; Boeddeker, Christoph; Haeb-Umbach, Reinhold (2022 International Workshop on Acoustic Signal Enhancement (IWAENC))
The scope of speech enhancement has changed from a monolithic view of single, independent tasks, to a joint processing of complex conversational speech recordings. Training and evaluation of these single tasks requires synthetic data with access to intermediate signals that is as close as possible to the evaluation scenario. As such data often is not available, many works instead use specialized databases for the training of each system component, e.g WSJ0-mix for source separation. We present a Multi-purpose Multi-Speaker Mixture Signal Generator (MMS-MSG) for generating a variety of speech mixture signals based on any speech corpus, ranging from classical anechoic mixtures (e.g., WSJ0-mix) over reverberant mixtures (e.g., SMS-WSJ) to meeting-style data. Its highly modular and flexible structure allows for the simulation of diverse environments and dynamic mixing, while simultaneously enabling an easy extension and modification to generate new scenarios and mixture types. These meetings can be used for prototyping, evaluation, or training purposes. We provide example evaluation data and baseline results for meetings based on the WSJ corpus. Further, we demonstrate the usefulness for realistic scenarios by using MMS-MSG to provide training data for the LibriCSS database.

Monaural source separation: From anechoic to reverberant environments - Cord-Landwehr, Tobias; Boeddeker, Christoph; von Neumann, Thilo; Zorila, Catalin; Doddipatla, Rama; Haeb-Umbach, Reinhold (2022 International Workshop on Acoustic Signal Enhancement (IWAENC))
Impressive progress in neural network-based single-channel speech source separation has been made in recent years. But those improvements have been mostly reported on anechoic data, a situation that is hardly met in practice. Taking the SepFormer as a starting point, which achieves state-of-the-art performance on anechoic mixtures, we gradually modify it to optimize its performance on reverberant mixtures. Although this leads to a word error rate improvement by 7 percentage points compared to the standard SepFormer implementation, the system ends up with only marginally better performance than a PIT-BLSTM separation system, that is optimized with rather straightforward means. This is surprising and at the same time sobering, challenging the practical usefulness of many improvements reported in recent years for monaural source separation on nonreverberant data.

A Meeting Transcription System for an Ad-Hoc Acoustic Sensor Network - Gburrek, Tobias; Boeddeker, Christoph; von Neumann, Thilo; Cord-Landwehr, Tobias; Schmalenstroeer, Joerg; Haeb-Umbach, Reinhold

End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend - Zhang, Wangyou; Boeddeker, Christoph; Watanabe, Shinji; Nakatani, Tomohiro; Delcroix, Marc; Kinoshita, Keisuke; Ochiai, Tsubasa; Kamo, Naoyuki; Haeb-Umbach, Reinhold; Qian, Yanmin (ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

ESPnet-SE: End-To-End Speech Enhancement and Separation Toolkit Designed for ASR Integration - Li, Chenda; Shi, Jing; Zhang, Wangyou; Subramanian, Aswin Shanmugam; Chang, Xuankai; Kamo, Naoyuki; Hira, Moto; Hayashi, Tomoki; Boeddeker, Christoph; Chen, Zhuo; Watanabe, Shinji (2021 IEEE Spoken Language Technology Workshop (SLT))

Dual-Path RNN for Long Recording Speech Separation - Li, Chenda; Luo, Yi; Han, Cong; Li, Jinyu; Yoshioka, Takuya; Zhou, Tianyan; Delcroix, Marc; Kinoshita, Keisuke; Boeddeker, Christoph; Qian, Yanmin; Watanabe, Shinji; Chen, Zhuo (2021 IEEE Spoken Language Technology Workshop (SLT))

A Comparison and Combination of Unsupervised Blind Source Separation Techniques - Boeddeker, Christoph; Rautenberg, Frederik; Haeb-Umbach, Reinhold (ITG Conference on Speech Communication)
Unsupervised blind source separation methods do not require a training phase and thus cannot suffer from a train-test mismatch, which is a common concern in neural network based source separation. The unsupervised techniques can be categorized in two classes, those building upon the sparsity of speech in the Short-Time Fourier transform domain and those exploiting non-Gaussianity or non-stationarity of the source signals. In this contribution, spatial mixture models which fall in the first category and independent vector analysis (IVA) as a representative of the second category are compared w.r.t. their separation performance and the performance of a downstream speech recognizer on a reverberant dataset of reasonable size. Furthermore, we introduce a serial concatenation of the two, where the result of the mixture model serves as initialization of IVA, which achieves significantly better WER performance than each algorithm individually and even approaches the performance of a much more complex neural network based technique.

Convolutive Transfer Function Invariant SDR Training Criteria for Multi-Channel Reverberant Speech Separation - Boeddeker, Christoph; Zhang, Wangyou; Nakatani, Tomohiro; Kinoshita, Keisuke; Ochiai, Tsubasa; Delcroix, Marc; Kamo, Naoyuki; Qian, Yanmin; Haeb-Umbach, Reinhold (ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

Graph-PIT: Generalized Permutation Invariant Training for Continuous Separation of Arbitrary Numbers of Speakers - von Neumann, Thilo; Kinoshita, Keisuke; Boeddeker, Christoph; Delcroix, Marc; Haeb-Umbach, Reinhold (Interspeech 2021)
Automatic transcription of meetings requires handling of overlapped speech, which calls for continuous speech separation (CSS) systems. The uPIT criterion was proposed for utterance-level separation with neural networks and introduces the constraint that the total number of speakers must not exceed the number of output channels. When processing meeting-like data in a segment-wise manner, i.e., by separating overlapping segments independently and stitching adjacent segments to continuous output streams, this constraint has to be fulfilled for any segment. In this contribution, we show that this constraint can be significantly relaxed. We propose a novel graph-based PIT criterion, which casts the assignment of utterances to output channels in a graph coloring problem. It only requires that the number of concurrently active speakers must not exceed the number of output channels. As a consequence, the system can process an arbitrary number of speakers and arbitrarily long segments and thus can handle more diverse scenarios. Further, the stitching algorithm for obtaining a consistent output order in neighboring segments is of less importance and can even be eliminated completely, not the least reducing the computational effort. Experiments on meeting-style WSJ data show improvements in recognition performance over using the uPIT criterion.
Keywords: Continuous speech separation; automatic speech recognition; overlapped speech; permutation invariant training

Speeding Up Permutation Invariant Training for Source Separation - von Neumann, Thilo; Boeddeker, Christoph; Kinoshita, Keisuke; Delcroix, Marc; Haeb-Umbach, Reinhold (Speech Communication; 14th ITG Conference)

Towards a speaker diarization system for the CHiME 2020 dinner party transcription - Boeddeker, Christoph; Cord-Landwehr, Tobias; Heitkaemper, Jens; Zorila, Catalin; Hayakawa, Daichi; Li, Mohan; Liu, Min; Doddipatla, Rama; Haeb-Umbach, Reinhold (Proc. CHiME 2020 Workshop on Speech Processing in Everyday Environments)

Jointly optimal denoising, dereverberation, and source separation - Nakatani, Tomohiro; Boeddeker, Christoph; Kinoshita, Keisuke; Ikeshita, Rintaro; Delcroix, Marc; Haeb-Umbach, Reinhold (IEEE/ACM Transactions on Audio, Speech, and Language Processing)

Demystifying TasNet: A Dissecting Approach - Heitkaemper, Jens; Jakobeit, Darius; Boeddeker, Christoph; Drude, Lukas; Haeb-Umbach, Reinhold (ICASSP 2020 Virtual Barcelona Spain)
In recent years time domain speech separation has excelled over frequency domain separation in single channel scenarios and noise-free environments. In this paper we dissect the gains of the time-domain audio separation network (TasNet) approach by gradually replacing components of an utterance-level permutation invariant training (u-PIT) based separation system in the frequency domain until the TasNet system is reached, thus blending components of frequency domain approaches with those of time domain approaches. Some of the intermediate variants achieve comparable signal-to-distortion ratio (SDR) gains to TasNet, but retain the advantage of frequency domain processing: compatibility with classic signal processing tools such as frequency-domain beamforming and the human interpretability of the masks. Furthermore, we show that the scale invariant signal-to-distortion ratio (si-SDR) criterion used as loss function in TasNet is related to a logarithmic mean square error criterion and that it is this criterion which contributes most reliable to the performance advantage of TasNet. Finally, we critically assess which gains in a noise-free single channel environment generalize to more realistic reverberant conditions.
Keywords: voice activity detection; speech activity detection; neural network; statistical speech processing

CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings - Watanabe, Shinji; Mandel, Michael; Barker, Jon; Vincent, Emmanuel; Arora, Ashish; Chang, Xuankai; Khudanpur, Sanjeev; Manohar, Vimal; Povey, Daniel; Raj, Desh; Snyder, David; Subramanian, Aswin Shanmugam; et al. (arXiv:2004.09249)
Following the success of the 1st, 2nd, 3rd, 4th and 5th CHiME challenges we organize the 6th CHiME Speech Separation and Recognition Challenge (CHiME-6). The new challenge revisits the previous CHiME-5 challenge and further considers the problem of distant multi-microphone conversational speech diarization and recognition in everyday home environments. Speech material is the same as the previous CHiME-5 recordings except for accurate array synchronization. The material was elicited using a dinner party scenario with efforts taken to capture data that is representative of natural conversational speech. This paper provides a baseline description of the CHiME-6 challenge for both segmented multispeaker speech recognition (Track 1) and unsegmented multispeaker speech recognition (Track 2). Of note, Track 2 is the first challenge activity in the community to tackle an unsegmented multispeaker speech recognition scenario with a complete set of reproducible open source baselines providing speech enhancement, speaker diarization, and speech recognition modules.

End-to-End Training of Time Domain Audio Separation and Recognition - von Neumann, Thilo; Kinoshita, Keisuke; Drude, Lukas; Boeddeker, Christoph; Delcroix, Marc; Nakatani, Tomohiro; Haeb-Umbach, Reinhold (ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))
The rising interest in single-channel multi-speaker speech separation sparked development of End-to-End (E2E) approaches to multispeaker speech recognition. However, up until now, state-of-theart neural network–based time domain source separation has not yet been combined with E2E speech recognition. We here demonstrate how to combine a separation module based on a Convolutional Time domain Audio Separation Network (Conv-TasNet) with an E2E speech recognizer and how to train such a model jointly by distributing it over multiple GPUs or by approximating truncated back-propagation for the convolutional front-end. To put this work into perspective and illustrate the complexity of the design space, we provide a compact overview of single-channel multi-speaker recognition systems. Our experiments show a word error rate of 11.0% on WSJ0-2mix and indicate that our joint time domain model can yield substantial improvements over cascade DNN-HMM and monolithic E2E frequency domain systems proposed so far.

Multi-Talker ASR for an Unknown Number of Sources: Joint Training of Source Counting, Separation and ASR - von Neumann, Thilo; Boeddeker, Christoph; Drude, Lukas; Kinoshita, Keisuke; Delcroix, Marc; Nakatani, Tomohiro; Haeb-Umbach, Reinhold (Proc. Interspeech 2020)
Most approaches to multi-talker overlapped speech separation and recognition assume that the number of simultaneously active speakers is given, but in realistic situations, it is typically unknown. To cope with this, we extend an iterative speech extraction system with mechanisms to count the number of sources and combine it with a single-talker speech recognizer to form the first end-to-end multi-talker automatic speech recognition system for an unknown number of active speakers. Our experiments show very promising performance in counting accuracy, source separation and speech recognition on simulated clean mixtures from WSJ0-2mix and WSJ0-3mix. Among others, we set a new state-of-the-art word error rate on the WSJ0-2mix database. Furthermore, our system generalizes well to a larger number of speakers than it ever saw during training, as shown in experiments with the WSJ0-4mix database.

SMS-WSJ: Database, performance measures, and baseline recipe for multi-channel source separation and recognition - Drude, Lukas; Heitkaemper, Jens; Boeddeker, Christoph; Haeb-Umbach, Reinhold (ArXiv e-prints)
We present a multi-channel database of overlapping speech for training, evaluation, and detailed analysis of source separation and extraction algorithms: SMS-WSJ -- Spatialized Multi-Speaker Wall Street Journal. It consists of artificially mixed speech taken from the WSJ database, but unlike earlier databases we consider all WSJ0+1 utterances and take care of strictly separating the speaker sets present in the training, validation and test sets. When spatializing the data we ensure a high degree of randomness w.r.t. room size, array center and rotation, as well as speaker position. Furthermore, this paper offers a critical assessment of recently proposed measures of source separation performance. Alongside the code to generate the database we provide a source separation baseline and a Kaldi recipe with competitive word error rates to provide common ground for evaluation.

An Investigation Into the Effectiveness of Enhancement in ASR Training and Test for Chime-5 Dinner Party Transcription - Zorila, Catalin; Boeddeker, Christoph; Doddipatla, Rama; Haeb-Umbach, Reinhold (ASRU 2019, Sentosa, Singapore)
Despite the strong modeling power of neural network acoustic models, speech enhancement has been shown to deliver additional word error rate improvements if multi-channel data is available. However, there has been a longstanding debate whether enhancement should also be carried out on the ASR training data. In an extensive experimental evaluation on the acoustically very challenging CHiME-5 dinner party data we show that: (i) cleaning up the training data can lead to substantial error rate reductions, and (ii) enhancement in training is advisable as long as enhancement in test is at least as strong as in training. This approach stands in contrast and delivers larger gains than the common strategy reported in the literature to augment the training database with additional artificially degraded speech. Together with an acoustic model topology consisting of initial CNN layers followed by factorized TDNN layers we achieve with 41.6% and 43.2% WER on the DEV and EVAL test sets, respectively, a new single-system state-of-the-art result on the CHiME-5 data. This is a 8% relative improvement compared to the best word error rate published so far for a speech recognizer without system combination.

Guided Source Separation Meets a Strong ASR Backend: Hitachi/Paderborn University Joint Investigation for Dinner Party ASR - Kanda, Naoyuki; Boeddeker, Christoph; Heitkaemper, Jens; Fujita, Yusuke; Horiguchi, Shota; Haeb-Umbach, Reinhold (INTERSPEECH 2019, Graz, Austria)
In this paper, we present Hitachi and Paderborn University’s joint effort for automatic speech recognition (ASR) in a dinner party scenario. The main challenges of ASR systems for dinner party recordings obtained by multiple microphone arrays are (1) heavy speech overlaps, (2) severe noise and reverberation, (3) very natural onversational content, and possibly (4) insufficient training data. As an example of a dinner party scenario, we have chosen the data presented during the CHiME-5 speech recognition challenge, where the baseline ASR had a 73.3% word error rate (WER), and even the best performing system at the CHiME-5 challenge had a 46.1% WER. We extensively investigated a combination of the guided source separation-based speech enhancement technique and an already proposed strong ASR backend and found that a tight combination of these techniques provided substantial accuracy improvements. Our final system achieved WERs of 39.94% and 41.64% for the development and evaluation data, respectively, both of which are the best published results for the dataset. We also investigated with additional training data on the official small data in the CHiME-5 corpus to assess the intrinsic difficulty of this ASR task.

Integration neural network based beamforming and weighted prediction error dereverberation - Drude, Lukas; Boeddeker, Christoph; Heymann, Jahn; Kinoshita, Keisuke; Delcroix, Marc; Nakatani, Tomohiro; Haeb-Umbach, Reinhold (INTERSPEECH 2018, Hyderabad, India)
The weighted prediction error (WPE) algorithm has proven to be a very successful dereverberation method for the REVERB challenge. Likewise, neural network based mask estimation for beamforming demonstrated very good noise suppression in the CHiME 3 and CHiME 4 challenges. Recently, it has been shown that this estimator can also be trained to perform dereverberation and denoising jointly. However, up to now a comparison of a neural beamformer and WPE is still missing, so is an investigation into a combination of the two. Therefore, we here provide an extensive evaluation of both and consequently propose variants to integrate deep neural network based beamforming with WPE. For these integrated variants we identify a consistent word error rate (WER) reduction on two distinct databases. In particular, our study shows that deep learning based beamforming benefits from a model-based dereverberation technique (i.e. WPE) and vice versa. Our key findings are: (a) Neural beamforming yields the lower WERs in comparison to WPE the more channels and noise are present. (b) Integration of WPE and a neural beamformer consistently outperforms all stand-alone systems.

NARA-WPE: A Python package for weighted prediction error dereverberation in Numpy and Tensorflow for online and offline processing - Drude, Lukas; Heymann, Jahn; Boeddeker, Christoph; Haeb-Umbach, Reinhold (ITG 2018, Oldenburg, Germany)
NARA-WPE is a Python software package providing implementations of the weighted prediction error (WPE) dereverberation algorithm. WPE has been shown to be a highly effective tool for speech dereverberation, thus improving the perceptual quality of the signal and improving the recognition performance of downstream automatic speech recognition (ASR). It is suitable both for single-channel and multi-channel applications. The package consist of (1) a Numpy implementation which can easily be integrated into a custom Python toolchain, and (2) a TensorFlow implementation which allows integration into larger computational graphs and enables backpropagation through WPE to train more advanced front-ends. This package comprises of an iterative offline (batch) version, a block-online version, and a frame-online version which can be used in moderately low latency applications, e.g. digital speech assistants.

Exploring Practical Aspects of Neural Mask-Based Beamforming for Far-Field Speech Recognition - Boeddeker, Christoph; Erdogan, Hakan; Yoshioka, Takuya; Haeb-Umbach, Reinhold (ICASSP 2018, Calgary, Canada)
This work examines acoustic beamformers employing neural networks (NNs) for mask prediction as front-end for automatic speech recognition (ASR) systems for practical scenarios like voice-enabled home devices. To test the versatility of the mask predicting network, the system is evaluated with different recording hardware, different microphone array designs, and different acoustic models of the downstream ASR system. Significant gains in recognition accuracy are obtained in all configurations despite the fact that the NN had been trained on mismatched data. Unlike previous work, the NN is trained on a feature level objective, which gives some performance advantage over a mask related criterion. Furthermore, different approaches for realizing online, or adaptive, NN-based beamforming are explored, where the online algorithms still show significant gains compared to the baseline performance.

Front-End Processing for the CHiME-5 Dinner Party Scenario - Boeddeker, Christoph; Heitkaemper, Jens; Schmalenstroeer, Joerg; Drude, Lukas; Heymann, Jahn; Haeb-Umbach, Reinhold (Proc. CHiME 2018 Workshop on Speech Processing in Everyday Environments, Hyderabad, India)
This contribution presents a speech enhancement system for the CHiME-5 Dinner Party Scenario. The front-end employs multi-channel linear time-variant filtering and achieves its gains without the use of a neural network. We present an adaptation of blind source separation techniques to the CHiME-5 database which we call Guided Source Separation (GSS). Using the baseline acoustic and language model, the combination of Weighted Prediction Error based dereverberation, guided source separation, and beamforming reduces the WER by 10:54% (relative) for the single array track and by 21:12% (relative) on the multiple array track.

The RWTH/UPB System Combination for the CHiME 2018 Workshop - Kitza, Markus; Michel, Wilfried; Boeddeker, Christoph; Heitkaemper, Jens; Menne, Tobias; Schlüter, Ralf; Ney, Hermann; Schmalenstroeer, Joerg; Drude, Lukas; Heymann, Jahn; Haeb-Umbach, Reinhold (Proc. CHiME 2018 Workshop on Speech Processing in Everyday Environments, Hyderabad, India)
This paper describes the systems for the single-array track and the multiple-array track of the 5th CHiME Challenge. The final system is a combination of multiple systems, using Confusion Network Combination (CNC). The different systems presented here are utilizing different front-ends and training sets for a Bidirectional Long Short-Term Memory (BLSTM) Acoustic Model (AM). The front-end was replaced by enhancements provided by Paderborn University [1]. The back-end has been implemented using RASR [2] and RETURNN [3]. Additionally, a system combination including the hypothesis word graphs from the system of the submission [1] has been performed, which results in the final best system.

On the Computation of Complex-valued Gradients with Application to Statistically Optimum Beamforming - Boeddeker, Christoph; Hanebrink, Patrick; Drude, Lukas; Heymann, Jahn; Haeb-Umbach, Reinhold
This report describes the computation of gradients by algorithmic differentiation for statistically optimum beamforming operations. Especially the derivation of complex-valued functions is a key component of this approach. Therefore the real-valued algorithmic differentiation is extended via the complex-valued chain rule. In addition to the basic mathematic operations the derivative of the eigenvalue problem with complex-valued eigenvectors is one of the key results of this report. The potential of this approach is shown with experimental results on the CHiME-3 challenge database. There, the beamforming task is used as a front-end for an ASR system. With the developed derivatives a joint optimization of a speech enhancement and speech recognition system w.r.t. the recognition optimization criterion is possible.

Optimizing Neural-Network Supported Acoustic Beamforming by Algorithmic Differentiation - Boeddeker, Christoph; Hanebrink, Patrick; Drude, Lukas; Heymann, Jahn; Haeb-Umbach, Reinhold (Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP))
In this paper we show how a neural network for spectral mask estimation for an acoustic beamformer can be optimized by algorithmic differentiation. Using the beamformer output SNR as the objective function to maximize, the gradient is propagated through the beamformer all the way to the neural network which provides the clean speech and noise masks from which the beamformer coefficients are estimated by eigenvalue decomposition. A key theoretical result is the derivative of an eigenvalue problem involving complex-valued eigenvectors. Experimental results on the CHiME-3 challenge database demonstrate the effectiveness of the approach. The tools developed in this paper are a key component for an end-to-end optimization of speech enhancement and speech recognition.

BEAMNET: End-to-End Training of a Beamformer-Supported Multi-Channel ASR System - Heymann, Jahn; Drude, Lukas; Boeddeker, Christoph; Hanebrink, Patrick; Haeb-Umbach, Reinhold (Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP))
This paper presents an end-to-end training approach for a beamformer-supported multi-channel ASR system. A neural network which estimates masks for a statistically optimum beamformer is jointly trained with a network for acoustic modeling. To update its parameters, we propagate the gradients from the acoustic model all the way through feature extraction and the complex valued beamforming operation. Besides avoiding a mismatch between the front-end and the back-end, this approach also eliminates the need for stereo data, i.e., the parallel availability of clean and noisy versions of the signals. Instead, it can be trained with real noisy multichannel data only. Also, relying on the signal statistics for beamforming, the approach makes no assumptions on the configuration of the microphone array. We further observe a performance gain through joint training in terms of word error rate in an evaluation of the system on the CHiME 4 dataset.

Multi-Stage Coherence Drift Based Sampling Rate Synchronization for Acoustic Beamforming - Schmalenstroeer, Joerg; Heymann, Jahn; Drude, Lukas; Boeddeker, Christoph; Haeb-Umbach, Reinhold (IEEE 19th International Workshop on Multimedia Signal Processing (MMSP))
Multi-channel speech enhancement algorithms rely on a synchronous sampling of the microphone signals. This, however, cannot always be guaranteed, especially if the sensors are distributed in an environment. To avoid performance degradation the sampling rate offset needs to be estimated and compensated for. In this contribution we extend the recently proposed coherence drift based method in two important directions. First, the increasing phase shift in the short-time Fourier transform domain is estimated from the coherence drift in a Matched Filterlike fashion, where intermediate estimates are weighted by their instantaneous SNR. Second, an observed bias is removed by iterating between offset estimation and compensation by resampling a couple of times. The effectiveness of the proposed method is demonstrated by speech recognition results on the output of a beamformer with and without sampling rate offset compensation between the input channels. We compare MVDR and maximum-SNR beamformers in reverberant environments and further show that both benefit from a novel phase normalization, which we also propose in this contribution.

Blind Speech Separation based on Complex Spherical k-Mode Clustering - Drude, Lukas; Boeddeker, Christoph; Haeb-Umbach, Reinhold (Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP))