TY - GEN AB - Syntactic annotation of corpora in the form of part-of-speech (POS) tags is a key requirement for both linguistic research and subsequent automated natural language processing (NLP) tasks. This problem is commonly tackled using machine learning methods, i.e., by training a POS tagger on a sufficiently large corpus of labeled data. While the problem of POS tagging can essentially be considered as solved for modern languages, historical corpora turn out to be much more difficult, especially due to the lack of native speakers and sparsity of training data. Moreover, most texts have no sentences as we know them today, nor a common orthography. These irregularities render the task of automated POS tagging more difficult and error-prone. Under these circumstances, instead of forcing the POS tagger to predict and commit to a single tag, it should be enabled to express its uncertainty. In this paper, we consider POS tagging within the framework of set-valued prediction, which allows the POS tagger to express its uncertainty via predicting a set of candidate POS tags instead of guessing a single one. The goal is to guarantee a high confidence that the correct POS tag is included while keeping the number of candidates small. In our experimental study, we find that extending state-of-the-art POS taggers to set-valued prediction yields more precise and robust taggings, especially for unknown words, i.e., words not occurring in the training data. AU - Heid, Stefan Helmut AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 17605 T2 - Journal of Data Mining and Digital Humanities TI - Reliable Part-of-Speech Tagging of Historical Corpora through Set-Valued Prediction ER - TY - CONF AU - Tornede, Alexander AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 20306 T2 - Workshop MetaLearn 2020 @ NeurIPS 2020 TI - Towards Meta-Algorithm Selection ER - TY - CHAP AU - El Mesaoudi-Paul, Adil AU - Weiß, Dimitri AU - Bengs, Viktor AU - Hüllermeier, Eyke AU - Tierney, Kevin ID - 18014 SN - 0302-9743 T2 - Learning and Intelligent Optimization. LION 2020. TI - Pool-Based Realtime Algorithm Configuration: A Preselection Bandit Approach VL - 12096 ER - TY - GEN AB - We consider an extension of the contextual multi-armed bandit problem, in which, instead of selecting a single alternative (arm), a learner is supposed to make a preselection in the form of a subset of alternatives. More specifically, in each iteration, the learner is presented a set of arms and a context, both described in terms of feature vectors. The task of the learner is to preselect $k$ of these arms, among which a final choice is made in a second step. In our setup, we assume that each arm has a latent (context-dependent) utility, and that feedback on a preselection is produced according to a Plackett-Luce model. We propose the CPPL algorithm, which is inspired by the well-known UCB algorithm, and evaluate this algorithm on synthetic and real data. In particular, we consider an online algorithm selection scenario, which served as a main motivation of our problem setting. Here, an instance (which defines the context) from a certain problem class (such as SAT) can be solved by different algorithms (the arms), but only $k$ of these algorithms can actually be run. AU - El Mesaoudi-Paul, Adil AU - Bengs, Viktor AU - Hüllermeier, Eyke ID - 18017 T2 - arXiv:2002.04275 TI - Online Preselection with Context Information under the Plackett-Luce Model ER - TY - CONF AB - Algorithm selection (AS) deals with the automatic selection of an algorithm from a fixed set of candidate algorithms most suitable for a specific instance of an algorithmic problem class, where "suitability" often refers to an algorithm's runtime. Due to possibly extremely long runtimes of candidate algorithms, training data for algorithm selection models is usually generated under time constraints in the sense that not all algorithms are run to completion on all instances. Thus, training data usually comprises censored information, as the true runtime of algorithms timed out remains unknown. However, many standard AS approaches are not able to handle such information in a proper way. On the other side, survival analysis (SA) naturally supports censored data and offers appropriate ways to use such data for learning distributional models of algorithm runtime, as we demonstrate in this work. We leverage such models as a basis of a sophisticated decision-theoretic approach to algorithm selection, which we dub Run2Survive. Moreover, taking advantage of a framework of this kind, we advocate a risk-averse approach to algorithm selection, in which the avoidance of a timeout is given high priority. In an extensive experimental study with the standard benchmark ASlib, our approach is shown to be highly competitive and in many cases even superior to state-of-the-art AS approaches. AU - Tornede, Alexander AU - Wever, Marcel Dominik AU - Werner, Stefan AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 18276 T2 - ACML 2020 TI - Run2Survive: A Decision-theoretic Approach to Algorithm Selection based on Survival Analysis ER - TY - JOUR AU - Richter, Cedric AU - Hüllermeier, Eyke AU - Jakobs, Marie-Christine AU - Wehrheim, Heike ID - 16725 JF - Journal of Automated Software Engineering TI - Algorithm Selection for Software Validation Based on Graph Kernels ER - TY - CONF AB - In multi-label classification (MLC), each instance is associated with a set of class labels, in contrast to standard classification where an instance is assigned a single label. Binary relevance (BR) learning, which reduces a multi-label to a set of binary classification problems, one per label, is arguably the most straight-forward approach to MLC. In spite of its simplicity, BR proved to be competitive to more sophisticated MLC methods, and still achieves state-of-the-art performance for many loss functions. Somewhat surprisingly, the optimal choice of the base learner for tackling the binary classification problems has received very little attention so far. Taking advantage of the label independence assumption inherent to BR, we propose a label-wise base learner selection method optimizing label-wise macro averaged performance measures. In an extensive experimental evaluation, we find that or approach, called LiBRe, can significantly improve generalization performance. AU - Wever, Marcel Dominik AU - Tornede, Alexander AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 15629 TI - LiBRe: Label-Wise Selection of Base Learners in Binary Relevance for Multi-Label Classification ER - TY - JOUR AB - In software engineering, the imprecise requirements of a user are transformed to a formal requirements specification during the requirements elicitation process. This process is usually guided by requirements engineers interviewing the user. We want to partially automate this first step of the software engineering process in order to enable users to specify a desired software system on their own. With our approach, users are only asked to provide exemplary behavioral descriptions. The problem of synthesizing a requirements specification from examples can partially be reduced to the problem of grammatical inference, to which we apply an active coevolutionary learning approach. However, this approach would usually require many feedback queries to be sent to the user. In this work, we extend and generalize our active learning approach to receive knowledge from multiple oracles, also known as proactive learning. The ‘user oracle’ represents input received from the user and the ‘knowledge oracle’ represents available, formalized domain knowledge. We call our two-oracle approach the ‘first apply knowledge then query’ (FAKT/Q) algorithm. We compare FAKT/Q to the active learning approach and provide an extensive benchmark evaluation. As result we find that the number of required user queries is reduced and the inference process is sped up significantly. Finally, with so-called On-The-Fly Markets, we present a motivation and an application of our approach where such knowledge is available. AU - Wever, Marcel Dominik AU - van Rooijen, Lorijn AU - Hamann, Heiko ID - 15025 IS - 2 JF - Evolutionary Computation TI - Multi-Oracle Coevolutionary Learning of Requirements Specifications from Examples in On-The-Fly Markets VL - 28 ER - TY - GEN AB - We study the problem of learning choice functions, which play an important role in various domains of application, most notably in the field of economics. Formally, a choice function is a mapping from sets to sets: Given a set of choice alternatives as input, a choice function identifies a subset of most preferred elements. Learning choice functions from suitable training data comes with a number of challenges. For example, the sets provided as input and the subsets produced as output can be of any size. Moreover, since the order in which alternatives are presented is irrelevant, a choice function should be symmetric. Perhaps most importantly, choice functions are naturally context-dependent, in the sense that the preference in favor of an alternative may depend on what other options are available. We formalize the problem of learning choice functions and present two general approaches based on two representations of context-dependent utility functions. Both approaches are instantiated by means of appropriate neural network architectures, and their performance is demonstrated on suitable benchmark tasks. AU - Pfannschmidt, Karlson AU - Gupta, Pritha AU - Hüllermeier, Eyke ID - 19523 T2 - arXiv:1901.10860 TI - Learning Choice Functions: Concepts and Architectures ER - TY - JOUR AU - Merten, Marie-Luis AU - Seemann, Nina AU - Wever, Marcel Dominik ID - 17565 IS - 142 JF - Niederdeutsches Jahrbuch TI - Grammatikwandel digital-kulturwissenschaftlich erforscht. Mittelniederdeutscher Sprachausbau im interdisziplinären Zugriff ER - TY - GEN AB - A common statistical task lies in showing asymptotic normality of certain statistics. In many of these situations, classical textbook results on weak convergence theory suffice for the problem at hand. However, there are quite some scenarios where stronger results are needed in order to establish an asymptotic normal approximation uniformly over a family of probability measures. In this note we collect some results in this direction. We restrict ourselves to weak convergence in $\mathbb R^d$ with continuous limit measures. AU - Bengs, Viktor AU - Holzmann, Hajo ID - 18018 T2 - arXiv:1903.09864 TI - Uniform approximation in classical weak convergence theory ER - TY - GEN AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke AU - Hetzer, Alexander ID - 8868 TI - Towards Automated Machine Learning for Multi-Label Classification ER - TY - JOUR AU - Tagne, V. K. AU - Fotso, S. AU - Fono, L. A. AU - Hüllermeier, Eyke ID - 10578 IS - 2 JF - New Mathematics and Natural Computation TI - Choice Functions Generated by Mallows and Plackett–Luce Relations VL - 15 ER - TY - JOUR AU - Couso, Ines AU - Borgelt, Christian AU - Hüllermeier, Eyke AU - Kruse, Rudolf ID - 15001 JF - IEEE Computational Intelligence Magazine SN - 1556-603X TI - Fuzzy Sets in Data Analysis: From Statistical Foundations to Machine Learning ER - TY - JOUR AB - Many problem settings in machine learning are concerned with the simultaneous prediction of multiple target variables of diverse type. Amongst others, such problem settings arise in multivariate regression, multi-label classification, multi-task learning, dyadic prediction, zero-shot learning, network inference, and matrix completion. These subfields of machine learning are typically studied in isolation, without highlighting or exploring important relationships. In this paper, we present a unifying view on what we call multi-target prediction (MTP) problems and methods. First, we formally discuss commonalities and differences between existing MTP problems. To this end, we introduce a general framework that covers the above subfields as special cases. As a second contribution, we provide a structured overview of MTP methods. This is accomplished by identifying a number of key properties, which distinguish such methods and determine their suitability for different types of problems. Finally, we also discuss a few challenges for future research. AU - Waegeman, Willem AU - Dembczynski, Krzysztof AU - Hüllermeier, Eyke ID - 15002 IS - 2 JF - Data Mining and Knowledge Discovery SN - 1573-756X TI - Multi-target prediction: a unifying view on problems and methods VL - 33 ER - TY - CONF AU - Mortier, Thomas AU - Wydmuch, Marek AU - Dembczynski, Krzysztof AU - Hüllermeier, Eyke AU - Waegeman, Willem ID - 15003 T2 - Proceedings of the 31st Benelux Conference on Artificial Intelligence {(BNAIC} 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn 2019), Brussels, Belgium, November 6-8, 2019 TI - Set-Valued Prediction in Multi-Class Classification ER - TY - CHAP AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 15004 SN - 0302-9743 T2 - Discovery Science TI - Feature Selection for Analogy-Based Learning to Rank ER - TY - CHAP AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 15005 SN - 0302-9743 T2 - KI 2019: Advances in Artificial Intelligence TI - Analogy-Based Preference Learning with Kernels ER - TY - CHAP AU - Nguyen, Vu-Linh AU - Destercke, Sébastien AU - Hüllermeier, Eyke ID - 15006 SN - 0302-9743 T2 - Discovery Science TI - Epistemic Uncertainty Sampling ER - TY - CONF AU - Melnikov, Vitaly AU - Hüllermeier, Eyke ID - 15007 T2 - Proceedings ACML, Asian Conference on Machine Learning (Proceedings of Machine Learning Research, 101) TI - Learning to Aggregate: Tackling the Aggregation/Disaggregation Problem for OWA ER -