TY - GEN AB - Syntactic annotation of corpora in the form of part-of-speech (POS) tags is a key requirement for both linguistic research and subsequent automated natural language processing (NLP) tasks. This problem is commonly tackled using machine learning methods, i.e., by training a POS tagger on a sufficiently large corpus of labeled data. While the problem of POS tagging can essentially be considered as solved for modern languages, historical corpora turn out to be much more difficult, especially due to the lack of native speakers and sparsity of training data. Moreover, most texts have no sentences as we know them today, nor a common orthography. These irregularities render the task of automated POS tagging more difficult and error-prone. Under these circumstances, instead of forcing the POS tagger to predict and commit to a single tag, it should be enabled to express its uncertainty. In this paper, we consider POS tagging within the framework of set-valued prediction, which allows the POS tagger to express its uncertainty via predicting a set of candidate POS tags instead of guessing a single one. The goal is to guarantee a high confidence that the correct POS tag is included while keeping the number of candidates small. In our experimental study, we find that extending state-of-the-art POS taggers to set-valued prediction yields more precise and robust taggings, especially for unknown words, i.e., words not occurring in the training data. AU - Heid, Stefan Helmut AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 17605 T2 - Journal of Data Mining and Digital Humanities TI - Reliable Part-of-Speech Tagging of Historical Corpora through Set-Valued Prediction ER - TY - CONF AU - Tornede, Alexander AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 20306 T2 - Workshop MetaLearn 2020 @ NeurIPS 2020 TI - Towards Meta-Algorithm Selection ER - TY - CHAP AU - El Mesaoudi-Paul, Adil AU - Weiß, Dimitri AU - Bengs, Viktor AU - Hüllermeier, Eyke AU - Tierney, Kevin ID - 18014 SN - 0302-9743 T2 - Learning and Intelligent Optimization. LION 2020. TI - Pool-Based Realtime Algorithm Configuration: A Preselection Bandit Approach VL - 12096 ER - TY - GEN AB - We consider an extension of the contextual multi-armed bandit problem, in which, instead of selecting a single alternative (arm), a learner is supposed to make a preselection in the form of a subset of alternatives. More specifically, in each iteration, the learner is presented a set of arms and a context, both described in terms of feature vectors. The task of the learner is to preselect $k$ of these arms, among which a final choice is made in a second step. In our setup, we assume that each arm has a latent (context-dependent) utility, and that feedback on a preselection is produced according to a Plackett-Luce model. We propose the CPPL algorithm, which is inspired by the well-known UCB algorithm, and evaluate this algorithm on synthetic and real data. In particular, we consider an online algorithm selection scenario, which served as a main motivation of our problem setting. Here, an instance (which defines the context) from a certain problem class (such as SAT) can be solved by different algorithms (the arms), but only $k$ of these algorithms can actually be run. AU - El Mesaoudi-Paul, Adil AU - Bengs, Viktor AU - Hüllermeier, Eyke ID - 18017 T2 - arXiv:2002.04275 TI - Online Preselection with Context Information under the Plackett-Luce Model ER - TY - CONF AB - Algorithm selection (AS) deals with the automatic selection of an algorithm from a fixed set of candidate algorithms most suitable for a specific instance of an algorithmic problem class, where "suitability" often refers to an algorithm's runtime. Due to possibly extremely long runtimes of candidate algorithms, training data for algorithm selection models is usually generated under time constraints in the sense that not all algorithms are run to completion on all instances. Thus, training data usually comprises censored information, as the true runtime of algorithms timed out remains unknown. However, many standard AS approaches are not able to handle such information in a proper way. On the other side, survival analysis (SA) naturally supports censored data and offers appropriate ways to use such data for learning distributional models of algorithm runtime, as we demonstrate in this work. We leverage such models as a basis of a sophisticated decision-theoretic approach to algorithm selection, which we dub Run2Survive. Moreover, taking advantage of a framework of this kind, we advocate a risk-averse approach to algorithm selection, in which the avoidance of a timeout is given high priority. In an extensive experimental study with the standard benchmark ASlib, our approach is shown to be highly competitive and in many cases even superior to state-of-the-art AS approaches. AU - Tornede, Alexander AU - Wever, Marcel Dominik AU - Werner, Stefan AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 18276 T2 - ACML 2020 TI - Run2Survive: A Decision-theoretic Approach to Algorithm Selection based on Survival Analysis ER - TY - JOUR AU - Richter, Cedric AU - Hüllermeier, Eyke AU - Jakobs, Marie-Christine AU - Wehrheim, Heike ID - 16725 JF - Journal of Automated Software Engineering TI - Algorithm Selection for Software Validation Based on Graph Kernels ER - TY - CONF AB - In multi-label classification (MLC), each instance is associated with a set of class labels, in contrast to standard classification where an instance is assigned a single label. Binary relevance (BR) learning, which reduces a multi-label to a set of binary classification problems, one per label, is arguably the most straight-forward approach to MLC. In spite of its simplicity, BR proved to be competitive to more sophisticated MLC methods, and still achieves state-of-the-art performance for many loss functions. Somewhat surprisingly, the optimal choice of the base learner for tackling the binary classification problems has received very little attention so far. Taking advantage of the label independence assumption inherent to BR, we propose a label-wise base learner selection method optimizing label-wise macro averaged performance measures. In an extensive experimental evaluation, we find that or approach, called LiBRe, can significantly improve generalization performance. AU - Wever, Marcel Dominik AU - Tornede, Alexander AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 15629 TI - LiBRe: Label-Wise Selection of Base Learners in Binary Relevance for Multi-Label Classification ER - TY - JOUR AB - In software engineering, the imprecise requirements of a user are transformed to a formal requirements specification during the requirements elicitation process. This process is usually guided by requirements engineers interviewing the user. We want to partially automate this first step of the software engineering process in order to enable users to specify a desired software system on their own. With our approach, users are only asked to provide exemplary behavioral descriptions. The problem of synthesizing a requirements specification from examples can partially be reduced to the problem of grammatical inference, to which we apply an active coevolutionary learning approach. However, this approach would usually require many feedback queries to be sent to the user. In this work, we extend and generalize our active learning approach to receive knowledge from multiple oracles, also known as proactive learning. The ‘user oracle’ represents input received from the user and the ‘knowledge oracle’ represents available, formalized domain knowledge. We call our two-oracle approach the ‘first apply knowledge then query’ (FAKT/Q) algorithm. We compare FAKT/Q to the active learning approach and provide an extensive benchmark evaluation. As result we find that the number of required user queries is reduced and the inference process is sped up significantly. Finally, with so-called On-The-Fly Markets, we present a motivation and an application of our approach where such knowledge is available. AU - Wever, Marcel Dominik AU - van Rooijen, Lorijn AU - Hamann, Heiko ID - 15025 IS - 2 JF - Evolutionary Computation TI - Multi-Oracle Coevolutionary Learning of Requirements Specifications from Examples in On-The-Fly Markets VL - 28 ER - TY - GEN AB - We study the problem of learning choice functions, which play an important role in various domains of application, most notably in the field of economics. Formally, a choice function is a mapping from sets to sets: Given a set of choice alternatives as input, a choice function identifies a subset of most preferred elements. Learning choice functions from suitable training data comes with a number of challenges. For example, the sets provided as input and the subsets produced as output can be of any size. Moreover, since the order in which alternatives are presented is irrelevant, a choice function should be symmetric. Perhaps most importantly, choice functions are naturally context-dependent, in the sense that the preference in favor of an alternative may depend on what other options are available. We formalize the problem of learning choice functions and present two general approaches based on two representations of context-dependent utility functions. Both approaches are instantiated by means of appropriate neural network architectures, and their performance is demonstrated on suitable benchmark tasks. AU - Pfannschmidt, Karlson AU - Gupta, Pritha AU - Hüllermeier, Eyke ID - 19523 T2 - arXiv:1901.10860 TI - Learning Choice Functions: Concepts and Architectures ER - TY - JOUR AU - Merten, Marie-Luis AU - Seemann, Nina AU - Wever, Marcel Dominik ID - 17565 IS - 142 JF - Niederdeutsches Jahrbuch TI - Grammatikwandel digital-kulturwissenschaftlich erforscht. Mittelniederdeutscher Sprachausbau im interdisziplinären Zugriff ER - TY - GEN AB - A common statistical task lies in showing asymptotic normality of certain statistics. In many of these situations, classical textbook results on weak convergence theory suffice for the problem at hand. However, there are quite some scenarios where stronger results are needed in order to establish an asymptotic normal approximation uniformly over a family of probability measures. In this note we collect some results in this direction. We restrict ourselves to weak convergence in $\mathbb R^d$ with continuous limit measures. AU - Bengs, Viktor AU - Holzmann, Hajo ID - 18018 T2 - arXiv:1903.09864 TI - Uniform approximation in classical weak convergence theory ER - TY - GEN AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke AU - Hetzer, Alexander ID - 8868 TI - Towards Automated Machine Learning for Multi-Label Classification ER - TY - JOUR AU - Tagne, V. K. AU - Fotso, S. AU - Fono, L. A. AU - Hüllermeier, Eyke ID - 10578 IS - 2 JF - New Mathematics and Natural Computation TI - Choice Functions Generated by Mallows and Plackett–Luce Relations VL - 15 ER - TY - JOUR AU - Couso, Ines AU - Borgelt, Christian AU - Hüllermeier, Eyke AU - Kruse, Rudolf ID - 15001 JF - IEEE Computational Intelligence Magazine SN - 1556-603X TI - Fuzzy Sets in Data Analysis: From Statistical Foundations to Machine Learning ER - TY - JOUR AB - Many problem settings in machine learning are concerned with the simultaneous prediction of multiple target variables of diverse type. Amongst others, such problem settings arise in multivariate regression, multi-label classification, multi-task learning, dyadic prediction, zero-shot learning, network inference, and matrix completion. These subfields of machine learning are typically studied in isolation, without highlighting or exploring important relationships. In this paper, we present a unifying view on what we call multi-target prediction (MTP) problems and methods. First, we formally discuss commonalities and differences between existing MTP problems. To this end, we introduce a general framework that covers the above subfields as special cases. As a second contribution, we provide a structured overview of MTP methods. This is accomplished by identifying a number of key properties, which distinguish such methods and determine their suitability for different types of problems. Finally, we also discuss a few challenges for future research. AU - Waegeman, Willem AU - Dembczynski, Krzysztof AU - Hüllermeier, Eyke ID - 15002 IS - 2 JF - Data Mining and Knowledge Discovery SN - 1573-756X TI - Multi-target prediction: a unifying view on problems and methods VL - 33 ER - TY - CONF AU - Mortier, Thomas AU - Wydmuch, Marek AU - Dembczynski, Krzysztof AU - Hüllermeier, Eyke AU - Waegeman, Willem ID - 15003 T2 - Proceedings of the 31st Benelux Conference on Artificial Intelligence {(BNAIC} 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn 2019), Brussels, Belgium, November 6-8, 2019 TI - Set-Valued Prediction in Multi-Class Classification ER - TY - CHAP AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 15004 SN - 0302-9743 T2 - Discovery Science TI - Feature Selection for Analogy-Based Learning to Rank ER - TY - CHAP AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 15005 SN - 0302-9743 T2 - KI 2019: Advances in Artificial Intelligence TI - Analogy-Based Preference Learning with Kernels ER - TY - CHAP AU - Nguyen, Vu-Linh AU - Destercke, Sébastien AU - Hüllermeier, Eyke ID - 15006 SN - 0302-9743 T2 - Discovery Science TI - Epistemic Uncertainty Sampling ER - TY - CONF AU - Melnikov, Vitaly AU - Hüllermeier, Eyke ID - 15007 T2 - Proceedings ACML, Asian Conference on Machine Learning (Proceedings of Machine Learning Research, 101) TI - Learning to Aggregate: Tackling the Aggregation/Disaggregation Problem for OWA ER - TY - CONF AU - Epple, Nico AU - Dari, Simone AU - Drees, Ludwig AU - Protschky, Valentin AU - Riener, Andreas ID - 15009 SN - 9781728105604 T2 - 2019 IEEE Intelligent Vehicles Symposium (IV) TI - Influence of Cruise Control on Driver Guidance - a Comparison between System Generations and Countries ER - TY - CONF AU - Tornede, Alexander AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ED - Hoffmann, Frank ED - Hüllermeier, Eyke ED - Mikut, Ralf ID - 15011 SN - 978-3-7315-0979-0 T2 - Proceedings - 29. Workshop Computational Intelligence, Dortmund, 28. - 29. November 2019 TI - Algorithm Selection as Recommendation: From Collaborative Filtering to Dyad Ranking ER - TY - CONF AU - Brinker, Klaus AU - Hüllermeier, Eyke ID - 15013 T2 - Proceedings ECML/PKDD, European Conference on Machine Learning and Knowledge Discovery in Databases TI - A Reduction of Label Ranking to Multiclass Classification ER - TY - CONF AU - Hüllermeier, Eyke AU - Couso, Ines AU - Diestercke, Sebastian ID - 15014 T2 - Proceedings SUM 2019, International Conference on Scalable Uncertainty Management TI - Learning from Imprecise Data: Adjustments of Optimistic and Pessimistic Variants ER - TY - JOUR AU - Henzgen, Sascha AU - Hüllermeier, Eyke ID - 15015 JF - ACM Transactions on Knowledge Discovery from Data SN - 1556-4681 TI - Mining Rank Data ER - TY - JOUR AU - Bengs, Viktor AU - Eulert, Matthias AU - Holzmann, Hajo ID - 14027 JF - Journal of Multivariate Analysis SN - 0047-259X TI - Asymptotic confidence sets for the jump curve in bivariate regression problems ER - TY - JOUR AU - Bengs, Viktor AU - Holzmann, Hajo ID - 14028 JF - Electronic Journal of Statistics SN - 1935-7524 TI - Adaptive confidence sets for kink estimation ER - TY - GEN AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Tornede, Alexander AU - Hüllermeier, Eyke ID - 13132 T2 - INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft TI - From Automated to On-The-Fly Machine Learning ER - TY - CONF AB - Existing tools for automated machine learning, such as Auto-WEKA, TPOT, auto-sklearn, and more recently ML-Plan, have shown impressive results for the tasks of single-label classification and regression. Yet, there is only little work on other types of machine learning problems so far. In particular, there is almost no work on automating the engineering of machine learning solutions for multi-label classification (MLC). We show how the scope of ML-Plan, an AutoML-tool for multi-class classification, can be extended towards MLC using MEKA, which is a multi-label extension of the well-known Java library WEKA. The resulting approach recursively refines MEKA's multi-label classifiers, nesting other multi-label classifiers for meta algorithms and single-label classifiers provided by WEKA as base learners. In our evaluation, we find that the proposed approach yields strong results and performs significantly better than a set of baselines we compare with. AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Tornede, Alexander AU - Hüllermeier, Eyke ID - 10232 TI - Automating Multi-Label Classification Extending ML-Plan ER - TY - JOUR AU - Rohlfing, Katharina AU - Leonardi, Giuseppe AU - Nomikou, Iris AU - Rączaszek-Leonardi, Joanna AU - Hüllermeier, Eyke ID - 20243 JF - IEEE Transactions on Cognitive and Developmental Systems TI - Multimodal Turn-Taking: Motivations, Methodological Challenges, and Novel Approaches ER - TY - CONF AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke AU - Faez, Amin ID - 2479 T2 - SCC TI - (WIP) Towards the Automated Composition of Machine Learning Services ER - TY - GEN AB - Object ranking is an important problem in the realm of preference learning. On the basis of training data in the form of a set of rankings of objects, which are typically represented as feature vectors, the goal is to learn a ranking function that predicts a linear order of any new set of objects. Current approaches commonly focus on ranking by scoring, i.e., on learning an underlying latent utility function that seeks to capture the inherent utility of each object. These approaches, however, are not able to take possible effects of context-dependence into account, where context-dependence means that the utility or usefulness of an object may also depend on what other objects are available as alternatives. In this paper, we formalize the problem of context-dependent ranking and present two general approaches based on two natural representations of context-dependent ranking functions. Both approaches are instantiated by means of appropriate neural network architectures, which are evaluated on suitable benchmark task. AU - Pfannschmidt, Karlson AU - Gupta, Pritha AU - Hüllermeier, Eyke ID - 19524 T2 - arXiv:1803.05796 TI - Deep Architectures for Learning Context-dependent Ranking Functions ER - TY - CONF AU - Mohr, Felix AU - Lettmann, Theodor AU - Hüllermeier, Eyke AU - Wever, Marcel Dominik ID - 2857 T2 - Proceedings of the 1st ICAPS Workshop on Hierarchical Planning TI - Programmatic Task Network Planning ER - TY - JOUR AU - Ramaswamy, Arunselvan AU - Bhatnagar, Shalabh ID - 24150 IS - 6 JF - IEEE Transactions on Automatic Control TI - Stability of stochastic approximations with “controlled markov” noise and temporal difference learning VL - 64 ER - TY - JOUR AU - Demirel, Burak AU - Ramaswamy, Arunselvan AU - Quevedo, Daniel E AU - Karl, Holger ID - 24151 IS - 4 JF - IEEE Control Systems Letters TI - Deepcas: A deep reinforcement learning algorithm for control-aware scheduling VL - 2 ER - TY - CONF AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 2471 T2 - SCC TI - On-The-Fly Service Construction with Prototypes ER - TY - JOUR AB - In machine learning, so-called nested dichotomies are utilized as a reduction technique, i.e., to decompose a multi-class classification problem into a set of binary problems, which are solved using a simple binary classifier as a base learner. The performance of the (multi-class) classifier thus produced strongly depends on the structure of the decomposition. In this paper, we conduct an empirical study, in which we compare existing heuristics for selecting a suitable structure in the form of a nested dichotomy. Moreover, we propose two additional heuristics as natural completions. One of them is the Best-of-K heuristic, which picks the (presumably) best among K randomly generated nested dichotomies. Surprisingly, and in spite of its simplicity, it turns out to outperform the state of the art. AU - Melnikov, Vitalik AU - Hüllermeier, Eyke ID - 3402 JF - Machine Learning SN - 1573-0565 TI - On the effectiveness of heuristics for learning nested dichotomies: an empirical analysis ER - TY - JOUR AB - Automated machine learning (AutoML) seeks to automatically select, compose, and parametrize machine learning algorithms, so as to achieve optimal performance on a given task (dataset). Although current approaches to AutoML have already produced impressive results, the field is still far from mature, and new techniques are still being developed. In this paper, we present ML-Plan, a new approach to AutoML based on hierarchical planning. To highlight the potential of this approach, we compare ML-Plan to the state-of-the-art frameworks Auto-WEKA, auto-sklearn, and TPOT. In an extensive series of experiments, we show that ML-Plan is highly competitive and often outperforms existing approaches. AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 3510 JF - Machine Learning KW - AutoML KW - Hierarchical Planning KW - HTN planning KW - ML-Plan SN - 0885-6125 TI - ML-Plan: Automated Machine Learning via Hierarchical Planning ER - TY - CONF AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 3552 T2 - Proceedings of the Symposium on Intelligent Data Analysis TI - Reduction Stumps for Multi-Class Classification ER - TY - CONF AB - In automated machine learning (AutoML), the process of engineering machine learning applications with respect to a specific problem is (partially) automated. Various AutoML tools have already been introduced to provide out-of-the-box machine learning functionality. More specifically, by selecting machine learning algorithms and optimizing their hyperparameters, these tools produce a machine learning pipeline tailored to the problem at hand. Except for TPOT, all of these tools restrict the maximum number of processing steps of such a pipeline. However, as TPOT follows an evolutionary approach, it suffers from performance issues when dealing with larger datasets. In this paper, we present an alternative approach leveraging a hierarchical planning to configure machine learning pipelines that are unlimited in length. We evaluate our approach and find its performance to be competitive with other AutoML tools, including TPOT. AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 3852 KW - automated machine learning KW - complex pipelines KW - hierarchical planning T2 - ICML 2018 AutoML Workshop TI - ML-Plan for Unlimited-Length Machine Learning Pipelines ER - TY - CONF AB - In multinomial classification, reduction techniques are commonly used to decompose the original learning problem into several simpler problems. For example, by recursively bisecting the original set of classes, so-called nested dichotomies define a set of binary classification problems that are organized in the structure of a binary tree. In contrast to the existing one-shot heuristics for constructing nested dichotomies and motivated by recent work on algorithm configuration, we propose a genetic algorithm for optimizing the structure of such dichotomies. A key component of this approach is the proposed genetic representation that facilitates the application of standard genetic operators, while still supporting the exchange of partial solutions under recombination. We evaluate the approach in an extensive experimental study, showing that it yields classifiers with superior generalization performance. AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 2109 KW - Classification KW - Hierarchical Decomposition KW - Indirect Encoding T2 - Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, Kyoto, Japan, July 15-19, 2018 TI - Ensembles of Evolved Nested Dichotomies for Classification ER - TY - GEN AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 17713 TI - Automated Multi-Label Classification based on ML-Plan ER - TY - GEN AU - Mohr, Felix AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ID - 17714 TI - Automated machine learning service composition ER - TY - GEN AU - Graf, Helena ID - 5693 TI - Ranking of Classification Algorithms in AutoML ER - TY - GEN AU - Scheibl, Manuel ID - 5936 TI - Learning about learning curves from dataset properties ER - TY - CHAP AU - Schäfer, Dirk AU - Hüllermeier, Eyke ID - 6423 SN - 0302-9743 T2 - Discovery Science TI - Preference-Based Reinforcement Learning Using Dyad Ranking ER - TY - GEN ED - Abiteboul, S. ED - Arenas, M. ED - Barceló, P. ED - Bienvenu, M. ED - Calvanese, D. ED - David, C. ED - Hull, R. ED - Hüllermeier, Eyke ED - Kimelfeld, B. ED - Libkin, L. ED - Martens, W. ED - Milo, T. ED - Murlak, F. ED - Neven, F. ED - Ortiz, M. ED - Schwentick, T. ED - Stoyanovich, J. ED - Su, J. ED - Suciu, D. ED - Vianu, V. ED - Yi, K. ID - 10591 IS - 1 TI - Research Directions for Principles of Data Management VL - 7 ER - TY - CHAP AU - Couso, Ines AU - Hüllermeier, Eyke ED - Mostaghim, Sanaz ED - Nürnberger, Andreas ED - Borgelt, Christian ID - 10783 T2 - Frontiers in Computational Intelligence TI - Statistical Inference for Incomplete Ranking Data: A Comparison of two likelihood-based estimators ER - TY - JOUR AU - Schäfer, D. AU - Hüllermeier, Eyke ID - 16038 IS - 5 JF - Machine Learning TI - Dyad ranking using Plackett-Luce models based on joint feature representations VL - 107 ER - TY - CONF AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 10145 T2 - Proc. 32 nd AAAI Conference on Artificial Intelligence (AAAI) TI - Learning to Rank Based on Analogical Reasoning ER - TY - CONF AU - El Mesaoudi-Paul, Adil AU - Hüllermeier, Eyke AU - Busa-Fekete, Robert ID - 10148 T2 - Proc. 35th Int. Conference on Machine Learning (ICML) TI - Ranking Distributions based on Noisy Sorting ER - TY - CONF AU - Hesse, M. AU - Timmermann, J. AU - Hüllermeier, Eyke AU - Trächtler, Ansgar ID - 10149 T2 - Proc. 4th Int. Conference on System-Integrated Intelligence: Intelligent, Flexible and Connected Systems in Products and Production, Procedia Manufacturing 24 TI - A Reinforcement Learning Strategy for the Swing-Up of the Double Pendulum on a Cart ER - TY - CHAP AU - Mencia, E.Loza AU - Fürnkranz, J. AU - Hüllermeier, Eyke AU - Rapp, M. ED - Jair Escalante, H. ED - Escalera, S. ED - Guyon, I. ED - Baro, X. ED - Güclüütürk, Y. ED - Güclü, U. ED - van Gerven, M.A.J. ID - 10152 T2 - Explainable and Interpretable Models in Computer Vision and Machine Learning TI - Learning interpretable rules for multi-label classification ER - TY - CONF AU - Nguyen, Vu-Linh AU - Destercke, Sebastian AU - Masson, M.-H. AU - Hüllermeier, Eyke ID - 10181 T2 - Proc. 27th Int.Joint Conference on Artificial Intelligence (IJCAI) TI - Reliable Multi-class Classification based on Pairwise Epistemic and Aleatoric Uncertainty ER - TY - CONF AU - Schäfer, Dirk AU - Hüllermeier, Eyke ID - 10184 T2 - Proc. 21st Int. Conference on Discovery Science (DS) TI - Preference-Based Reinforcement Learning Using Dyad Ranking ER - TY - JOUR AU - Schäfer, Dirk AU - Hüllermeier, Eyke ID - 10276 IS - 5 JF - Machine Learning TI - Dyad Ranking Using Plackett-Luce Models based on joint feature representations VL - 107 ER - TY - GEN AU - Seemann, Nina AU - Geierhos, Michaela AU - Merten, Marie-Luis AU - Tophinke, Doris AU - Wever, Marcel Dominik AU - Hüllermeier, Eyke ED - Eckart, Kerstin ED - Schlechtweg, Dominik ID - 1379 T2 - Postersession Computerlinguistik der 40. Jahrestagung der Deutschen Gesellschaft für Sprachwissenschaft TI - Supporting the Cognitive Process in Annotation Tasks ER - TY - JOUR AU - Ramaswamy, Arunselvan AU - Bhatnagar, Shalabh ID - 24152 IS - 5 JF - IEEE Transactions on Automatic Control TI - Analysis of gradient descent methods with nondiminishing bounded errors VL - 63 ER - TY - JOUR AU - Ramaswamy, Arunselvan AU - Bhatnagar, Shalabh ID - 24153 IS - 3 JF - Mathematics of Operations Research TI - A generalization of the Borkar-Meyn theorem for stochastic recursive inclusions VL - 42 ER - TY - CONF AU - Melnikov, Vitalik AU - Hüllermeier, Eyke ID - 3325 T2 - Proceedings. 27. Workshop Computational Intelligence, Dortmund, 23. - 24. November 2017 TI - Optimizing the Structure of Nested Dichotomies: A Comparison of Two Heuristics ER - TY - CONF AB - Whenever customers have to decide between different instances of the same product, they are interested in buying the best product. In contrast, companies are interested in reducing the construction effort (and usually as a consequence thereof, the quality) to gain profit. The described setting is widely known as opposed preferences in quality of the product and also applies to the context of service-oriented computing. In general, service-oriented computing emphasizes the construction of large software systems out of existing services, where services are small and self-contained pieces of software that adhere to a specified interface. Several implementations of the same interface are considered as several instances of the same service. Thereby, customers are interested in buying the best service implementation for their service composition wrt. to metrics, such as costs, energy, memory consumption, or execution time. One way to ensure the service quality is to employ certificates, which can come in different kinds: Technical certificates proving correctness can be automatically constructed by the service provider and again be automatically checked by the user. Digital certificates allow proof of the integrity of a product. Other certificates might be rolled out if service providers follow a good software construction principle, which is checked in annual audits. Whereas all of these certificates are handled differently in service markets, what they have in common is that they influence the buying decisions of customers. In this paper, we review state-of-the-art developments in certification with respect to service-oriented computing. We not only discuss how certificates are constructed and handled in service-oriented computing but also review the effects of certificates on the market from an economic perspective. AU - Jakobs, Marie-Christine AU - Krämer, Julia AU - van Straaten, Dirk AU - Lettmann, Theodor ED - Marcelo De Barros, Janusz Klink,Tadeus Uhl, Thomas Prinz ID - 115 T2 - The Ninth International Conferences on Advanced Service Computing (SERVICE COMPUTATION) TI - Certification Matters for Service Markets ER - TY - CONF AB - In this paper, we present the annotation challenges we have encountered when working on a historical language that was undergoing elaboration processes. We especially focus on syntactic ambiguity and gradience in Middle Low German, which causes uncertainty to some extent. Since current annotation tools consider construction contexts and the dynamics of the grammaticalization only partially, we plan to extend CorA – a web-based annotation tool for historical and other non-standard language data – to capture elaboration phenomena and annotator unsureness. Moreover, we seek to interactively learn morphological as well as syntactic annotations. AU - Seemann, Nina AU - Merten, Marie-Luis AU - Geierhos, Michaela AU - Tophinke, Doris AU - Hüllermeier, Eyke ID - 1158 T2 - Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature TI - Annotation Challenges for Reconstructing the Structural Elaboration of Middle Low German ER - TY - GEN AU - Schnitker, Nino Noel ID - 5694 TI - Genetischer Algorithmus zur Erstellung von Ensembles von Nested Dichotomies ER - TY - GEN AU - Gupta, Pritha AU - Hetzer, Alexander AU - Tornede, Tanja AU - Gottschalk, Sebastian AU - Kornelsen, Andreas AU - Osterbrink, Sebastian AU - Pfannschmidt, Karlson AU - Hüllermeier, Eyke ID - 5722 TI - jPL: A Java-based Software Framework for Preference Learning ER - TY - GEN AU - Hetzer, Alexander AU - Tornede, Tanja ID - 5724 TI - Solving the Container Pre-Marshalling Problem using Reinforcement Learning and Structured Output Prediction ER - TY - CONF AB - Today, software verification tools have reached the maturity to be used for large scale programs. Different tools perform differently well on varying code. A software developer is hence faced with the problem of choosing a tool appropriate for her program at hand. A ranking of tools on programs could facilitate the choice. Such rankings can, however, so far only be obtained by running all considered tools on the program.In this paper, we present a machine learning approach to predicting rankings of tools on programs. The method builds upon so-called label ranking algorithms, which we complement with appropriate kernels providing a similarity measure for programs. Our kernels employ a graph representation for software source code that mixes elements of control flow and program dependence graphs with abstract syntax trees. Using data sets from the software verification competition SV-COMP, we demonstrate our rank prediction technique to generalize well and achieve a rather high predictive accuracy (rank correlation > 0.6). AU - Czech, Mike AU - Hüllermeier, Eyke AU - Jakobs, Marie-Christine AU - Wehrheim, Heike ID - 71 T2 - Proceedings of the 3rd International Workshop on Software Analytics TI - Predicting Rankings of Software Verification Tools ER - TY - GEN AB - Software verification competitions, such as the annual SV-COMP, evaluate software verification tools with respect to their effectivity and efficiency. Typically, the outcome of a competition is a (possibly category-specific) ranking of the tools. For many applications, such as building portfolio solvers, it would be desirable to have an idea of the (relative) performance of verification tools on a given verification task beforehand, i.e., prior to actually running all tools on the task.In this paper, we present a machine learning approach to predicting rankings of tools on verification tasks. The method builds upon so-called label ranking algorithms, which we complement with appropriate kernels providing a similarity measure for verification tasks. Our kernels employ a graph representation for software source code that mixes elements of control flow and program dependence graphs with abstract syntax trees. Using data sets from SV-COMP, we demonstrate our rank prediction technique to generalize well and achieve a rather high predictive accuracy. In particular, our method outperforms a recently proposed feature-based approach of Demyanova et al. (when applied to rank predictions). AU - Czech, Mike AU - Hüllermeier, Eyke AU - Jakobs, Marie-Christine AU - Wehrheim, Heike ID - 72 TI - Predicting Rankings of Software Verification Competitions ER - TY - GEN AU - Fürnkranz, J. AU - Hüllermeier, Eyke ID - 10589 T2 - Encyclopedia of Machine Learning and Data Mining TI - Preference Learning ER - TY - CHAP AU - Fürnkranz, J. AU - Hüllermeier, Eyke ED - Sammut, C. ED - Webb, G.I. ID - 10784 T2 - Encyclopedia of Machine Learning and Data Mining TI - Preference Learning VL - 107 ER - TY - CONF AB - These days, there is a strong rise in the needs for machine learning applications, requiring an automation of machine learning engineering which is referred to as AutoML. In AutoML the selection, composition and parametrization of machine learning algorithms is automated and tailored to a specific problem, resulting in a machine learning pipeline. Current approaches reduce the AutoML problem to optimization of hyperparameters. Based on recursive task networks, in this paper we present one approach from the field of automated planning and one evolutionary optimization approach. Instead of simply parametrizing a given pipeline, this allows for structure optimization of machine learning pipelines, as well. We evaluate the two approaches in an extensive evaluation, finding both approaches to have their strengths in different areas. Moreover, the two approaches outperform the state-of-the-art tool Auto-WEKA in many settings. AU - Wever, Marcel Dominik AU - Mohr, Felix AU - Hüllermeier, Eyke ID - 1180 T2 - 27th Workshop Computational Intelligence TI - Automatic Machine Learning: Hierachical Planning Versus Evolutionary Optimization ER - TY - CONF AU - Melnikov, Vitaly AU - Hüllermeier, Eyke ED - Hoffmann, F. ED - Hüllermeier, Eyke ED - Mikut, R. ID - 15397 T2 - in Proceedings 27th Workshop Computational Intelligence, Dortmund Germany TI - Optimizing the structure of nested dichotomies. A comparison of two heuristics ER - TY - CONF AU - Czech, M. AU - Hüllermeier, Eyke AU - Jacobs, M.C. AU - Wehrheim, Heike ID - 15399 T2 - in Proceedings ESEC/FSE Workshops 2017 - 3rd ACM SIGSOFT, International Workshop on Software Analytics (SWAN 2017), Paderborn Germany TI - Predicting rankings of software verification tools ER - TY - CONF AU - Couso, Ines AU - Dubois, D. AU - Hüllermeier, Eyke ID - 15110 T2 - in Proceedings SUM 2017, 11th International Conference on Scalable Uncertainty Management, Granada, Spain TI - Maximum likelihood estimation and coarse data ER - TY - CONF AU - Ewerth, Ralph AU - Springstein, M. AU - Müller, E. AU - Balz, A. AU - Gehlhaar, J. AU - Naziyok, T. AU - Dembczynski, K. AU - Hüllermeier, Eyke ID - 10204 T2 - Proc. IEEE Int. Conf. on Multimedia and Expo (ICME 2017) TI - Estimating relative depth in single images via rankboost ER - TY - CONF AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke AU - Couso, Ines ID - 10205 T2 - Proc. 34th Int. Conf. on Machine Learning (ICML 2017) TI - Statistical Inference for Incomplete Ranking Data: The Case of Rank-Dependent Coarsening ER - TY - CONF AU - Mohr, Felix AU - Lettmann, Theodor AU - Hüllermeier, Eyke ID - 10206 T2 - Proc. 40th Annual German Conference on Advances in Artificial Intelligence (KI 2017) TI - Planning with Independent Task Networks ER - TY - CONF AU - Czech, M. AU - Hüllermeier, Eyke AU - Jakobs, M.-C. AU - Wehrheim, Heike ID - 10207 T2 - Proc. 3rd ACM SIGSOFT Int. I Workshop on Software Analytics (SWAN@ESEC/SIGSOFT FSE 2017 TI - Predicting rankings of software verification tools ER - TY - CONF AU - Couso, Ines AU - Dubois, D. AU - Hüllermeier, Eyke ID - 10208 T2 - Proc. 11th Int. Conf. on Scalable Uncertainty Management (SUM 2017) TI - Maximum Likelihood Estimation and Coarse Data ER - TY - CONF AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ID - 10209 T2 - Proc. AAAI 2017, 32nd AAAI Conference on Artificial Intelligence TI - Learning to Rank based on Analogical Reasoning ER - TY - CONF AU - Hoffmann, F. AU - Hüllermeier, Eyke AU - Mikut, R. ID - 10212 TI - (Hrsg.) Proceedings 27. Workshop Computational Intelligence, KIT Scientific Publishing, Karlsruhe, Germany 2017 ER - TY - CONF AU - Melnikov, Vitaly AU - Hüllermeier, Eyke ID - 10213 T2 - Proceedings 27. Workshop Computational Intelligence, Dortmund, Germany 2017 TI - Optimizing the Structure of Nested Dichotomies: A Comparison of Two Heuristics ER - TY - CONF AU - Shaker, Ammar AU - Heldt, W. AU - Hüllermeier, Eyke ID - 10216 T2 - Proceedings ECML/PKDD, European Conference on Machine Learning and Knowledge Discovery in Databases, Skopje, Macedonia TI - Learning TSK Fuzzy Rules from Data Streams ER - TY - JOUR AU - Bräuning, M. AU - Hüllermeier, Eyke AU - Keller, T. AU - Glaum, M. ID - 10267 IS - 1 JF - European Journal of Operational Research TI - Lexicographic preferences for predictive modeling of human decision making. A new machine learning method with an application in accounting VL - 258 ER - TY - JOUR AU - Platenius, M.-C. AU - Shaker, Ammar AU - Becker, M. AU - Hüllermeier, Eyke AU - Schäfer, W. ID - 10268 IS - 8 JF - IEEE Transactions on Software Engineering TI - Imprecise Matching of Requirements Specifications for Software Services Using Fuzzy Logic VL - 43 ER - TY - JOUR AU - Hüllermeier, Eyke ID - 10269 JF - The Computing Research Repository (CoRR) TI - From Knowledge-based to Data-driven Modeling of Fuzzy Rule-based Systems: A Critical Reflection ER - TY - JOUR AU - Ramaswamy, Arunselvan AU - Bhatnagar, Shalabh ID - 24154 IS - 8 JF - Stochastics TI - Stochastic recursive inclusion in two timescales with an application to the lagrangian dual problem VL - 88 ER - TY - JOUR AU - Melnikov, Vitalik AU - Hüllermeier, Eyke AU - Kaimann, Daniel AU - Frick, Bernd AU - Gupta, Pritha ID - 3318 JF - Schedae Informaticae SN - 2083-8476 TI - Pairwise versus Pointwise Ranking: A Case Study VL - 25 ER - TY - JOUR AB - Today, software components are provided by global markets in the form of services. In order to optimally satisfy service requesters and service providers, adequate techniques for automatic service matching are needed. However, a requester’s requirements may be vague and the information available about a provided service may be incomplete. As a consequence, fuzziness is induced into the matching procedure. The contribution of this paper is the development of a systematic matching procedure that leverages concepts and techniques from fuzzy logic and possibility theory based on our formal distinction between different sources and types of fuzziness in the context of service matching. In contrast to existing methods, our approach is able to deal with imprecision and incompleteness in service specifications and to inform users about the extent of induced fuzziness in order to improve the user’s decision-making. We demonstrate our approach on the example of specifications for service reputation based on ratings given by previous users. Our evaluation based on real service ratings shows the utility and applicability of our approach. AU - Platenius, Marie Christin AU - Shaker, Ammar AU - Becker, Matthias AU - Hüllermeier, Eyke AU - Schäfer, Wilhelm ID - 190 IS - 8 JF - IEEE Transactions on Software Engineering (TSE), presented at ICSE 2017 TI - Imprecise Matching of Requirements Specifications for Software Services using Fuzzy Logic ER - TY - CONF AB - In this paper, we propose a framework for a class of learning problems that we refer to as “learning to aggregate”. Roughly, learning-to-aggregate problems are supervised machine learning problems, in which instances are represented in the form of a composition of a (variable) number on constituents; such compositions are associated with an evaluation, score, or label, which is the target of the prediction task, and which can presumably be modeled in the form of a suitable aggregation of the properties of its constituents. Our learning-to-aggregate framework establishes a close connection between machine learning and a branch of mathematics devoted to the systematic study of aggregation functions. We specifically focus on a class of functions called uninorms, which combine conjunctive and disjunctive modes of aggregation. Experimental results for a corresponding model are presented for a review data set, for which the aggregation problem consists of combining different reviewer opinions about a paper into an overall decision of acceptance or rejection. AU - Melnikov, Vitaly AU - Hüllermeier, Eyke ID - 184 T2 - Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD 2016) TI - Learning to Aggregate Using Uninorms ER - TY - GEN AU - Fürnkranz, J. AU - Hüllermeier, Eyke ED - Sammut, C. ED - Webb, G.I. ID - 10785 T2 - Encyclopedia of Machine Learning and Data Mining TI - Preference Learning ER - TY - CONF AU - Labreuche, C. AU - Hüllermeier, Eyke AU - Vojtas, P. AU - Fallah Tehrani, A. ED - Busa-Fekete, R. ED - Hüllermeier, Eyke ED - Mousseau, V. ED - Pfannschmidt, Karlson ID - 15400 T2 - in Proceedings DA2PL 2016 EURO Mini Conference From Multiple Criteria Decision Aid to Preference Learning, Paderborn Germany TI - On the identifiability of models in multi-criteria preference learning ER - TY - CONF AU - Schäfer, D. AU - Hüllermeier, Eyke ED - Busa-Fekete, R. ED - Hüllermeier, Eyke ED - Mousseau, V. ED - Pfannschmidt, Karlson ID - 15401 T2 - in Proceedings DA2PL`2016 Euro Mini Conference From Multiple Criteria Decision Aid to Preference Learning, Paderborn, Germany TI - Preference -based reinforcement learning using dyad ranking ER - TY - CONF AU - Couso, Ines AU - Ahmadi Fahandar, Mohsen AU - Hüllermeier, Eyke ED - Busa-Fekete, R. ED - Hüllermeier, Eyke ED - Mousseau, V. ED - Pfannschmidt, Karlson ID - 15402 T2 - in Proceedings DA2PL 2016 EURO Mini Conference From Multiple Criteria Decision Aid to Preference Learning, Paderborn Germany TI - Statistical Inference for Incomplete Ranking Data: A Comparison of two likelihood-based estimators ER - TY - CONF AU - Lu, S. AU - Hüllermeier, Eyke ED - Hüllermeier, Eyke ED - Hoffmann, F. ED - Mikut, R. ID - 15403 T2 - in Proceedings 26th Workshop Computational Intelligence, Dortmund Germany TI - Support vector classification on noisy data using fuzzy superset losses ER - TY - CONF AU - Schäfer, D. AU - Hüllermeier, Eyke ID - 15404 T2 - in Workshop LWDA "Lernen, Wissen, Daten, Analysen" Potsdam, Germany TI - Plackett-Luce networks for dyad ranking ER - TY - CONF AU - Pfannschmidt, Karlson AU - Hüllermeier, Eyke AU - Held, S. AU - Neiger, R. ID - 15111 T2 - In Proceedings IPMU 16th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Part 1, Eindhoven, The Netherlands TI - Evaluating tests in medical diagnosis-Combining machine learning with game-theoretical concepts ER - TY - JOUR AU - Leinweber, M. AU - Fober, T. AU - Strickert, M. AU - Baumgärtner, L. AU - Klebe, G. AU - Freisleben, B. AU - Hüllermeier, Eyke ID - 16041 IS - 6 JF - IEEE Transactions on Knowledge and Data Engineering TI - CavSimBase: A database for large scale comparison of protein binding sites VL - 28 ER - TY - THES AU - Mohr, Felix ID - 141 TI - Towards Automated Service Composition Under Quality Constraints ER - TY - CHAP AU - Fürnkranz, J. AU - Hüllermeier, Eyke ED - Sammut, C. ED - Webb, G.I. ID - 10214 T2 - Encyclopedia of Machine Learning and Data Mining TI - Preference Learning ER - TY - GEN ED - Hoffmann, F. ED - Hüllermeier, Eyke ED - Mikut, R. ID - 10221 TI - Proceedings 26. Workshop Computational Intelligence KIT Scientific Publishing, Karlsruhe, Germany ER -