TY - JOUR AU - Redder, Adrian AU - Ramaswamy, Arunselvan AU - Karl, Holger ID - 32854 IS - 13 JF - IFAC-PapersOnLine TI - Practical Network Conditions for the Convergence of Distributed Optimization VL - 55 ER - TY - CONF AB - Modern services often comprise several components, such as chained virtual network functions, microservices, or machine learning functions. Providing such services requires to decide how often to instantiate each component, where to place these instances in the network, how to chain them and route traffic through them. To overcome limitations of conventional, hardwired heuristics, deep reinforcement learning (DRL) approaches for self-learning network and service management have emerged recently. These model-free DRL approaches are more flexible but typically learn tabula rasa, i.e., disregard existing understanding of networks, services, and their coordination. Instead, we propose FutureCoord, a novel model-based AI approach that leverages existing understanding of networks and services for more efficient and effective coordination without time-intensive training. FutureCoord combines Monte Carlo Tree Search with a stochastic traffic model. This allows FutureCoord to estimate the impact of future incoming traffic and effectively optimize long-term effects, taking fluctuating demand and Quality of Service (QoS) requirements into account. Our extensive evaluation based on real-world network topologies, services, and traffic traces indicates that FutureCoord clearly outperforms state-of-the-art model-free and model-based approaches with up to 51% higher flow success ratios. AU - Werner, Stefan AU - Schneider, Stefan Balthasar AU - Karl, Holger ID - 29220 KW - network management KW - service management KW - AI KW - Monte Carlo Tree Search KW - model-based KW - QoS T2 - IEEE/IFIP Network Operations and Management Symposium (NOMS) TI - Use What You Know: Network and Service Coordination Beyond Certainty ER - TY - CONF AB - Datacenter applications have different resource requirements from network and developing flow scheduling heuristics for every workload is practically infeasible. In this paper, we show that deep reinforcement learning (RL) can be used to efficiently learn flow scheduling policies for different workloads without manual feature engineering. Specifically, we present LFS, which learns to optimize a high-level performance objective, e.g., maximize the number of flow admissions while meeting the deadlines. The LFS scheduler is trained through deep RL to learn a scheduling policy on continuous online flow arrivals. The evaluation results show that the trained LFS scheduler admits 1.05x more flows than the greedy flow scheduling heuristics under varying network load. AU - Hasnain, Asif AU - Karl, Holger ID - 20125 KW - Flow scheduling KW - Deadlines KW - Reinforcement learning T2 - 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC) TI - Learning Flow Scheduling ER - TY - THES AU - Hasnain, Asif ID - 27503 TI - Automating Network Resource Allocation for Coflows with Deadlines ER - TY - CONF AB - Data-parallel applications are developed using different data programming models, e.g., MapReduce, partition/aggregate. These models represent diverse resource requirements of application in a datacenter network, which can be represented by the coflow abstraction. The conventional method of creating hand-crafted coflow heuristics for admission or scheduling for different workloads is practically infeasible. In this paper, we propose a deep reinforcement learning (DRL)-based coflow admission scheme -- LCS -- that can learn an admission policy for a higher-level performance objective, i.e., maximize successful coflow admissions, without manual feature engineering. LCS is trained on a production trace, which has online coflow arrivals. The evaluation results show that LCS is able to learn a reasonable admission policy that admits more coflows than state-of-the-art Varys heuristic while meeting their deadlines. AU - Hasnain, Asif AU - Karl, Holger ID - 21005 KW - Coflow scheduling KW - Reinforcement learning KW - Deadlines T2 - IEEE INFOCOM 2021 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) TI - Learning Coflow Admissions ER - TY - CONF AB - Services often consist of multiple chained components such as microservices in a service mesh, or machine learning functions in a pipeline. Providing these services requires online coordination including scaling the service, placing instance of all components in the network, scheduling traffic to these instances, and routing traffic through the network. Optimized service coordination is still a hard problem due to many influencing factors such as rapidly arriving user demands and limited node and link capacity. Existing approaches to solve the problem are often built on rigid models and assumptions, tailored to specific scenarios. If the scenario changes and the assumptions no longer hold, they easily break and require manual adjustments by experts. Novel self-learning approaches using deep reinforcement learning (DRL) are promising but still have limitations as they only address simplified versions of the problem and are typically centralized and thus do not scale to practical large-scale networks. To address these issues, we propose a distributed self-learning service coordination approach using DRL. After centralized training, we deploy a distributed DRL agent at each node in the network, making fast coordination decisions locally in parallel with the other nodes. Each agent only observes its direct neighbors and does not need global knowledge. Hence, our approach scales independently from the size of the network. In our extensive evaluation using real-world network topologies and traffic traces, we show that our proposed approach outperforms a state-of-the-art conventional heuristic as well as a centralized DRL approach (60% higher throughput on average) while requiring less time per online decision (1 ms). AU - Schneider, Stefan Balthasar AU - Qarawlus, Haydar AU - Karl, Holger ID - 21543 KW - network management KW - service management KW - coordination KW - reinforcement learning KW - distributed T2 - IEEE International Conference on Distributed Computing Systems (ICDCS) TI - Distributed Online Service Coordination Using Deep Reinforcement Learning ER - TY - CONF AB - In practical, large-scale networks, services are requested by users across the globe, e.g., for video streaming. Services consist of multiple interconnected components such as microservices in a service mesh. Coordinating these services requires scaling them according to continuously changing user demand, deploying instances at the edge close to their users, and routing traffic efficiently between users and connected instances. Network and service coordination is commonly addressed through centralized approaches, where a single coordinator knows everything and coordinates the entire network globally. While such centralized approaches can reach global optima, they do not scale to large, realistic networks. In contrast, distributed approaches scale well, but sacrifice solution quality due to their limited scope of knowledge and coordination decisions. To this end, we propose a hierarchical coordination approach that combines the good solution quality of centralized approaches with the scalability of distributed approaches. In doing so, we divide the network into multiple hierarchical domains and optimize coordination in a top-down manner. We compare our hierarchical with a centralized approach in an extensive evaluation on a real-world network topology. Our results indicate that hierarchical coordination can find close-to-optimal solutions in a fraction of the runtime of centralized approaches. AU - Schneider, Stefan Balthasar AU - Jürgens, Mirko AU - Karl, Holger ID - 20693 KW - network management KW - service management KW - coordination KW - hierarchical KW - scalability KW - nfv T2 - IFIP/IEEE International Symposium on Integrated Network Management (IM) TI - Divide and Conquer: Hierarchical Network and Service Coordination ER - TY - JOUR AB - Modern services consist of interconnected components,e.g., microservices in a service mesh or machine learning functions in a pipeline. These services can scale and run across multiple network nodes on demand. To process incoming traffic, service components have to be instantiated and traffic assigned to these instances, taking capacities, changing demands, and Quality of Service (QoS) requirements into account. This challenge is usually solved with custom approaches designed by experts. While this typically works well for the considered scenario, the models often rely on unrealistic assumptions or on knowledge that is not available in practice (e.g., a priori knowledge). We propose DeepCoord, a novel deep reinforcement learning approach that learns how to best coordinate services and is geared towards realistic assumptions. It interacts with the network and relies on available, possibly delayed monitoring information. Rather than defining a complex model or an algorithm on how to achieve an objective, our model-free approach adapts to various objectives and traffic patterns. An agent is trained offline without expert knowledge and then applied online with minimal overhead. Compared to a state-of-the-art heuristic, DeepCoord significantly improves flow throughput (up to 76%) and overall network utility (more than 2x) on realworld network topologies and traffic traces. It also supports optimizing multiple, possibly competing objectives, learns to respect QoS requirements, generalizes to scenarios with unseen, stochastic traffic, and scales to large real-world networks. For reproducibility and reuse, our code is publicly available. AU - Schneider, Stefan Balthasar AU - Khalili, Ramin AU - Manzoor, Adnan AU - Qarawlus, Haydar AU - Schellenberg, Rafael AU - Karl, Holger AU - Hecker, Artur ID - 21808 JF - Transactions on Network and Service Management KW - network management KW - service management KW - coordination KW - reinforcement learning KW - self-learning KW - self-adaptation KW - multi-objective TI - Self-Learning Multi-Objective Service Coordination Using Deep Reinforcement Learning ER - TY - GEN AB - Macrodiversity is a key technique to increase the capacity of mobile networks. It can be realized using coordinated multipoint (CoMP), simultaneously connecting users to multiple overlapping cells. Selecting which users to serve by how many and which cells is NP-hard but needs to happen continuously in real time as users move and channel state changes. Existing approaches often require strict assumptions about or perfect knowledge of the underlying radio system, its resource allocation scheme, or user movements, none of which is readily available in practice. Instead, we propose three novel self-learning and self-adapting approaches using model-free deep reinforcement learning (DRL): DeepCoMP, DD-CoMP, and D3-CoMP. DeepCoMP leverages central observations and control of all users to select cells almost optimally. DD-CoMP and D3-CoMP use multi-agent DRL, which allows distributed, robust, and highly scalable coordination. All three approaches learn from experience and self-adapt to varying scenarios, reaching 2x higher Quality of Experience than other approaches. They have very few built-in assumptions and do not need prior system knowledge, making them more robust to change and better applicable in practice than existing approaches. AU - Schneider, Stefan Balthasar AU - Karl, Holger AU - Khalili, Ramin AU - Hecker, Artur ID - 33854 KW - mobility management KW - coordinated multipoint KW - CoMP KW - cell selection KW - resource management KW - reinforcement learning KW - multi agent KW - MARL KW - self-learning KW - self-adaptation KW - QoE TI - DeepCoMP: Coordinated Multipoint Using Multi-Agent Deep Reinforcement Learning ER - TY - GEN AB - Network and service coordination is important to provide modern services consisting of multiple interconnected components, e.g., in 5G, network function virtualization (NFV), or cloud and edge computing. In this paper, I outline my dissertation research, which proposes six approaches to automate such network and service coordination. All approaches dynamically react to the current demand and optimize coordination for high service quality and low costs. The approaches range from centralized to distributed methods and from conventional heuristic algorithms and mixed-integer linear programs to machine learning approaches using supervised and reinforcement learning. I briefly discuss their main ideas and advantages over other state-of-the-art approaches and compare strengths and weaknesses. AU - Schneider, Stefan Balthasar ID - 35889 KW - nfv KW - coordination KW - machine learning KW - reinforcement learning KW - phd KW - digest TI - Conventional and Machine Learning Approaches for Network and Service Coordination ER -