Optimizing Communication for Latency Sensitive HPC Applications on up to 48 FPGAs Using ACCL - Meyer, Marius; Kenter, Tobias; Petrica, Lucian; O'Brien, Kenneth; Blott, Michaela; Plessl, Christian (arXiv:2403.18374)
Most FPGA boards in the HPC domain are well-suited for parallel scaling because of the direct integration of versatile and high-throughput network ports. However, the utilization of their network capabilities is often challenging and error-prone because the whole network stack and communication patterns have to be implemented and managed on the FPGAs. Also, this approach conceptually involves a trade-off between the performance potential of improved communication and the impact of resource consumption for communication infrastructure, since the utilized resources on the FPGAs could otherwise be used for computations. In this work, we investigate this trade-off, firstly, by using synthetic benchmarks to evaluate the different configuration options of the communication framework ACCL and their impact on communication latency and throughput. Finally, we use our findings to implement a shallow water simulation whose scalability heavily depends on low-latency communication. With a suitable configuration of ACCL, good scaling behavior can be shown to all 48 FPGAs installed in the system. Overall, the results show that the availability of inter-FPGA communication frameworks as well as the configurability of framework and network stack are crucial to achieve the best application performance with low latency communication.

Characterizing Microheterogeneity in Liquid Mixtures via Local Density Fluctuations - Lass, Michael; Kenter, Tobias; Plessl, Christian; Brehm, Martin (Entropy)
We present a novel approach to characterize and quantify microheterogeneity and microphase separation in computer simulations of complex liquid mixtures. Our post-processing method is based on local density fluctuations of the different constituents in sampling spheres of varying size. It can be easily applied to both molecular dynamics (MD) and Monte Carlo (MC) simulations, including periodic boundary conditions. Multidimensional correlation of the density distributions yields a clear picture of the domain formation due to the subtle balance of different interactions. We apply our approach to the example of force field molecular dynamics simulations of imidazolium-based ionic liquids with different side chain lengths at different temperatures, namely 1-ethyl-3-methylimidazolium chloride, 1-hexyl-3-methylimidazolium chloride, and 1-decyl-3-methylimidazolium chloride, which are known to form distinct liquid domains. We put the results into the context of existing microheterogeneity analyses and demonstrate the advantages and sensitivity of our novel method. Furthermore, we show how to estimate the configuration entropy from our analysis, and we investigate voids in the system. The analysis has been implemented into our program package TRAVIS and is thus available as free software.

Scalable quantum detector tomography by high-performance computing - Schapeler, Timon; Schade, Robert; Lass, Michael; Plessl, Christian; Bartley, Tim (arXiv:2404.02844)
At large scales, quantum systems may become advantageous over their classical counterparts at performing certain tasks. Developing tools to analyse these systems at the relevant scales, in a manner consistent with quantum mechanics, is therefore critical to benchmarking performance and characterising their operation. While classical computational approaches cannot perform like-for-like computations of quantum systems beyond a certain scale, classical high-performance computing (HPC) may nevertheless be useful for precisely these characterisation and certification tasks. By developing open-source customised algorithms using high-performance computing, we perform quantum tomography on a megascale quantum photonic detector covering a Hilbert space of $10^6$. This requires finding $10^8$ elements of the matrix corresponding to the positive operator valued measure (POVM), the quantum description of the detector, and is achieved in minutes of computation time. Moreover, by exploiting the structure of the problem, we achieve highly efficient parallel scaling, paving the way for quantum objects up to a system size of $10^{12}$ elements to be reconstructed using this method. In general, this shows that a consistent quantum mechanical description of quantum phenomena is applicable at everyday scales. More concretely, this enables the reconstruction of large-scale quantum sources, processes and detectors used in computation and sampling tasks, which may be necessary to prove their nonclassical character or quantum computational advantage.

Noctua 2 Supercomputer - Bauer, Carsten; Kenter, Tobias; Lass, Michael; Mazur, Lukas; Meyer, Marius; Nitsche, Holger; Riebler, Heinrich; Schade, Robert; Schwarz, Michael; Winnwa, Nils; Wiens, Alex; Wu, Xin; et al. (Journal of large-scale research facilities)
Noctua 2 is a supercomputer operated at the Paderborn Center for Parallel Computing (PC2) at Paderborn University in Germany. Noctua 2 was inaugurated in 2022 and is an Atos BullSequana XH2000 system. It consists mainly of three node types: 1) CPU Compute nodes with AMD EPYC processors in different main memory configurations, 2) GPU nodes with NVIDIA A100 GPUs, and 3) FPGA nodes with Xilinx Alveo U280 and Intel Stratix 10 FPGA cards. While CPUs and GPUs are known off-the-shelf components in HPC systems, the operation of a large number of FPGA cards from different vendors and a dedicated FPGA-to-FPGA network are unique characteristics of Noctua 2. This paper describes in detail the overall setup of Noctua 2 and gives insights into the operation of the cluster from a hardware, software and facility perspective.
Keywords: Noctua 2; Supercomputer; FPGA; PC2; Paderborn Center for Parallel Computing

Enabling Performance Portability for Shallow Water Equations on CPUs, GPUs, and FPGAs with SYCL - Büttner, Markus; Alt, Christoph; Kenter, Tobias; Köstler, Harald; Plessl, Christian; Aizinger, Vadym (Proceedings of the Platform for Advanced Scientific Computing Conference (PASC))

Multi-FPGA Designs and Scaling of HPC Challenge Benchmarks via MPI and Circuit-Switched Inter-FPGA Networks - Meyer, Marius; Kenter, Tobias; Plessl, Christian (ACM Transactions on Reconfigurable Technology and Systems)
While FPGA accelerator boards and their respective high-level design tools are maturing, there is still a lack of multi-FPGA applications, libraries, and not least, benchmarks and reference implementations towards sustained HPC usage of these devices. As in the early days of GPUs in HPC, for workloads that can reasonably be decoupled into loosely coupled working sets, multi-accelerator support can be achieved by using standard communication interfaces like MPI on the host side. However, for performance and productivity, some applications can profit from a tighter coupling of the accelerators. FPGAs offer unique opportunities here when extending the dataflow characteristics to their communication interfaces. In this work, we extend the HPCC FPGA benchmark suite by multi-FPGA support and three missing benchmarks that particularly characterize or stress inter-device communication: b_eff, PTRANS, and LINPACK. With all benchmarks implemented for current boards with Intel and Xilinx FPGAs, we established a baseline for multi-FPGA performance. Additionally, for the communication-centric benchmarks, we explored the potential of direct FPGA-to-FPGA communication with a circuit-switched inter-FPGA network that is currently only available for one of the boards. The evaluation with parallel execution on up to 26 FPGA boards makes use of one of the largest academic FPGA installations.
Keywords: General Computer Science

Computing and Compressing Electron Repulsion Integrals on FPGAs - Wu, Xin; Kenter, Tobias; Schade, Robert; Kühne, Thomas; Plessl, Christian (2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM))
The computation of electron repulsion integrals (ERIs) over Gaussian-type orbitals (GTOs) is a challenging problem in quantum-mechanics-based atomistic simulations. In practical simulations, several trillions of ERIs may have to be computed for every time step. In this work, we investigate FPGAs as accelerators for the ERI computation. We use template parameters, here within the Intel oneAPI tool flow, to create customized designs for 256 different ERI quartet classes, based on their orbitals. To maximize data reuse, all intermediates are buffered in FPGA on-chip memory with customized layout. The pre-calculation of intermediates also helps to overcome data dependencies caused by multi-dimensional recurrence relations. The involved loop structures are partially or even fully unrolled for high throughput of FPGA kernels. Furthermore, a lossy compression algorithm utilizing arbitrary bitwidth integers is integrated in the FPGA kernels. To our best knowledge, this is the first work on ERI computation on FPGAs that supports more than just the single most basic quartet class. Also, the integration of ERI computation and compression it a novelty that is not even covered by CPU or GPU libraries so far. Our evaluation shows that using 16-bit integer for the ERI compression, the fastest FPGA kernels exceed the performance of 10 GERIS ($10 × 10^9$ ERIs per second) on one Intel Stratix 10 GX 2800 FPGA, with maximum absolute errors around $10^{-7}$ - $10^{-5}$ Hartree. The measured throughput can be accurately explained by a performance model. The FPGA kernels deployed on 2 FPGAs outperform similar computations using the widely used libint reference on a two-socket server with 40 Xeon Gold 6148 CPU cores of the same process technology by factors up to 6.0x and on a new two-socket server with 128 EPYC 7713 CPU cores by up to 1.9x.

Breaking the exascale barrier for the electronic structure problem in ab-initio molecular dynamics - Schade, Robert; Kenter, Tobias; Elgabarty, Hossam; Lass, Michael; Kühne, Thomas; Plessl, Christian (The International Journal of High Performance Computing Applications)
The non-orthogonal local submatrix method applied to electronic structure–based molecular dynamics simulations is shown to exceed 1.1 EFLOP/s in FP16/FP32-mixed floating-point arithmetic when using 4400 NVIDIA A100 GPUs of the Perlmutter system. This is enabled by a modification of the original method that pushes the sustained fraction of the peak performance to about 80%. Example calculations are performed for SARS-CoV-2 spike proteins with up to 83 million atoms.
Keywords: Hardware and Architecture; Theoretical Computer Science; Software

Shallow Water DG Simulations on FPGAs: Design and Comparison of a Novel Code Generation Pipeline - Alt, Christoph; Kenter, Tobias; Faghih-Naini, Sara; Faj, Jennifer; Opdenhövel, Jan-Oliver; Plessl, Christian; Aizinger, Vadym; Hönig, Jan; Köstler, Harald (Lecture Notes in Computer Science)

A computation of D(9) using FPGA Supercomputing - Van Hirtum, Lennart; De Causmaecker, Patrick; Goemaere, Jens; Kenter, Tobias; Riebler, Heinrich; Lass, Michael; Plessl, Christian (arXiv:2304.03039)
This preprint makes the claim of having computed the $9^{th}$ Dedekind Number. This was done by building an efficient FPGA Accelerator for the core operation of the process, and parallelizing it on the Noctua 2 Supercluster at Paderborn University. The resulting value is 286386577668298411128469151667598498812366. This value can be verified in two steps. We have made the data file containing the 490M results available, each of which can be verified separately on CPU, and the whole file sums to our proposed value.

Scalable Multi-FPGA Design of a Discontinuous Galerkin Shallow-Water Model on Unstructured Meshes - Faj, Jennifer; Kenter, Tobias; Faghih-Naini, Sara; Plessl, Christian; Aizinger, Vadym (Proceedings of the Platform for Advanced Scientific Computing Conference (PASC))

Compute Centers I: Heterogeneous Execution Environments - Hansmeier, Tim; Kenter, Tobias; Meyer, Marius; Riebler, Heinrich; Platzner, Marco; Plessl, Christian (On-The-Fly Computing -- Individualized IT-services in dynamic markets)

Mutation Tree Reconstruction of Tumor Cells on FPGAs Using a Bit-Level Matrix Representation - Opdenhövel, Jan-Oliver; Plessl, Christian; Kenter, Tobias (Proceedings of the 13th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART))

Roadmap on Electronic Structure Codes in the Exascale Era - Gavini, Vikram; Baroni, Stefano; Blum, Volker; Bowler, David R.; Buccheri, Alexander; Chelikowsky, James R.; Das, Sambit; Dawson, William; Delugas, Pietro; Dogan, Mehmet; Draxl, Claudia; Galli, Giulia; et al. (arXiv:2209.12747)
Electronic structure calculations have been instrumental in providing many important insights into a range of physical and chemical properties of various molecular and solid-state systems. Their importance to various fields, including materials science, chemical sciences, computational chemistry and device physics, is underscored by the large fraction of available public supercomputing resources devoted to these calculations. As we enter the exascale era, exciting new opportunities to increase simulation numbers, sizes, and accuracies present themselves. In order to realize these promises, the community of electronic structure software developers will however first have to tackle a number of challenges pertaining to the efficient use of new architectures that will rely heavily on massive parallelism and hardware accelerators. This roadmap provides a broad overview of the state-of-the-art in electronic structure calculations and of the various new directions being pursued by the community. It covers 14 electronic structure codes, presenting their current status, their development priorities over the next five years, and their plans towards tackling the challenges and leveraging the opportunities presented by the advent of exascale computing.

A High-Fidelity Flow Solver for Unstructured Meshes on Field-Programmable Gate Arrays: Design, Evaluation, and Future Challenges - Karp, Martin; Podobas, Artur; Kenter, Tobias; Jansson, Niclas; Plessl, Christian; Schlatter, Philipp; Markidis, Stefano (International Conference on High Performance Computing in Asia-Pacific Region)

CP2K on the road to exascale - Kühne, Thomas; Plessl, Christian; Schade, Robert; Schütt, Ole (arXiv:2205.14741)
The CP2K program package, which can be considered as the swiss army knife of atomistic simulations, is presented with a special emphasis on ab-initio molecular dynamics using the second-generation Car-Parrinello method. After outlining current and near-term development efforts with regards to massively parallel low-scaling post-Hartree-Fock and eigenvalue solvers, novel approaches on how we plan to take full advantage of future low-precision hardware architectures are introduced. Our focus here is on combining our submatrix method with the approximate computing paradigm to address the immanent exascale era.

Parallel quantum chemistry on noisy intermediate-scale quantum computers - Schade, Robert; Bauer, Carsten; Tamoev, Konstantin; Mazur, Lukas; Plessl, Christian; Kühne, Thomas (Phys. Rev. Research)
A parallel hybrid quantum-classical algorithm for the solution of the quantum-chemical ground-state energy problem on gate-based quantum computers is presented. This approach is based on the reduced density-matrix functional theory (RDMFT) formulation of the electronic structure problem. For that purpose, the density-matrix functional of the full system is decomposed into an indirectly coupled sum of density-matrix functionals for all its subsystems using the adaptive cluster approximation to RDMFT. The approximations involved in the decomposition and the adaptive cluster approximation itself can be systematically converged to the exact result. The solutions for the density-matrix functionals of the effective subsystems involves a constrained minimization over many-particle states that are approximated by parametrized trial states on the quantum computer similarly to the variational quantum eigensolver. The independence of the density-matrix functionals of the effective subsystems introduces a new level of parallelization and allows for the computational treatment of much larger molecules on a quantum computer with a given qubit count. In addition, for the proposed algorithm techniques are presented to reduce the qubit count, the number of quantum programs, as well as its depth. The evaluation of a density-matrix functional as the essential part of our approach is demonstrated for Hubbard-like systems on IBM quantum computers based on superconducting transmon qubits.

Roadmap on Electronic Structure Codes in the Exascale Era - Gavini, Vikram; Baroni, Stefano; Blum, Volker; Bowler, David R.; Buccheri, Alexander; Chelikowsky, James R.; Das, Sambit; Dawson, William; Delugas, Pietro; Dogan, Mehmet; Draxl, Claudia; Galli, Giulia; et al. (arXiv:2209.12747)
Electronic structure calculations have been instrumental in providing many important insights into a range of physical and chemical properties of various molecular and solid-state systems. Their importance to various fields, including materials science, chemical sciences, computational chemistry and device physics, is underscored by the large fraction of available public supercomputing resources devoted to these calculations. As we enter the exascale era, exciting new opportunities to increase simulation numbers, sizes, and accuracies present themselves. In order to realize these promises, the community of electronic structure software developers will however first have to tackle a number of challenges pertaining to the efficient use of new architectures that will rely heavily on massive parallelism and hardware accelerators. This roadmap provides a broad overview of the state-of-the-art in electronic structure calculations and of the various new directions being pursued by the community. It covers 14 electronic structure codes, presenting their current status, their development priorities over the next five years, and their plans towards tackling the challenges and leveraging the opportunities presented by the advent of exascale computing.

Towards electronic structure-based ab-initio molecular dynamics simulations with hundreds of millions of atoms - Schade, Robert; Kenter, Tobias; Elgabarty, Hossam; Lass, Michael; Schütt, Ole; Lazzaro, Alfio; Pabst, Hans; Mohr, Stephan; Hutter, Jürg; Kühne, Thomas; Plessl, Christian (Parallel Computing)
Keywords: Artificial Intelligence; Computer Graphics and Computer-Aided Design; Computer Networks and Communications; Hardware and Architecture; Theoretical Computer Science; Software

In-depth FPGA Accelerator Performance Evaluation with Single Node Benchmarks from the HPC Challenge Benchmark Suite for Intel and Xilinx FPGAs using OpenCL - Meyer, Marius; Kenter, Tobias; Plessl, Christian (Journal of Parallel and Distributed Computing)

The Strong Scaling Advantage of FPGAs in HPC for N-body Simulations - Menzel, Johannes; Plessl, Christian; Kenter, Tobias (ACM Transactions on Reconfigurable Technology and Systems)
N-body methods are one of the essential algorithmic building blocks of high-performance and parallel computing. Previous research has shown promising performance for implementing n-body simulations with pairwise force calculations on FPGAs. However, to avoid challenges with accumulation and memory access patterns, the presented designs calculate each pair of forces twice, along with both force sums of the involved particles. Also, they require large problem instances with hundreds of thousands of particles to reach their respective peak performance, limiting the applicability for strong scaling scenarios. This work addresses both issues by presenting a novel FPGA design that uses each calculated force twice and overlaps data transfers and computations in a way that allows to reach peak performance even for small problem instances, outperforming previous single precision results even in double precision, and scaling linearly over multiple interconnected FPGAs. For a comparison across architectures, we provide an equally optimized CPU reference, which for large problems actually achieves higher peak performance per device, however, given the strong scaling advantages of the FPGA design, in parallel setups with few thousand particles per device, the FPGA platform achieves highest performance and power efficiency.

High-Performance Spectral Element Methods on Field-Programmable Gate Arrays : Implementation, Evaluation, and Future Projection - Karp, Martin; Podobas, Artur; Jansson, Niclas; Kenter, Tobias; Plessl, Christian; Schlatter, Philipp; Markidis, Stefano (2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS))

HighPerMeshes – A Domain-Specific Language for Numerical Algorithms on Unstructured Grids - Alhaddad, Samer; Förstner, Jens; Groth, Stefan; Grünewald, Daniel; Grynko, Yevgen; Hannig, Frank; Kenter, Tobias; Pfreundt, Franz-Josef; Plessl, Christian; Schotte, Merlind; Steinke, Thomas; Teich, Jürgen; et al. (Euro-Par 2020: Parallel Processing Workshops)
Solving partial differential equations on unstructured grids is a cornerstone of engineering and scientific computing. Nowadays, heterogeneous parallel platforms with CPUs, GPUs, and FPGAs enable energy-efficient and computationally demanding simulations. We developed the HighPerMeshes C++-embedded Domain-Specific Language (DSL) for bridging the abstraction gap between the mathematical and algorithmic formulation of mesh-based algorithms for PDE problems on the one hand and an increasing number of heterogeneous platforms with their different parallel programming and runtime models on the other hand. Thus, the HighPerMeshes DSL aims at higher productivity in the code development process for multiple target platforms. We introduce the concepts as well as the basic structure of the HighPerMeshes DSL, and demonstrate its usage with three examples, a Poisson and monodomain problem, respectively, solved by the continuous finite element method, and the discontinuous Galerkin method for Maxwell’s equation. The mapping of the abstract algorithmic description onto parallel hardware, including distributed memory compute clusters, is presented. Finally, the achievable performance and scalability are demonstrated for a typical example problem on a multi-core CPU cluster.
Keywords: tet_topic_hpc

Evaluating the Design Space for Offloading 3D FFT Calculations to an FPGA for High-Performance Computing - Ramaswami, Arjun; Kenter, Tobias; Kühne, Thomas; Plessl, Christian (Applied Reconfigurable Computing. Architectures, Tools, and Applications)

The HighPerMeshes framework for numerical algorithms on unstructured grids - Alhaddad, Samer; Förstner, Jens; Groth, Stefan; Grünewald, Daniel; Grynko, Yevgen; Hannig, Frank; Kenter, Tobias; Pfreundt, Franz‐Josef; Plessl, Christian; Schotte, Merlind; Steinke, Thomas; Teich, Jürgen; et al. (Concurrency and Computation: Practice and Experience)
Keywords: tet_topic_hpc

High-Performance Spectral Element Methods on Field-Programmable Gate Arrays : Implementation, Evaluation, and Future Projection - Karp, Martin; Podobas, Artur; Jansson, Niclas; Kenter, Tobias; Plessl, Christian; Schlatter, Philipp; Markidis, Stefano (2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS))

CP2K: An electronic structure and molecular dynamics software package - Quickstep: Efficient and accurate electronic structure calculations - Kühne, Thomas; Iannuzzi, Marcella; Ben, Mauro Del; Rybkin, Vladimir V.; Seewald, Patrick; Stein, Frederick; Laino, Teodoro; Khaliullin, Rustam Z.; Schütt, Ole; Schiffmann, Florian; Golze, Dorothea; Wilhelm, Jan; et al. (The Journal of Chemical Physics)
CP2K is an open source electronic structure and molecular dynamics software package to perform atomistic simulations of solid-state, liquid, molecular, and biological systems. It is especially aimed at massively parallel and linear-scaling electronic structure methods and state-of-theart ab initio molecular dynamics simulations. Excellent performance for electronic structure calculations is achieved using novel algorithms implemented for modern high-performance computing systems. This review revisits the main capabilities of CP2K to perform efficient and accurate electronic structure simulations. The emphasis is put on density functional theory and multiple post–Hartree–Fock methods using the Gaussian and plane wave approach and its augmented all-electron extension.

A Submatrix-Based Method for Approximate Matrix Function Evaluation in the Quantum Chemistry Code CP2K - Lass, Michael; Schade, Robert; Kühne, Thomas; Plessl, Christian (Proc. International Conference for High Performance Computing, Networking, Storage and Analysis (SC))
Electronic structure calculations based on density-functional theory (DFT) represent a significant part of today's HPC workloads and pose high demands on high-performance computing resources. To perform these quantum-mechanical DFT calculations on complex large-scale systems, so-called linear scaling methods instead of conventional cubic scaling methods are required. In this work, we take up the idea of the submatrix method and apply it to the DFT computations in the software package CP2K. For that purpose, we transform the underlying numeric operations on distributed, large, sparse matrices into computations on local, much smaller and nearly dense matrices. This allows us to exploit the full floating-point performance of modern CPUs and to make use of dedicated accelerator hardware, where performance has been limited by memory bandwidth before. We demonstrate both functionality and performance of our implementation and show how it can be accelerated with GPUs and FPGAs.

Evaluating FPGA Accelerator Performance with a Parameterized OpenCL Adaptation of Selected Benchmarks of the HPCChallenge Benchmark Suite - Meyer, Marius; Kenter, Tobias; Plessl, Christian (2020 IEEE/ACM International Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC))
FPGAs have found increasing adoption in data center applications since a new generation of high-level tools have become available which noticeably reduce development time for FPGA accelerators and still provide high-quality results. There is, however, no high-level benchmark suite available, which specifically enables a comparison of FPGA architectures, programming tools, and libraries for HPC applications. To fill this gap, we have developed an OpenCL-based open-source implementation of the HPCC benchmark suite for Xilinx and Intel FPGAs. This benchmark can serve to analyze the current capabilities of FPGA devices, cards, and development tool flows, track progress over time, and point out specific difficulties for FPGA acceleration in the HPC domain. Additionally, the benchmark documents proven performance optimization patterns. We will continue optimizing and porting the benchmark for new generations of FPGAs and design tools and encourage active participation to create a valuable tool for the community. To fill this gap, we have developed an OpenCL-based open-source implementation of the HPCC benchmark suite for Xilinx and Intel FPGAs. This benchmark can serve to analyze the current capabilities of FPGA devices, cards, and development tool flows, track progress over time, and point out specific difficulties for FPGA acceleration in the HPC domain. Additionally, the benchmark documents proven performance optimization patterns. We will continue optimizing and porting the benchmark for new generations of FPGAs and design tools and encourage active participation to create a valuable tool for the community.
Keywords: FPGA; OpenCL; High Level Synthesis; HPC benchmarking

Accurate Sampling with Noisy Forces from Approximate Computing - Rengaraj, Varadarajan; Lass, Michael; Plessl, Christian; Kühne, Thomas (Computation)
In scientific computing, the acceleration of atomistic computer simulations by means of custom hardware is finding ever-growing application. A major limitation, however, is that the high efficiency in terms of performance and low power consumption entails the massive usage of low precision computing units. Here, based on the approximate computing paradigm, we present an algorithmic method to compensate for numerical inaccuracies due to low accuracy arithmetic operations rigorously, yet still obtaining exact expectation values using a properly modified Langevin-type equation.

Transparent Acceleration for Heterogeneous Platforms with Compilation to OpenCL - Riebler, Heinrich; Vaz, Gavin Francis; Kenter, Tobias; Plessl, Christian (ACM Trans. Archit. Code Optim. (TACO))
Keywords: htrop

OpenCL Implementation of Cannon's Matrix Multiplication Algorithm on Intel Stratix 10 FPGAs - Gorlani, Paolo; Kenter, Tobias; Plessl, Christian (Proceedings of the International Conference on Field-Programmable Technology (FPT))
Stratix 10 FPGA cards have a good potential for the acceleration of HPC workloads since the Stratix 10 product line introduces devices with a large number of DSP and memory blocks. The high level synthesis of OpenCL codes can play a fundamental role for FPGAs in HPC, because it allows to implement different designs with lower development effort compared to hand optimized HDL. However, Stratix 10 cards are still hard to fully exploit using the Intel FPGA SDK for OpenCL. The implementation of designs with thousands of concurrent arithmetic operations often suffers from place and route problems that limit the maximum frequency or entirely prevent a successful synthesis. In order to overcome these issues for the implementation of the matrix multiplication, we formulate Cannon's matrix multiplication algorithm with regard to its efficient synthesis within the FPGA logic. We obtain a two-level block algorithm, where the lower level sub-matrices are multiplied using our Cannon's algorithm implementation. Following this design approach with multiple compute units, we are able to get maximum frequencies close to and above 300 MHz with high utilization of DSP and memory blocks. This allows for performance results above 1 TeraFLOPS.

A General Algorithm to Calculate the Inverse Principal p-th Root of Symmetric Positive Definite Matrices - Richters, Dorothee; Lass, Michael; Walther, Andrea; Plessl, Christian; Kühne, Thomas (Communications in Computational Physics)
We address the general mathematical problem of computing the inverse p-th root of a given matrix in an efficient way. A new method to construct iteration functions that allow calculating arbitrary p-th roots and their inverses of symmetric positive definite matrices is presented. We show that the order of convergence is at least quadratic and that adaptively adjusting a parameter q always leads to an even faster convergence. In this way, a better performance than with previously known iteration schemes is achieved. The efficiency of the iterative functions is demonstrated for various matrices with different densities, condition numbers and spectral radii.

FPGAs im Rechenzentrum - Platzner, Marco; Plessl, Christian (Informatik Spektrum)

Using Approximate Computing for the Calculation of Inverse Matrix p-th Roots - Lass, Michael; Kühne, Thomas; Plessl, Christian (Embedded Systems Letters)
Approximate computing has shown to provide new ways to improve performance and power consumption of error-resilient applications. While many of these applications can be found in image processing, data classification or machine learning, we demonstrate its suitability to a problem from scientific computing. Utilizing the self-correcting behavior of iterative algorithms, we show that approximate computing can be applied to the calculation of inverse matrix p-th roots which are required in many applications in scientific computing. Results show great opportunities to reduce the computational effort and bandwidth required for the execution of the discussed algorithm, especially when targeting special accelerator hardware.

Sprint diagnostic with GPS and inertial sensor fusion - Mertens, Jan Cedric; Boschmann, Alexander; Schmidt, M.; Plessl, Christian (Sports Engineering)

OpenCL-based FPGA Design to Accelerate the Nodal Discontinuous Galerkin Method for Unstructured Meshes - Kenter, Tobias; Mahale, Gopinath; Alhaddad, Samer; Grynko, Yevgen; Schmitt, Christian; Afzal, Ayesha; Hannig, Frank; Förstner, Jens; Plessl, Christian (Proc. Int. Symp. on Field-Programmable Custom Computing Machines (FCCM))
The exploration of FPGAs as accelerators for scientific simulations has so far mostly been focused on small kernels of methods working on regular data structures, for example in the form of stencil computations for finite difference methods. In computational sciences, often more advanced methods are employed that promise better stability, convergence, locality and scaling. Unstructured meshes are shown to be more effective and more accurate, compared to regular grids, in representing computation domains of various shapes. Using unstructured meshes, the discontinuous Galerkin method preserves the ability to perform explicit local update operations for simulations in the time domain. In this work, we investigate FPGAs as target platform for an implementation of the nodal discontinuous Galerkin method to find time-domain solutions of Maxwell's equations in an unstructured mesh. When maximizing data reuse and fitting constant coefficients into suitably partitioned on-chip memory, high computational intensity allows us to implement and feed wide data paths with hundreds of floating point operators. By decoupling off-chip memory accesses from the computations, high memory bandwidth can be sustained, even for the irregular access pattern required by parts of the application. Using the Intel/Altera OpenCL SDK for FPGAs, we present different implementation variants for different polynomial orders of the method. In different phases of the algorithm, either computational or bandwidth limits of the Arria 10 platform are almost reached, thus outperforming a highly multithreaded CPU implementation by around 2x.
Keywords: tet_topic_hpc

A Massively Parallel Algorithm for the Approximate Calculation of Inverse p-th Roots of Large Sparse Matrices - Lass, Michael; Mohr, Stephan; Wiebeler, Hendrik; Kühne, Thomas; Plessl, Christian (Proc. Platform for Advanced Scientific Computing (PASC) Conference)
We present the submatrix method, a highly parallelizable method for the approximate calculation of inverse p-th roots of large sparse symmetric matrices which are required in different scientific applications. Following the idea of Approximate Computing, we allow imprecision in the final result in order to utilize the sparsity of the input matrix and to allow massively parallel execution. For an n x n matrix, the proposed algorithm allows to distribute the calculations over n nodes with only little communication overhead. The result matrix exhibits the same sparsity pattern as the input matrix, allowing for efficient reuse of allocated data structures. We evaluate the algorithm with respect to the error that it introduces into calculated results, as well as its performance and scalability. We demonstrate that the error is relatively limited for well-conditioned matrices and that results are still valuable for error-resilient applications like preconditioning even for ill-conditioned matrices. We discuss the execution time and scaling of the algorithm on a theoretical level and present a distributed implementation of the algorithm using MPI and OpenMP. We demonstrate the scalability of this implementation by running it on a high-performance compute cluster comprised of 1024 CPU cores, showing a speedup of 665x compared to single-threaded execution.
Keywords: approximate computing; linear algebra; matrix inversion; matrix p-th roots; numeric algorithm; parallel computing

Automated Code Acceleration Targeting Heterogeneous OpenCL Devices - Riebler, Heinrich; Vaz, Gavin Francis; Kenter, Tobias; Plessl, Christian (Proc. ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP))
Keywords: htrop

Efficient Branch and Bound on FPGAs Using Work Stealing and Instance-Specific Designs - Riebler, Heinrich; Lass, Michael; Mittendorf, Robert; Löcke, Thomas; Plessl, Christian (ACM Transactions on Reconfigurable Technology and Systems (TRETS))
Branch and bound (B&B) algorithms structure the search space as a tree and eliminate infeasible solutions early by pruning subtrees that cannot lead to a valid or optimal solution. Custom hardware designs significantly accelerate the execution of these algorithms. In this article, we demonstrate a high-performance B&B implementation on FPGAs. First, we identify general elements of B&B algorithms and describe their implementation as a finite state machine. Then, we introduce workers that autonomously cooperate using work stealing to allow parallel execution and full utilization of the target FPGA. Finally, we explore advantages of instance-specific designs that target a specific problem instance to improve performance. We evaluate our concepts by applying them to a branch and bound problem, the reconstruction of corrupted AES keys obtained from cold-boot attacks. The evaluation shows that our work stealing approach is scalable with the available resources and provides speedups proportional to the number of workers. Instance-specific designs allow us to achieve an overall speedup of 47 × compared to the fastest implementation of AES key reconstruction so far. Finally, we demonstrate how instance-specific designs can be generated just-in-time such that the provided speedups outweigh the additional time required for design synthesis.
Keywords: coldboot

Flexible FPGA design for FDTD using OpenCL - Kenter, Tobias; Förstner, Jens; Plessl, Christian (Proc. Int. Conf. on Field Programmable Logic and Applications (FPL))
Compared to classical HDL designs, generating FPGA with high-level synthesis from an OpenCL specification promises easier exploration of different design alternatives and, through ready-to-use infrastructure and common abstractions for host and memory interfaces, easier portability between different FPGA families. In this work, we evaluate the extent of this promise. To this end, we present a parameterized FDTD implementation for photonic microcavity simulations. Our design can trade-off different forms of parallelism and works for two independent OpenCL-based FPGA design flows. Hence, we can target FPGAs from different vendors and different FPGA families. We describe how we used pre-processor macros to achieve this flexibility and to work around different shortcomings of the current tools. Choosing the right design configurations, we are able to present two extremely competitive solutions for very different FPGA targets, reaching up to 172 GFLOPS sustained performance. With the portability and flexibility demonstrated, code developers not only avoid vendor lock-in, but can even make best use of real trade-offs between different architectures.
Keywords: tet_topic_hpc

High-Throughput and Low-Latency Network Communication with NetIO - Schumacher, Jörn; Plessl, Christian; Vandelli, Wainer (Journal of Physics: Conference Series)

ReconOS - Agne, Andreas; Platzner, Marco; Plessl, Christian; Happe, Markus; Lübbers, Enno (FPGAs for Software Programmers)
In this chapter, we present an introduction to the ReconOS operating system for reconfigurable computing. ReconOS offers a unified multi-threaded programming model and operating system services for threads executing in software and threads mapped to reconfigurable hardware. By supporting standard POSIX operating system functions for both software and hardware threads, ReconOS particularly caters to developers with a software background, because developers can use well-known mechanisms such as semaphores, mutexes, condition variables, and message queues for developing hybrid applications with threads running on the CPU and FPGA concurrently. Through the semantic integration of hardware accelerators into a standard operating system environment, ReconOS allows for rapid design space exploration, supports a structured application development process and improves the portability of applications between different reconfigurable computing systems.

Using Just-in-Time Code Generation for Transparent Resource Management in Heterogeneous Systems - Riebler, Heinrich; Vaz, Gavin Francis; Plessl, Christian; Trainiti, Ettore M. G.; Durelli, Gianluca C.; Bolchini, Cristiana (Proc. HiPEAC Workshop on Reonfigurable Computing (WRC))

Microdisk Cavity FDTD Simulation on FPGA using OpenCL - Kenter, Tobias; Plessl, Christian (Proc. Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC))

Using Approximate Computing in Scientific Codes - Lass, Michael; Kühne, Thomas; Plessl, Christian (Workshop on Approximate Computing (AC))

Using Just-in-Time Code Generation for Transparent Resource Management in Heterogeneous Systems - Riebler, Heinrich; Vaz, Gavin Francis; Plessl, Christian; Trainiti, Ettore M. G.; Durelli, Gianluca C.; Del Sozzo, Emanuele; Santambrogio, Marco D.; Bolchini, Christina (Proceedings of International Forum on Research and Technologies for Society and Industry (RTSI))
Hardware accelerators are becoming popular in academia and industry. To move one step further from the state-of-the-art multicore plus accelerator approaches, we present in this paper our innovative SAVEHSA architecture. It comprises of a heterogeneous hardware platform with three different high-end accelerators attached over PCIe (GPGPU, FPGA and Intel MIC). Such systems can process parallel workloads very efficiently whilst being more energy efficient than regular CPU systems. To leverage the heterogeneity, the workload has to be distributed among the computing units in a way that each unit is well-suited for the assigned task and executable code must be available. To tackle this problem we present two software components; the first can perform resource allocation at runtime while respecting system and application goals (in terms of throughput, energy, latency, etc.) and the second is able to analyze an application and generate executable code for an accelerator at runtime. We demonstrate the first proof-of-concept implementation of our framework on the heterogeneous platform, discuss different runtime policies and measure the introduced overheads.

Self-aware Compute Nodes - Agne, Andreas; Happe, Markus; Lösch, Achim; Plessl, Christian; Platzner, Marco (Self-aware Computing Systems)
Many modern compute nodes are heterogeneous multi-cores that integrate several CPU cores with fixed function or reconfigurable hardware cores. Such systems need to adapt task scheduling and mapping to optimise for performance and energy under varying workloads and, increasingly important, for thermal and fault management and are thus relevant targets for self-aware computing. In this chapter, we take up the generic reference architecture for designing self-aware and self-expressive computing systems and refine it for heterogeneous multi-cores. We present ReconOS, an architecture, programming model and execution environment for heterogeneous multi-cores, and show how the components of the reference architecture can be implemented on top of ReconOS. In particular, the unique feature of dynamic partial reconfiguration supports self-expression through starting and terminating reconfigurable hardware cores. We detail a case study that runs two applications on an architecture with one CPU and 12 reconfigurable hardware cores and present self-expression strategies for adapting under performance, temperature and even conflicting constraints. The case study demonstrates that the reference architecture as a model for self-aware computing is highly useful as it allows us to structure and simplify the design process, which will be essential for designing complex future compute nodes. Furthermore, ReconOS is used as a base technology for flexible protocol stacks in Chapter 10, an approach for self-aware computing at the networking level.

Potential and Methods for Embedding Dynamic Offloading Decisions into Application Code - Vaz, Gavin Francis; Riebler, Heinrich; Kenter, Tobias; Plessl, Christian (Computers and Electrical Engineering)
A broad spectrum of applications can be accelerated by offloading computation intensive parts to reconfigurable hardware. However, to achieve speedups, the number of loop it- erations (trip count) needs to be sufficiently large to amortize offloading overheads. Trip counts are frequently not known at compile time, but only at runtime just before entering a loop. Therefore, we propose to generate code for both the CPU and the coprocessor, and defer the offloading decision to the application runtime. We demonstrate how a toolflow, based on the LLVM compiler framework, can automatically embed dynamic offloading de- cisions into the application code. We perform in-depth static and dynamic analysis of pop- ular benchmarks, which confirm the general potential of such an approach. We also pro- pose to optimize the offloading process by decoupling the runtime decision from the loop execution (decision slack). The feasibility of our approach is demonstrated by a toolflow that automatically identifies suitable data-parallel loops and generates code for the FPGA coprocessor of a Convey HC-1. We evaluate the integrated toolflow with representative loops executed for different input data sizes.

Performance-centric scheduling with task migration for a heterogeneous compute node in the data center - Lösch, Achim; Beisel, Tobias; Kenter, Tobias; Plessl, Christian; Platzner, Marco (Proceedings of the 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE))
The use of heterogeneous computing resources, such as Graphic Processing Units or other specialized coprocessors, has become widespread in recent years because of their per- formance and energy efficiency advantages. Approaches for managing and scheduling tasks to heterogeneous resources are still subject to research. Although queuing systems have recently been extended to support accelerator resources, a general solution that manages heterogeneous resources at the operating system- level to exploit a global view of the system state is still missing.In this paper we present a user space scheduler that enables task scheduling and migration on heterogeneous processing resources in Linux. Using run queues for available resources we perform scheduling decisions based on the system state and on task characterization from earlier measurements. With a pro- gramming pattern that supports the integration of checkpoints into applications, we preempt tasks and migrate them between three very different compute resources. Considering static and dynamic workload scenarios, we show that this approach can gain up to 17% performance, on average 7%, by effectively avoiding idle resources. We demonstrate that a work-conserving strategy without migration is no suitable alternative.