TY - JOUR
AB - The rise of exascale supercomputers has fueled competition among GPU vendors, driving lattice QCD developers to write code that supports multiple APIs. Moreover, new developments in algorithms and physics research require frequent updates to existing software. These challenges have to be balanced against constantly changing personnel. At the same time, there is a wide range of applications for HISQ fermions in QCD studies. This situation encourages the development of software featuring a HISQ action that is flexible, high-performing, open source, easy to use, and easy to adapt. In this technical paper, we explain the design strategy, provide implementation details, list available algorithms and modules, and show key performance indicators for SIMULATeQCD, a simple multi-GPU lattice code for large-scale QCD calculations, mainly developed and used by the HotQCD collaboration. The code is publicly available on GitHub.
AU - Mazur, Lukas
AU - Bollweg, Dennis
AU - Clarke, David A.
AU - Altenkort, Luis
AU - Kaczmarek, Olaf
AU - Larsen, Rasmus
AU - Shu, Hai-Tao
AU - Goswami, Jishnu
AU - Scior, Philipp
AU - Sandmeyer, Hauke
AU - Neumann, Marius
AU - Dick, Henrik
AU - Ali, Sajid
AU - Kim, Jangho
AU - Schmidt, Christian
AU - Petreczky, Peter
AU - Mukherjee, Swagato
ID - 46120
JF - Computer Physics Communications
TI - SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations
ER -
TY - JOUR
AU - Altenkort, Luis
AU - Eller, Alexander M.
AU - Francis, Anthony
AU - Kaczmarek, Olaf
AU - Mazur, Lukas
AU - Moore, Guy D.
AU - Shu, Hai-Tao
ID - 46119
IS - 1
JF - Physical Review D
SN - 2470-0010
TI - Viscosity of pure-glue QCD from the lattice
VL - 108
ER -
TY - JOUR
AB - While FPGA accelerator boards and their respective high-level design tools are maturing, there is still a lack of multi-FPGA applications, libraries, and not least, benchmarks and reference implementations towards sustained HPC usage of these devices. As in the early days of GPUs in HPC, for workloads that can reasonably be decoupled into loosely coupled working sets, multi-accelerator support can be achieved by using standard communication interfaces like MPI on the host side. However, for performance and productivity, some applications can profit from a tighter coupling of the accelerators. FPGAs offer unique opportunities here when extending the dataflow characteristics to their communication interfaces.
In this work, we extend the HPCC FPGA benchmark suite by multi-FPGA support and three missing benchmarks that particularly characterize or stress inter-device communication: b_eff, PTRANS, and LINPACK. With all benchmarks implemented for current boards with Intel and Xilinx FPGAs, we established a baseline for multi-FPGA performance. Additionally, for the communication-centric benchmarks, we explored the potential of direct FPGA-to-FPGA communication with a circuit-switched inter-FPGA network that is currently only available for one of the boards. The evaluation with parallel execution on up to 26 FPGA boards makes use of one of the largest academic FPGA installations.
AU - Meyer, Marius
AU - Kenter, Tobias
AU - Plessl, Christian
ID - 38041
JF - ACM Transactions on Reconfigurable Technology and Systems
KW - General Computer Science
SN - 1936-7406
TI - Multi-FPGA Designs and Scaling of HPC Challenge Benchmarks via MPI and Circuit-Switched Inter-FPGA Networks
ER -
TY - CHAP
AU - Hansmeier, Tim
AU - Kenter, Tobias
AU - Meyer, Marius
AU - Riebler, Heinrich
AU - Platzner, Marco
AU - Plessl, Christian
ED - Haake, Claus-Jochen
ED - Meyer auf der Heide, Friedhelm
ED - Platzner, Marco
ED - Wachsmuth, Henning
ED - Wehrheim, Heike
ID - 45893
T2 - On-The-Fly Computing -- Individualized IT-services in dynamic markets
TI - Compute Centers I: Heterogeneous Execution Environments
VL - 412
ER -
TY - CONF
AU - Opdenhövel, Jan-Oliver
AU - Plessl, Christian
AU - Kenter, Tobias
ID - 46190
T2 - Proceedings of the 13th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies
TI - Mutation Tree Reconstruction of Tumor Cells on FPGAs Using a Bit-Level Matrix Representation
ER -
TY - CONF
AU - Faj, Jennifer
AU - Kenter, Tobias
AU - Faghih-Naini, Sara
AU - Plessl, Christian
AU - Aizinger, Vadym
ID - 46188
T2 - Proceedings of the Platform for Advanced Scientific Computing Conference
TI - Scalable Multi-FPGA Design of a Discontinuous Galerkin Shallow-Water Model on Unstructured Meshes
ER -
TY - CONF
AU - Prouveur, Charles
AU - Haefele, Matthieu
AU - Kenter, Tobias
AU - Voss, Nils
ID - 46189
T2 - Proceedings of the Platform for Advanced Scientific Computing Conference
TI - FPGA Acceleration for HPC Supercapacitor Simulations
ER -
TY - CONF
AB - The computation of electron repulsion integrals (ERIs) over Gaussian-type orbitals (GTOs) is a challenging problem in quantum-mechanics-based atomistic simulations. In practical simulations, several trillions of ERIs may have to be
computed for every time step.
In this work, we investigate FPGAs as accelerators for the ERI computation. We use template parameters, here within the Intel oneAPI tool flow, to create customized designs for 256 different ERI quartet classes, based on their orbitals. To maximize data reuse, all intermediates are buffered in FPGA on-chip memory with customized layout. The pre-calculation of intermediates also helps to overcome data dependencies caused by multi-dimensional recurrence
relations. The involved loop structures are partially or even fully unrolled for high throughput of FPGA kernels. Furthermore, a lossy compression algorithm utilizing arbitrary bitwidth integers is integrated in the FPGA kernels. To our
best knowledge, this is the first work on ERI computation on FPGAs that supports more than just the single most basic quartet class. Also, the integration of ERI computation and compression it a novelty that is not even covered by CPU or GPU libraries so far.
Our evaluation shows that using 16-bit integer for the ERI compression, the fastest FPGA kernels exceed the performance of 10 GERIS ($10 \times 10^9$ ERIs per second) on one Intel Stratix 10 GX 2800 FPGA, with maximum absolute errors around $10^{-7}$ - $10^{-5}$ Hartree. The measured throughput can be accurately explained by a performance model. The FPGA kernels deployed on 2 FPGAs outperform similar computations using the widely used libint reference on a two-socket server with 40 Xeon Gold 6148 CPU cores of the same process technology by factors up to 6.0x and on a new two-socket server with 128 EPYC 7713 CPU cores by up to 1.9x.
AU - Wu, Xin
AU - Kenter, Tobias
AU - Schade, Robert
AU - Kühne, Thomas
AU - Plessl, Christian
ID - 43228
T2 - 2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)
TI - Computing and Compressing Electron Repulsion Integrals on FPGAs
ER -
TY - JOUR
AB - The non-orthogonal local submatrix method applied to electronic structure–based molecular dynamics simulations is shown to exceed 1.1 EFLOP/s in FP16/FP32-mixed floating-point arithmetic when using 4400 NVIDIA A100 GPUs of the Perlmutter system. This is enabled by a modification of the original method that pushes the sustained fraction of the peak performance to about 80%. Example calculations are performed for SARS-CoV-2 spike proteins with up to 83 million atoms.
AU - Schade, Robert
AU - Kenter, Tobias
AU - Elgabarty, Hossam
AU - Lass, Michael
AU - Kühne, Thomas
AU - Plessl, Christian
ID - 45361
JF - The International Journal of High Performance Computing Applications
KW - Hardware and Architecture
KW - Theoretical Computer Science
KW - Software
SN - 1094-3420
TI - Breaking the exascale barrier for the electronic structure problem in ab-initio molecular dynamics
ER -
TY - GEN
AB - Viscous hydrodynamics serves as a successful mesoscopic description of the
Quark-Gluon Plasma produced in relativistic heavy-ion collisions. In order to
investigate, how such an effective description emerges from the underlying
microscopic dynamics we calculate the hydrodynamic and non-hydrodynamic modes
of linear response in the sound channel from a first-principle calculation in
kinetic theory. We do this with a new approach wherein we discretize the
collision kernel to directly calculate eigenvalues and eigenmodes of the
evolution operator. This allows us to study the Green's functions at any point
in the complex frequency space. Our study focuses on scalar theory with quartic
interaction and we find that the analytic structure of Green's functions in the
complex plane is far more complicated than just poles or cuts which is a first
step towards an equivalent study in QCD kinetic theory.
AU - Ochsenfeld, Stephan
AU - Schlichting, Sören
ID - 50172
T2 - arXiv:2308.04491
TI - Hydrodynamic and Non-hydrodynamic Excitations in Kinetic Theory -- A Numerical Analysis in Scalar Field Theory
ER -