---
_id: '40171'
abstract:
- lang: eng
text: We present a convolutional framework which significantly reduces the complexity
and thus, the computational effort for distributed reinforcement learning control
of dynamical systems governed by partial differential equations (PDEs). Exploiting
translational equivariances, the high-dimensional distributed control problem
can be transformed into a multi-agent control problem with many identical, uncoupled
agents. Furthermore, using the fact that information is transported with finite
velocity in many cases, the dimension of the agents’ environment can be drastically
reduced using a convolution operation over the state space of the PDE, by which
we effectively tackle the curse of dimensionality otherwise present in deep reinforcement
learning. In this setting, the complexity can be flexibly adjusted via the kernel
width or by using a stride greater than one (meaning that we do not place an actuator
at each sensor location). Moreover, scaling from smaller to larger domains – or
the transfer between different domains – becomes a straightforward task requiring
little effort. We demonstrate the performance of the proposed framework using
several PDE examples with increasing complexity, where stabilization is achieved
by training a low-dimensional deep deterministic policy gradient agent using minimal
computing resources.
article_type: original
author:
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
- first_name: Jan
full_name: Stenner, Jan
id: '65520'
last_name: Stenner
- first_name: Vikas
full_name: Chidananda, Vikas
last_name: Chidananda
- first_name: Oliver
full_name: Wallscheid, Oliver
id: '11291'
last_name: Wallscheid
orcid: https://orcid.org/0000-0001-9362-8777
- first_name: Steven L.
full_name: Brunton, Steven L.
last_name: Brunton
- first_name: Kunihiko
full_name: Taira, Kunihiko
last_name: Taira
citation:
ama: 'Peitz S, Stenner J, Chidananda V, Wallscheid O, Brunton SL, Taira K. Distributed
Control of Partial Differential Equations Using Convolutional Reinforcement Learning.
Physica D: Nonlinear Phenomena. 2024;461:134096. doi:10.1016/j.physd.2024.134096'
apa: 'Peitz, S., Stenner, J., Chidananda, V., Wallscheid, O., Brunton, S. L., &
Taira, K. (2024). Distributed Control of Partial Differential Equations Using
Convolutional Reinforcement Learning. Physica D: Nonlinear Phenomena, 461,
134096. https://doi.org/10.1016/j.physd.2024.134096'
bibtex: '@article{Peitz_Stenner_Chidananda_Wallscheid_Brunton_Taira_2024, title={Distributed
Control of Partial Differential Equations Using Convolutional Reinforcement Learning},
volume={461}, DOI={10.1016/j.physd.2024.134096},
journal={Physica D: Nonlinear Phenomena}, publisher={Elsevier}, author={Peitz,
Sebastian and Stenner, Jan and Chidananda, Vikas and Wallscheid, Oliver and Brunton,
Steven L. and Taira, Kunihiko}, year={2024}, pages={134096} }'
chicago: 'Peitz, Sebastian, Jan Stenner, Vikas Chidananda, Oliver Wallscheid, Steven
L. Brunton, and Kunihiko Taira. “Distributed Control of Partial Differential Equations
Using Convolutional Reinforcement Learning.” Physica D: Nonlinear Phenomena
461 (2024): 134096. https://doi.org/10.1016/j.physd.2024.134096.'
ieee: 'S. Peitz, J. Stenner, V. Chidananda, O. Wallscheid, S. L. Brunton, and K.
Taira, “Distributed Control of Partial Differential Equations Using Convolutional
Reinforcement Learning,” Physica D: Nonlinear Phenomena, vol. 461, p. 134096,
2024, doi: 10.1016/j.physd.2024.134096.'
mla: 'Peitz, Sebastian, et al. “Distributed Control of Partial Differential Equations
Using Convolutional Reinforcement Learning.” Physica D: Nonlinear Phenomena,
vol. 461, Elsevier, 2024, p. 134096, doi:10.1016/j.physd.2024.134096.'
short: 'S. Peitz, J. Stenner, V. Chidananda, O. Wallscheid, S.L. Brunton, K. Taira,
Physica D: Nonlinear Phenomena 461 (2024) 134096.'
date_created: 2023-01-26T07:56:26Z
date_updated: 2024-02-23T10:53:42Z
department:
- _id: '655'
doi: 10.1016/j.physd.2024.134096
intvolume: ' 461'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.1016/j.physd.2024.134096
oa: '1'
page: '134096'
publication: 'Physica D: Nonlinear Phenomena'
publisher: Elsevier
status: public
title: Distributed Control of Partial Differential Equations Using Convolutional
Reinforcement Learning
type: journal_article
user_id: '47427'
volume: 461
year: '2024'
...
---
_id: '54838'
author:
- first_name: Septimus
full_name: Boshoff, Septimus
last_name: Boshoff
- first_name: Jan
full_name: Stenner, Jan
id: '65520'
last_name: Stenner
- first_name: Daniel
full_name: Weber, Daniel
id: '24041'
last_name: Weber
orcid: 0000-0003-3367-5998
- first_name: Marvin
full_name: Meyer, Marvin
last_name: Meyer
- first_name: Vikas
full_name: Chidananda, Vikas
last_name: Chidananda
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
- first_name: Oliver
full_name: Wallscheid, Oliver
id: '11291'
last_name: Wallscheid
orcid: https://orcid.org/0000-0001-9362-8777
citation:
ama: 'Boshoff S, Stenner J, Weber D, et al. Hybrid control of interconnected power
converters using both expert-driven droop and data-driven reinforcement learning
approaches. In: IEEE Power and Energy Student Summit (PESS). VDE; 2023:124-129.'
apa: Boshoff, S., Stenner, J., Weber, D., Meyer, M., Chidananda, V., Peitz, S.,
& Wallscheid, O. (2023). Hybrid control of interconnected power converters
using both expert-driven droop and data-driven reinforcement learning approaches.
IEEE Power and Energy Student Summit (PESS), 124–129.
bibtex: '@inproceedings{Boshoff_Stenner_Weber_Meyer_Chidananda_Peitz_Wallscheid_2023,
title={Hybrid control of interconnected power converters using both expert-driven
droop and data-driven reinforcement learning approaches}, booktitle={IEEE Power
and Energy Student Summit (PESS)}, publisher={VDE}, author={Boshoff, Septimus
and Stenner, Jan and Weber, Daniel and Meyer, Marvin and Chidananda, Vikas and
Peitz, Sebastian and Wallscheid, Oliver}, year={2023}, pages={124–129} }'
chicago: Boshoff, Septimus, Jan Stenner, Daniel Weber, Marvin Meyer, Vikas Chidananda,
Sebastian Peitz, and Oliver Wallscheid. “Hybrid Control of Interconnected Power
Converters Using Both Expert-Driven Droop and Data-Driven Reinforcement Learning
Approaches.” In IEEE Power and Energy Student Summit (PESS), 124–29. VDE,
2023.
ieee: S. Boshoff et al., “Hybrid control of interconnected power converters
using both expert-driven droop and data-driven reinforcement learning approaches,”
in IEEE Power and Energy Student Summit (PESS), 2023, pp. 124–129.
mla: Boshoff, Septimus, et al. “Hybrid Control of Interconnected Power Converters
Using Both Expert-Driven Droop and Data-Driven Reinforcement Learning Approaches.”
IEEE Power and Energy Student Summit (PESS), VDE, 2023, pp. 124–29.
short: 'S. Boshoff, J. Stenner, D. Weber, M. Meyer, V. Chidananda, S. Peitz, O.
Wallscheid, in: IEEE Power and Energy Student Summit (PESS), VDE, 2023, pp. 124–129.'
date_created: 2024-06-21T07:12:10Z
date_updated: 2024-06-21T08:28:45Z
department:
- _id: '655'
- _id: '52'
language:
- iso: eng
main_file_link:
- url: https://ieeexplore.ieee.org/document/10564542
page: 124-129
project:
- _id: '286'
grant_number: 01IS20164
name: 'DARE: DARE: Trainings-, Validierungs- und Benchmarkwerkzeuge zur Entwicklung
datengetriebener Betriebs- und Regelungsverfahren für intelligente, lokale Energiesysteme'
publication: IEEE Power and Energy Student Summit (PESS)
publication_identifier:
isbn:
- 978-3-8007-6318-4
publisher: VDE
status: public
title: Hybrid control of interconnected power converters using both expert-driven
droop and data-driven reinforcement learning approaches
type: conference
user_id: '47427'
year: '2023'
...
---
_id: '54839'
author:
- first_name: Marvin
full_name: Meyer, Marvin
last_name: Meyer
- first_name: Daniel
full_name: Weber, Daniel
id: '24041'
last_name: Weber
orcid: 0000-0003-3367-5998
- first_name: Vikas
full_name: Chidananda, Vikas
last_name: Chidananda
- first_name: Oliver
full_name: Schweins, Oliver
last_name: Schweins
- first_name: Jan
full_name: Stenner, Jan
id: '65520'
last_name: Stenner
- first_name: Septimus
full_name: Boshoff, Septimus
last_name: Boshoff
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
- first_name: Oliver
full_name: Wallscheid, Oliver
id: '11291'
last_name: Wallscheid
orcid: https://orcid.org/0000-0001-9362-8777
citation:
ama: 'Meyer M, Weber D, Chidananda V, et al. ElectricGrid.jl – Automated modeling
of decentralized electrical energy grids. In: IEEE Power and Energy Student
Summit (PESS). VDE; 2023:112-117.'
apa: Meyer, M., Weber, D., Chidananda, V., Schweins, O., Stenner, J., Boshoff, S.,
Peitz, S., & Wallscheid, O. (2023). ElectricGrid.jl – Automated modeling of
decentralized electrical energy grids. IEEE Power and Energy Student Summit
(PESS), 112–117.
bibtex: '@inproceedings{Meyer_Weber_Chidananda_Schweins_Stenner_Boshoff_Peitz_Wallscheid_2023,
title={ElectricGrid.jl – Automated modeling of decentralized electrical energy
grids}, booktitle={IEEE Power and Energy Student Summit (PESS)}, publisher={VDE},
author={Meyer, Marvin and Weber, Daniel and Chidananda, Vikas and Schweins, Oliver
and Stenner, Jan and Boshoff, Septimus and Peitz, Sebastian and Wallscheid, Oliver},
year={2023}, pages={112–117} }'
chicago: Meyer, Marvin, Daniel Weber, Vikas Chidananda, Oliver Schweins, Jan Stenner,
Septimus Boshoff, Sebastian Peitz, and Oliver Wallscheid. “ElectricGrid.Jl – Automated
Modeling of Decentralized Electrical Energy Grids.” In IEEE Power and Energy
Student Summit (PESS), 112–17. VDE, 2023.
ieee: M. Meyer et al., “ElectricGrid.jl – Automated modeling of decentralized
electrical energy grids,” in IEEE Power and Energy Student Summit (PESS),
2023, pp. 112–117.
mla: Meyer, Marvin, et al. “ElectricGrid.Jl – Automated Modeling of Decentralized
Electrical Energy Grids.” IEEE Power and Energy Student Summit (PESS),
VDE, 2023, pp. 112–17.
short: 'M. Meyer, D. Weber, V. Chidananda, O. Schweins, J. Stenner, S. Boshoff,
S. Peitz, O. Wallscheid, in: IEEE Power and Energy Student Summit (PESS), VDE,
2023, pp. 112–117.'
date_created: 2024-06-21T07:13:11Z
date_updated: 2024-06-21T08:28:35Z
department:
- _id: '655'
- _id: '52'
language:
- iso: eng
main_file_link:
- url: https://ieeexplore.ieee.org/document/10564557
page: 112-117
project:
- _id: '286'
grant_number: 01IS20164
name: 'DARE: DARE: Trainings-, Validierungs- und Benchmarkwerkzeuge zur Entwicklung
datengetriebener Betriebs- und Regelungsverfahren für intelligente, lokale Energiesysteme'
publication: IEEE Power and Energy Student Summit (PESS)
publication_identifier:
isbn:
- 978-3-8007-6318-4
publisher: VDE
status: public
title: ElectricGrid.jl – Automated modeling of decentralized electrical energy grids
type: conference
user_id: '47427'
year: '2023'
...
---
_id: '46784'
article_number: '5616'
author:
- first_name: Oliver
full_name: Wallscheid, Oliver
id: '11291'
last_name: Wallscheid
orcid: https://orcid.org/0000-0001-9362-8777
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
- first_name: Jan
full_name: Stenner, Jan
id: '65520'
last_name: Stenner
- first_name: Daniel
full_name: Weber, Daniel
id: '24041'
last_name: Weber
orcid: 0000-0003-3367-5998
- first_name: Septimus
full_name: Boshoff, Septimus
last_name: Boshoff
- first_name: Marvin
full_name: Meyer, Marvin
last_name: Meyer
- first_name: Vikas
full_name: Chidananda, Vikas
last_name: Chidananda
- first_name: Oliver
full_name: Schweins, Oliver
last_name: Schweins
citation:
ama: Wallscheid O, Peitz S, Stenner J, et al. ElectricGrid.jl - A Julia-based modeling
and simulationtool for power electronics-driven electric energy grids. Journal
of Open Source Software. 2023;8(89). doi:10.21105/joss.05616
apa: Wallscheid, O., Peitz, S., Stenner, J., Weber, D., Boshoff, S., Meyer, M.,
Chidananda, V., & Schweins, O. (2023). ElectricGrid.jl - A Julia-based modeling
and simulationtool for power electronics-driven electric energy grids. Journal
of Open Source Software, 8(89), Article 5616. https://doi.org/10.21105/joss.05616
bibtex: '@article{Wallscheid_Peitz_Stenner_Weber_Boshoff_Meyer_Chidananda_Schweins_2023,
title={ElectricGrid.jl - A Julia-based modeling and simulationtool for power electronics-driven
electric energy grids}, volume={8}, DOI={10.21105/joss.05616},
number={895616}, journal={Journal of Open Source Software}, publisher={The Open
Journal}, author={Wallscheid, Oliver and Peitz, Sebastian and Stenner, Jan and
Weber, Daniel and Boshoff, Septimus and Meyer, Marvin and Chidananda, Vikas and
Schweins, Oliver}, year={2023} }'
chicago: Wallscheid, Oliver, Sebastian Peitz, Jan Stenner, Daniel Weber, Septimus
Boshoff, Marvin Meyer, Vikas Chidananda, and Oliver Schweins. “ElectricGrid.Jl
- A Julia-Based Modeling and Simulationtool for Power Electronics-Driven Electric
Energy Grids.” Journal of Open Source Software 8, no. 89 (2023). https://doi.org/10.21105/joss.05616.
ieee: 'O. Wallscheid et al., “ElectricGrid.jl - A Julia-based modeling and
simulationtool for power electronics-driven electric energy grids,” Journal
of Open Source Software, vol. 8, no. 89, Art. no. 5616, 2023, doi: 10.21105/joss.05616.'
mla: Wallscheid, Oliver, et al. “ElectricGrid.Jl - A Julia-Based Modeling and Simulationtool
for Power Electronics-Driven Electric Energy Grids.” Journal of Open Source
Software, vol. 8, no. 89, 5616, The Open Journal, 2023, doi:10.21105/joss.05616.
short: O. Wallscheid, S. Peitz, J. Stenner, D. Weber, S. Boshoff, M. Meyer, V. Chidananda,
O. Schweins, Journal of Open Source Software 8 (2023).
date_created: 2023-09-04T11:05:03Z
date_updated: 2023-09-04T11:06:06Z
department:
- _id: '655'
- _id: '52'
doi: 10.21105/joss.05616
intvolume: ' 8'
issue: '89'
keyword:
- General Earth and Planetary Sciences
- General Environmental Science
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://joss.theoj.org/papers/10.21105/joss.05616
oa: '1'
publication: Journal of Open Source Software
publication_identifier:
issn:
- 2475-9066
publication_status: published
publisher: The Open Journal
status: public
title: ElectricGrid.jl - A Julia-based modeling and simulationtool for power electronics-driven
electric energy grids
type: journal_article
user_id: '47427'
volume: 8
year: '2023'
...