---
_id: '57892'
abstract:
- lang: eng
  text: 'The present paper discusses the extent to which Large Language Models (LLMs)
    may affect the scientific enterprise, reinforcing or mitigating existing structural
    inequalities expressed by the Matthew Effect and introducing a “bot delusion”
    in academia. In a theory-led thought experiment, we first focus on the academic
    publication and citation system and develop three scenarios of the anticipated
    consequences of using LLMs: reproducing content and status quo (Scenario 1), enabling
    content coherence evaluation (Scenario 2) and content evaluation (Scenario 3).
    Second, we discuss the interaction between the use of LLMs and academic (counter)norms
    for citation selection and their impact on the publication and citation system.
    Finally, we introduce communal counter-norms to capture academics’ loyal citation
    behavior and develop three future scenarios that academia may face when LLMs are
    widely used in the research process, namely status quo future of science, mixed-access
    future, and open science future.'
article_number: '103537'
article_type: original
author:
- first_name: Oliver
  full_name: Wieczorek, Oliver
  last_name: Wieczorek
- first_name: Isabel
  full_name: Steinhardt, Isabel
  id: '90339'
  last_name: Steinhardt
  orcid: https://orcid.org/0000-0002-2590-6189
- first_name: Rebecca
  full_name: Schmidt, Rebecca
  id: '94416'
  last_name: Schmidt
  orcid: https://orcid.org/0000-0002-2516-359X
- first_name: Sylvi
  full_name: Mauermeister, Sylvi
  id: '98032'
  last_name: Mauermeister
- first_name: Christian
  full_name: Schneijderberg, Christian
  last_name: Schneijderberg
citation:
  ama: Wieczorek O, Steinhardt I, Schmidt R, Mauermeister S, Schneijderberg C. The
    Bot Delusion. Large language models and anticipated consequences for academics’
    publication and citation behavior. <i>Futures</i>. 2024;166. doi:<a href="https://doi.org/10.1016/j.futures.2024.103537">10.1016/j.futures.2024.103537</a>
  apa: Wieczorek, O., Steinhardt, I., Schmidt, R., Mauermeister, S., &#38; Schneijderberg,
    C. (2024). The Bot Delusion. Large language models and anticipated consequences
    for academics’ publication and citation behavior. <i>Futures</i>, <i>166</i>,
    Article 103537. <a href="https://doi.org/10.1016/j.futures.2024.103537">https://doi.org/10.1016/j.futures.2024.103537</a>
  bibtex: '@article{Wieczorek_Steinhardt_Schmidt_Mauermeister_Schneijderberg_2024,
    title={The Bot Delusion. Large language models and anticipated consequences for
    academics’ publication and citation behavior}, volume={166}, DOI={<a href="https://doi.org/10.1016/j.futures.2024.103537">10.1016/j.futures.2024.103537</a>},
    number={103537}, journal={Futures}, publisher={Elsevier BV}, author={Wieczorek,
    Oliver and Steinhardt, Isabel and Schmidt, Rebecca and Mauermeister, Sylvi and
    Schneijderberg, Christian}, year={2024} }'
  chicago: Wieczorek, Oliver, Isabel Steinhardt, Rebecca Schmidt, Sylvi Mauermeister,
    and Christian Schneijderberg. “The Bot Delusion. Large Language Models and Anticipated
    Consequences for Academics’ Publication and Citation Behavior.” <i>Futures</i>
    166 (2024). <a href="https://doi.org/10.1016/j.futures.2024.103537">https://doi.org/10.1016/j.futures.2024.103537</a>.
  ieee: 'O. Wieczorek, I. Steinhardt, R. Schmidt, S. Mauermeister, and C. Schneijderberg,
    “The Bot Delusion. Large language models and anticipated consequences for academics’
    publication and citation behavior,” <i>Futures</i>, vol. 166, Art. no. 103537,
    2024, doi: <a href="https://doi.org/10.1016/j.futures.2024.103537">10.1016/j.futures.2024.103537</a>.'
  mla: Wieczorek, Oliver, et al. “The Bot Delusion. Large Language Models and Anticipated
    Consequences for Academics’ Publication and Citation Behavior.” <i>Futures</i>,
    vol. 166, 103537, Elsevier BV, 2024, doi:<a href="https://doi.org/10.1016/j.futures.2024.103537">10.1016/j.futures.2024.103537</a>.
  short: O. Wieczorek, I. Steinhardt, R. Schmidt, S. Mauermeister, C. Schneijderberg,
    Futures 166 (2024).
date_created: 2024-12-31T08:30:51Z
date_updated: 2024-12-31T08:36:28Z
department:
- _id: '121'
doi: 10.1016/j.futures.2024.103537
intvolume: '       166'
keyword:
- Large Language Models
- Matthew Effect
- Academic Publishing and Citation Systems
- Scientific Norms
- Thought Experiment
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://www.sciencedirect.com/science/article/pii/S0016328724002209?via%3Dihub
oa: '1'
publication: Futures
publication_identifier:
  issn:
  - 0016-3287
publication_status: published
publisher: Elsevier BV
quality_controlled: '1'
status: public
title: The Bot Delusion. Large language models and anticipated consequences for academics’
  publication and citation behavior
type: journal_article
user_id: '90339'
volume: 166
year: '2024'
...
---
_id: '20212'
abstract:
- lang: eng
  text: "Ideational impact refers to the uptake of a paper's ideas and concepts by
    subsequent research. It is defined in stark contrast to total citation impact,
    a measure predominantly used in research evaluation that assumes that all citations
    are equal. Understanding ideational impact is critical for evaluating research
    impact and understanding how scientific disciplines build a cumulative tradition.
    Research has only recently developed automated citation classification techniques
    to distinguish between different types of citations and generally does not emphasize
    the conceptual content of the citations and its ideational impact. To address
    this problem, we develop Deep Content-enriched Ideational Impact Classification
    (Deep-CENIC) as the first automated approach for ideational impact classification
    to support researchers' literature search practices. We evaluate Deep-CENIC on
    1,256 papers citing 24 information systems review articles from the IT business
    value domain. We show that Deep-CENIC significantly outperforms state-of-the-art
    benchmark models. We contribute to information systems research by operationalizing
    the concept of ideational impact, designing a recommender system for academic
    papers based on deep learning techniques, and empirically exploring the ideational
    impact of the IT business value domain.\r\n"
article_number: '113432'
author:
- first_name: Julian
  full_name: Prester, Julian
  last_name: Prester
- first_name: Gerit
  full_name: Wagner, Gerit
  last_name: Wagner
- first_name: Guido
  full_name: Schryen, Guido
  id: '72850'
  last_name: Schryen
- first_name: Nik Rushdi
  full_name: Hassan, Nik Rushdi
  last_name: Hassan
citation:
  ama: 'Prester J, Wagner G, Schryen G, Hassan NR. Classifying the Ideational Impact
    of Information Systems Review Articles: A Content-Enriched Deep Learning Approach.
    <i>Decision Support Systems</i>. 2021;140(January).'
  apa: 'Prester, J., Wagner, G., Schryen, G., &#38; Hassan, N. R. (2021). Classifying
    the Ideational Impact of Information Systems Review Articles: A Content-Enriched
    Deep Learning Approach. <i>Decision Support Systems</i>, <i>140</i>(January),
    Article 113432.'
  bibtex: '@article{Prester_Wagner_Schryen_Hassan_2021, title={Classifying the Ideational
    Impact of Information Systems Review Articles: A Content-Enriched Deep Learning
    Approach}, volume={140}, number={January113432}, journal={Decision Support Systems},
    author={Prester, Julian and Wagner, Gerit and Schryen, Guido and Hassan, Nik Rushdi},
    year={2021} }'
  chicago: 'Prester, Julian, Gerit Wagner, Guido Schryen, and Nik Rushdi Hassan. “Classifying
    the Ideational Impact of Information Systems Review Articles: A Content-Enriched
    Deep Learning Approach.” <i>Decision Support Systems</i> 140, no. January (2021).'
  ieee: 'J. Prester, G. Wagner, G. Schryen, and N. R. Hassan, “Classifying the Ideational
    Impact of Information Systems Review Articles: A Content-Enriched Deep Learning
    Approach,” <i>Decision Support Systems</i>, vol. 140, no. January, Art. no. 113432,
    2021.'
  mla: 'Prester, Julian, et al. “Classifying the Ideational Impact of Information
    Systems Review Articles: A Content-Enriched Deep Learning Approach.” <i>Decision
    Support Systems</i>, vol. 140, no. January, 113432, 2021.'
  short: J. Prester, G. Wagner, G. Schryen, N.R. Hassan, Decision Support Systems
    140 (2021).
date_created: 2020-10-27T13:28:21Z
date_updated: 2022-06-10T06:55:32Z
ddc:
- '000'
department:
- _id: '277'
file:
- access_level: open_access
  content_type: application/pdf
  creator: hsiemes
  date_created: 2020-10-27T13:31:01Z
  date_updated: 2020-10-27T13:31:01Z
  file_id: '20213'
  file_name: DECSUP-D-20-00312 - PREPUBLICATION.pdf
  file_size: 440903
  relation: main_file
file_date_updated: 2020-10-27T13:31:01Z
has_accepted_license: '1'
intvolume: '       140'
issue: January
keyword:
- Ideational impact
- citation classification
- academic recommender systems
- natural language processing
- deep learning
- cumulative tradition
language:
- iso: eng
oa: '1'
publication: Decision Support Systems
status: public
title: 'Classifying the Ideational Impact of Information Systems Review Articles:
  A Content-Enriched Deep Learning Approach'
type: journal_article
user_id: '72850'
volume: 140
year: '2021'
...
---
_id: '20844'
abstract:
- lang: eng
  text: Review papers are essential for knowledge development in IS. While some are
    cited twice a day, others accumulate single digit citations over a decade. The
    magnitude of these differences prompts us to analyze what distinguishes those
    reviews that have proven to be integral to scientific progress from those that
    might be considered less impactful. Our results highlight differences between
    reviews aimed at describing, understanding, explaining, and theory testing. Beyond
    the control variables, they demonstrate the importance of methodological transparency
    and the development of research agendas. These insights inform all stakeholders
    involved in the development and publication of review papers.
article_number: '103427'
author:
- first_name: Gerit
  full_name: Wagner, Gerit
  last_name: Wagner
- first_name: Julian
  full_name: Prester, Julian
  last_name: Prester
- first_name: Maria
  full_name: Roche, Maria
  last_name: Roche
- first_name: Guido
  full_name: Schryen, Guido
  id: '72850'
  last_name: Schryen
- first_name: Alexander
  full_name: Benlian, Alexander
  last_name: Benlian
- first_name: Guy
  full_name: Paré, Guy
  last_name: Paré
- first_name: Mathieu
  full_name: Templier, Mathieu
  last_name: Templier
citation:
  ama: Wagner G, Prester J, Roche M, et al. Which Factors Affect the Scientific Impact
    of Review Papers in IS Research? A Scientometric Study. <i>Information &#38; Management</i>.
    2021;58(3).
  apa: Wagner, G., Prester, J., Roche, M., Schryen, G., Benlian, A., Paré, G., &#38;
    Templier, M. (2021). Which Factors Affect the Scientific Impact of Review Papers
    in IS Research? A Scientometric Study. <i>Information &#38; Management</i>, <i>58</i>(3),
    Article 103427.
  bibtex: '@article{Wagner_Prester_Roche_Schryen_Benlian_Paré_Templier_2021, title={Which
    Factors Affect the Scientific Impact of Review Papers in IS Research? A Scientometric
    Study}, volume={58}, number={3103427}, journal={Information &#38; Management},
    author={Wagner, Gerit and Prester, Julian and Roche, Maria and Schryen, Guido
    and Benlian, Alexander and Paré, Guy and Templier, Mathieu}, year={2021} }'
  chicago: Wagner, Gerit, Julian Prester, Maria Roche, Guido Schryen, Alexander Benlian,
    Guy Paré, and Mathieu Templier. “Which Factors Affect the Scientific Impact of
    Review Papers in IS Research? A Scientometric Study.” <i>Information &#38; Management</i>
    58, no. 3 (2021).
  ieee: G. Wagner <i>et al.</i>, “Which Factors Affect the Scientific Impact of Review
    Papers in IS Research? A Scientometric Study,” <i>Information &#38; Management</i>,
    vol. 58, no. 3, Art. no. 103427, 2021.
  mla: Wagner, Gerit, et al. “Which Factors Affect the Scientific Impact of Review
    Papers in IS Research? A Scientometric Study.” <i>Information &#38; Management</i>,
    vol. 58, no. 3, 103427, 2021.
  short: G. Wagner, J. Prester, M. Roche, G. Schryen, A. Benlian, G. Paré, M. Templier,
    Information &#38; Management 58 (2021).
date_created: 2020-12-29T11:52:03Z
date_updated: 2022-06-10T06:53:12Z
ddc:
- '000'
department:
- _id: '277'
file:
- access_level: open_access
  content_type: application/pdf
  creator: hsiemes
  date_created: 2020-12-29T11:49:57Z
  date_updated: 2020-12-29T11:49:57Z
  file_id: '20845'
  file_name: INFMAN_Scientific Impact of Review Papers - Prepublication version.pdf
  file_size: 1584950
  relation: main_file
file_date_updated: 2020-12-29T11:49:57Z
has_accepted_license: '1'
intvolume: '        58'
issue: '3'
keyword:
- Literature review
- review papers
- scientometric
- scientific impact
- citation analysis
language:
- iso: eng
oa: '1'
publication: Information & Management
status: public
title: Which Factors Affect the Scientific Impact of Review Papers in IS Research?
  A Scientometric Study
type: journal_article
user_id: '72850'
volume: 58
year: '2021'
...
---
_id: '33372'
abstract:
- lang: eng
  text: '<jats:p>Academics may actively respond to the expectations of the academic
    status market, which have largely been shaped by the World University Rankings
    (WURs). This study empirically examines how academics’ citation patterns have
    changed in response to the rise of an “evaluation environment” in academia. We
    regard the WURs to be a macro-level trigger for cementing a bibliometric-based
    evaluation environment in academia. Our analyses of citation patterns in papers
    published in two higher education journals explicitly considered three distinct
    periods: the pre-WURs (1990–2003), the period of WURs implementation (2004–2010),
    and the period of adaption to WURs (2011–2017). We applied the nonparametric Kaplan–Meier
    method to compare first-citation speeds of papers published across the three periods.
    We found that not only has first-citation speed become faster, but first-citation
    probability has also increased following the emergence of the WURs. Applying Cox
    proportional hazard models to first-citation probabilities, we identified journal
    impact factors and third-party funding as factors influencing first-citation probability,
    while other author- and paper-related factors showed limited effects. We also
    found that the general effects of different factors on first-citation speeds have
    changed with the emergence of the WURs. The findings expand our understanding
    of the citation patterns of academics in the rise of WURs and provide practical
    grounds for research policy as well as higher education policy.</jats:p>'
article_number: '9515'
article_type: original
author:
- first_name: Soo Jeung
  full_name: Lee, Soo Jeung
  last_name: Lee
- first_name: Christian
  full_name: Schneijderberg, Christian
  last_name: Schneijderberg
- first_name: Yangson
  full_name: Kim, Yangson
  last_name: Kim
- first_name: Isabel
  full_name: Steinhardt, Isabel
  id: '90339'
  last_name: Steinhardt
  orcid: https://orcid.org/0000-0002-2590-6189
citation:
  ama: Lee SJ, Schneijderberg C, Kim Y, Steinhardt I. Have Academics’ Citation Patterns
    Changed in Response to the Rise of World University Rankings? A Test Using First-Citation
    Speeds. <i>Sustainability</i>. 2021;13(17). doi:<a href="https://doi.org/10.3390/su13179515">10.3390/su13179515</a>
  apa: Lee, S. J., Schneijderberg, C., Kim, Y., &#38; Steinhardt, I. (2021). Have
    Academics’ Citation Patterns Changed in Response to the Rise of World University
    Rankings? A Test Using First-Citation Speeds. <i>Sustainability</i>, <i>13</i>(17),
    Article 9515. <a href="https://doi.org/10.3390/su13179515">https://doi.org/10.3390/su13179515</a>
  bibtex: '@article{Lee_Schneijderberg_Kim_Steinhardt_2021, title={Have Academics’
    Citation Patterns Changed in Response to the Rise of World University Rankings?
    A Test Using First-Citation Speeds}, volume={13}, DOI={<a href="https://doi.org/10.3390/su13179515">10.3390/su13179515</a>},
    number={179515}, journal={Sustainability}, publisher={MDPI AG}, author={Lee, Soo
    Jeung and Schneijderberg, Christian and Kim, Yangson and Steinhardt, Isabel},
    year={2021} }'
  chicago: Lee, Soo Jeung, Christian Schneijderberg, Yangson Kim, and Isabel Steinhardt.
    “Have Academics’ Citation Patterns Changed in Response to the Rise of World University
    Rankings? A Test Using First-Citation Speeds.” <i>Sustainability</i> 13, no. 17
    (2021). <a href="https://doi.org/10.3390/su13179515">https://doi.org/10.3390/su13179515</a>.
  ieee: 'S. J. Lee, C. Schneijderberg, Y. Kim, and I. Steinhardt, “Have Academics’
    Citation Patterns Changed in Response to the Rise of World University Rankings?
    A Test Using First-Citation Speeds,” <i>Sustainability</i>, vol. 13, no. 17, Art.
    no. 9515, 2021, doi: <a href="https://doi.org/10.3390/su13179515">10.3390/su13179515</a>.'
  mla: Lee, Soo Jeung, et al. “Have Academics’ Citation Patterns Changed in Response
    to the Rise of World University Rankings? A Test Using First-Citation Speeds.”
    <i>Sustainability</i>, vol. 13, no. 17, 9515, MDPI AG, 2021, doi:<a href="https://doi.org/10.3390/su13179515">10.3390/su13179515</a>.
  short: S.J. Lee, C. Schneijderberg, Y. Kim, I. Steinhardt, Sustainability 13 (2021).
date_created: 2022-09-15T08:27:56Z
date_updated: 2022-09-15T08:37:41Z
ddc:
- '300'
department:
- _id: '121'
doi: 10.3390/su13179515
extern: '1'
file:
- access_level: closed
  content_type: application/pdf
  creator: isste
  date_created: 2022-09-15T08:36:50Z
  date_updated: 2022-09-15T08:36:50Z
  file_id: '33375'
  file_name: Lee et al. 2021 Academic Citations.pdf
  file_size: 2407463
  relation: main_file
  success: 1
file_date_updated: 2022-09-15T08:36:50Z
has_accepted_license: '1'
intvolume: '        13'
issue: '17'
keyword:
- world university rankings
- citation
- first-citation speed
- Minerva
- Studies in Higher Education
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://www.mdpi.com/2071-1050/13/17/9515
oa: '1'
publication: Sustainability
publication_identifier:
  issn:
  - 2071-1050
publication_status: published
publisher: MDPI AG
quality_controlled: '1'
status: public
title: Have Academics’ Citation Patterns Changed in Response to the Rise of World
  University Rankings? A Test Using First-Citation Speeds
type: journal_article
user_id: '90339'
volume: 13
year: '2021'
...
---
_id: '17019'
abstract:
- lang: eng
  text: The scientific impact of research papers is multi-dimensional and can be determined
    quantitatively by means of citation analysis and qualitatively by means of content
    analysis. Accounting for the widely acknowledged limitations of pure citation
    analysis, we adopt a knowledge-based perspective on scientific impact to develop
    a methodology for content-based citation analysis which allows determining how
    papers have enabled knowledge development in subsequent research (knowledge impact).
    As knowledge development differs between research genres, we develop a new knowledgebased
    citation analysis methodology for the genre of standalone literature reviews (LRs).
    We apply the suggested methodology to the IS business value domain by manually
    coding 22 LRs and 1,228 citing papers (CPs) and show that the results challenge
    the assumption that citations indicate knowledge impact. We derive implications
    for distinguishing knowledge impact from citation impact in the LR genre. Finally,
    we develop recommendations for authors of LRs, scientific evaluation committees
    and editorial boards of journals how to apply and benefit from the suggested methodology,
    and we discuss its efficiency and automatization.
author:
- first_name: Guido
  full_name: Schryen, Guido
  id: '72850'
  last_name: Schryen
- first_name: Gerit
  full_name: Wagner, Gerit
  last_name: Wagner
- first_name: Alexander
  full_name: Benlian, Alexander
  last_name: Benlian
citation:
  ama: 'Schryen G, Wagner G, Benlian A. <i>Distinguishing Knowledge Impact from Citation
    Impact: A Methodology for Analysing Knowledge Impact for the Literature Review
    Genre</i>.; 2020.'
  apa: 'Schryen, G., Wagner, G., &#38; Benlian, A. (2020). <i>Distinguishing Knowledge
    Impact from Citation Impact: A Methodology for Analysing Knowledge Impact for
    the Literature Review Genre</i>.'
  bibtex: '@book{Schryen_Wagner_Benlian_2020, title={Distinguishing Knowledge Impact
    from Citation Impact: A Methodology for Analysing Knowledge Impact for the Literature
    Review Genre}, author={Schryen, Guido and Wagner, Gerit and Benlian, Alexander},
    year={2020} }'
  chicago: 'Schryen, Guido, Gerit Wagner, and Alexander Benlian. <i>Distinguishing
    Knowledge Impact from Citation Impact: A Methodology for Analysing Knowledge Impact
    for the Literature Review Genre</i>, 2020.'
  ieee: 'G. Schryen, G. Wagner, and A. Benlian, <i>Distinguishing Knowledge Impact
    from Citation Impact: A Methodology for Analysing Knowledge Impact for the Literature
    Review Genre</i>. 2020.'
  mla: 'Schryen, Guido, et al. <i>Distinguishing Knowledge Impact from Citation Impact:
    A Methodology for Analysing Knowledge Impact for the Literature Review Genre</i>.
    2020.'
  short: 'G. Schryen, G. Wagner, A. Benlian, Distinguishing Knowledge Impact from
    Citation Impact: A Methodology for Analysing Knowledge Impact for the Literature
    Review Genre, 2020.'
date_created: 2020-05-19T15:12:33Z
date_updated: 2022-01-06T06:53:02Z
ddc:
- '000'
department:
- _id: '277'
file:
- access_level: open_access
  content_type: application/pdf
  creator: hsiemes
  date_created: 2020-05-19T15:09:28Z
  date_updated: 2020-05-19T15:09:28Z
  file_id: '17020'
  file_name: SSRN-id3581789.pdf
  file_size: 487351
  relation: main_file
file_date_updated: 2020-05-19T15:09:28Z
has_accepted_license: '1'
keyword:
- Scientific impact
- knowledge impact
- content-based citation analysis
- methodology
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://ssrn.com/abstract=3581789
oa: '1'
status: public
title: 'Distinguishing Knowledge Impact from Citation Impact: A Methodology for Analysing
  Knowledge Impact for the Literature Review Genre'
type: working_paper
user_id: '61579'
year: '2020'
...