--- title: "Detecting disclosure of generative-AI use" output: rmarkdown::html_vignette description: > How rt_ai_pmc() detects whether an article discloses the use (or non-use) of generative AI in preparing the manuscript, why it is gated to 2023 onward, and how to summarize it across a corpus. vignette: > %\VignetteIndexEntry{Detecting disclosure of generative-AI use} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", message = FALSE, warning = FALSE ) has_ggplot <- requireNamespace("ggplot2", quietly = TRUE) ``` ```{r setup} library(rtransparency) ``` ## What this indicator captures Since 2023, journals and publishers (ICMJE, COPE, Nature, Science, Elsevier and others) have asked authors to **disclose any use of generative AI** such as ChatGPT, GPT-4, Copilot, Gemini or other large language models when preparing a manuscript. `rt_ai_pmc()` detects whether an article carries such a disclosure. It is deliberately narrow. It targets AI used to **prepare the manuscript** (writing, editing, language, figures), and counts both directions of the disclosure: * a positive disclosure: *"the authors used ChatGPT to improve the readability of the manuscript"*; * a negative disclosure: *"no generative AI was used in the preparation of this work"*; * a dedicated section: a *"Declaration of generative AI"* heading. It does **not** count AI that is the article's research *method* (for example a paper that trains a deep-learning classifier, or one that studies ChatGPT as its subject). Those mention AI heavily but make no statement about using AI to write the paper, so they are not disclosures. A negative lookahead also keeps the tool sense of "large language model" from being read as something the authors edited. ## The 2023 year gate The practice did not exist before 2023, so evaluating it on older articles would be meaningless. `rt_ai_pmc()` therefore reads the publication `year` and: * returns `is_ai_pred = NA` for articles before 2023 (and reports the `year`); * returns `TRUE` or `FALSE` for 2023 onward. The bundled example article is from 2020, so it returns `NA`: ```{r} xml_path <- system.file( "extdata", "PMID32171256-PMC7071725.xml", package = "rtransparency" ) ai <- rt_ai_pmc(xml_path) c(year = ai$year, is_ai_pred = ai$is_ai_pred) ``` The three possible values are easy to act on: `TRUE` (a disclosure was found), `FALSE` (the article is from 2023 or later but carries no disclosure), and `NA` (the article predates the practice and was not assessed). `rt_summary()` drops the `NA`s, so a corpus prevalence is computed only over the articles where the indicator applies. ## What a disclosure says A disclosure can say that AI was used ("The authors used ChatGPT to improve the language of the manuscript") or that it was not ("No generative AI was used in the preparation of this work"); `is_ai_pred` counts both. Three further columns read the disclosure itself: `ai_used` (`TRUE` for stated use, `FALSE` for stated non-use, `NA` when there is no disclosure or it cannot be read), `ai_tools` (the tools named, such as `"ChatGPT; DeepL"`) and `ai_purpose` (language editing, translation, drafting, figures and images, code and analysis, or literature search). These use the same rules as the disclosure detector and have not been separately validated. ```{r} rt_ai(text = paste( "During the preparation of this work the authors used Claude 3.5 Sonnet", "and DeepL to translate and edit the text." ))[, c("is_ai_pred", "ai_used", "ai_tools", "ai_purpose")] ``` ## In the all-indicators output `rt_all_pmc()` includes the indicator, so a single pass over a corpus already carries `year` and `is_ai_pred` alongside the other nine indicators: ```{r} all_indicators <- rt_all_pmc(xml_path) all_indicators[, c("pmid", "year", "is_ai_pred")] ``` ## Across a corpus Because the indicator is so new, its corpus-level story is a **trend**: how fast disclosure is being adopted from 2023 onward. The bundled simulated corpus `rt_demo` includes an `is_ai_pred` column (`NA` before 2023) for illustration. ```{r} data(rt_demo) ai_by_year <- rt_summary(rt_demo, by = "year", indicators = "is_ai_pred") # Years before 2023 have no assessable articles (all NA), so no denominator; # keep only the years where the indicator applies. ai_by_year <- ai_by_year[ai_by_year$n_articles > 0, ] knitr::kable( ai_by_year[, c("year", "n_articles", "n_detected", "percent")], digits = 1, col.names = c("Year", "Assessed", "Disclosed", "%") ) ``` Only 2023 onward carries a denominator; earlier years have no assessable articles and are filtered out above. Plotted, the adoption curve is the point of the indicator (`rt_plot()` drops the empty earlier years automatically): ```{r, eval = has_ggplot, fig.width = 7, fig.height = 3.5, fig.alt = "Line chart of generative-AI-use disclosure prevalence by year from 2023"} library(ggplot2) rt_plot(rt_demo, type = "trend", year = "year", indicators = "is_ai_pred") + ggtitle("Disclosure of generative-AI use (simulated corpus)") ``` It also sits naturally next to the other indicators in a single prevalence chart; the AI bar simply reflects the 2023-onward subset: ```{r, eval = has_ggplot, fig.width = 7, fig.height = 3.5, fig.alt = "Bar chart of all transparency indicators including AI-use disclosure"} rt_plot(rt_demo) + ggtitle("Transparency indicators, including AI-use disclosure") ``` ## Notes on precision On a manual validation of 1,000 open-access PMC articles (almost all 2024-2026), the indicator flagged about 16% of articles, and inspected positives were genuine disclosures. The main thing it intentionally avoids is treating articles that *discuss* ChatGPT as their topic, or *use* AI as a research method, as disclosures: those are not statements about how the manuscript was written. As with every other indicator, `rt_ai_pmc()` returns the matched text (`ai_text`) so a positive call can always be inspected. For the other indicators and the corpus-summary tools used here, see the [Introduction](rtransparency.html) and [Summarizing transparency across a corpus](transparency-summary.html) articles.