Using {ellmer} to dynamically generate alt text in {shiny} apps | R bloggers

Alt text

First things first, if you’ve never heard of alt text or used it before, it’s a brief written description of an image that explains its context and purpose. It is used to improve accessibility by allowing screen readers to describe images or provide context if an image fails to load. For writing good alt texts, see this article by
Howardbut some good rules of thumb are:

• Keep it concise and relevant to the context of why the image is being used.
• The screen reader already says ‘Image of…’, so we don’t need to include this unless the style is important (drawing, cartoon, etc.).

Alt text within apps and dashboards

I don’t need to list the positives of interactive apps and dashboards, but one of the most important is interactivity and allows users to explore data in their own way. This is usually a good thing, but an often overlooked pitfall is that interactivity can overshadow accessibility. Whether it’s a beautiful widget that is difficult (or impossible) to use via the keyboard or interactive visualizations without meaningful alt text.

In this post we will look at a new approach to generating dynamic alt text for ggplot2 charts using {more}Posit’s new R package for querying large language models (LLM) from R. If you’re using Shiny for Python, then
chatlas will be interesting for you.

Why dynamic alt text needs care

Automatically generating alt text is attractive, but production glossy apps have limitations:

• Plots can be re-displayed regularly
• API calls may fail or have limited speed
• Accessibility should decrease gradually and not break the app
• A good implementation should be consistent, fault-tolerant and cheap to use.

Using {ellmer} in a shiny app

The first step is to set up a connection with your chosen LLM. I use Google Gemini Flash-2.5 because there is a generous free tier, but there are other models and providers available. In a Shiny app, this can be done outside the reactive context:

library(ellmer)
gemini <- chat_google_gemini()

## Using model = "gemini-2.5-flash".

Note: You must have a Google Gemini key saved in your .Renviron file as
GEMINI_API_KEYthis way the {ellmer} function can find it. More information about generating a Gemini API key can be found in the
Gemini Documents.

Then we have the function to generate the alt text:

library(ggplot2)

generate_alt_text = function(ggplot_obj, model) {
 temp <- tempfile(fileext = ".png")
 on.exit(unlink(temp))

 ggsave(
 temp,
 ggplot_obj,
 width = 6,
 height = 4,
 dpi = 150
 )

 tryCatch(
 model$chat(
 "
Generate concise alt text for this plot image.
Describe the chart type, variables shown,
key patterns or trends, and value ranges where visible.
 ",
 content_image_file(temp)
 ),
 error = function(e) {
 "Data visualisation showing trends and comparisons."
 }
 )
}

The function has a number of features that keep the output more reliable:

• Consistent image size and resolution – helps model reliability when reading axes and labels.
• Explicit cleanup of temporary files – we don’t need to save the images once the text is generated.
• Error handling – if the model call fails, the app still returns usable alt text. We’ve kept our backup text simple for demonstration purposes, but you can try adding more detail.
• External model initialization – created and propagated only once, instead of being recreated with each reactive update.

Examples

This section will create just a few sample plots and then look at what the LLM generates.

simple_plot = ggplot(iris) +
 aes(Sepal.Width, Sepal.Length) +
 geom_point()
simple_plot

simple_plot_alt = generate_alt_text(simple_plot, gemini)
paste("Alt text generated by AI: ", simple_plot_alt)

Alt text generated by AI:

Scatter plot with Sepal.Length on the y-axis (ranging from approximately 4.5 to 8.0) versus Sepal.Width on the x-axis (ranging from approximately 2.0 to 4.5). The data points appear to form two separate clusters: one with Sepal.Width between 2.0 and 3.0 and Sepal.Length between 5.0 and 8.0, and another with Sepal.Width between 3.0 and 4.5 and Sepal.Length between 4.5 and 6.5.

plot = ggplot(iris) +
 aes(Sepal.Width, Sepal.Length, colour = Species) +
 geom_point()
plot

plot_alt =
 generate_alt_text(plot, gemini)
paste("Alt text generated by AI: ", plot_alt)

Alt text generated by AI:

Scatter plot showing Sepal.Length on the y-axis (range 4.5-8.0) versus Sepal.Width on the x-axis (range 2.0-4.5), with points colored by species. Red points, called “setosa”, form a clear cluster with higher Sepal.Width (3.0-4.5) and lower Sepal.Length (4.5-5.8). Blue points, “virginica”, generally have a higher Sepal.Length (5.5-8.0) and a moderate Sepal.Width (2.5-3.8). Green points, “versicolor”, are in between, with moderate Sepal.Length (5.0-7.0) and Sepal.Width (2.0-3.5), overlapping with virginica.

complicated_plot = ggplot(iris) +
 aes(Sepal.Width, Sepal.Length, colour = Species) +
 geom_point() +
 geom_smooth(method = "lm")
complicated_plot

complicated_plot_alt =
 generate_alt_text(complicated_plot, gemini)
paste("Alt text generated by AI: ", complicated_plot_alt)

Alt text generated by AI:

Scatter plot with Sepal.Length on the y-axis (range 4.0-8.0) versus Sepal.Width on the x-axis (range 2.0-4.5). Points and linear regression lines are colored by iris species. Red points, “setosa”, cluster with lower Sepal.Length (4.0-5.8) and higher Sepal.Width (2.8-4.4). Green points, “versicolor”, and blue points, “virginica”, largely overlap showing higher Sepal.Length (5.0-8.0) and moderate Sepal.Width (2.0-3.8), with “virginica” generally having the longest sepals. All three species show a positive linear correlation, indicated by their respective regression lines and shaded confidence intervals, with increasing sepal width corresponding to increasing sepal length.

As we can see, the alt text can be very good and informative when using LLMs. One alternative I would like to point out is to actually include a summary of the data behind the plot. This way, screen reader users can still gain insight into the plot.

Using dynamic alt text in Shiny

Once generated, the alt text can be delivered directly to the UI:

• Through the alt argument from plotOutput()
• Or injected into custom HTML for more complex layouts

Because the text is generated from the displayed plot, it stays in sync with user input and filters.

Other considerations

Some apps may be more complex and/or have a large number of users. These types of apps need a little more attention to include features like these:

• Cache alt text for unmodified plots to reduce API usage
• Quick addition with familiar names or units of variables
• Manual overrides for critical images

Conclusion

AI-generated alt text works best as a supporting tool and not as a replacement for accessibility assessment. I also found it helpful to let users know that the alt text is AI-generated, so they know to take it with a grain of salt.

Dynamic alt text is a small feature with a big impact on the recording. By combining Shiny’s responsiveness with consistent rendering, error handling, and modern LLMs, we can make interactive data apps more accessible by default without increasing the burden on developers.

For updates and revisions to this article, see the original post

Related