going from Bookdown (rethinkpriorities package or Reinstein's bookdowns) to Quarto books

bookdown_to_quarto.md

Bookdown to Quarto -- steps and considerations

#rstats I think I'll switch to Quarto. It seems it can do all Bookdown can do & more, with better support.

- Anyone know tools/templates for converting projects from Rmd/Bookdown to Quarto?
- Am I losing something from making the switch?

@xieyihui

— 𝙳𝚊𝚟𝚒𝚍 𝚁𝚎𝚒𝚗𝚜𝚝𝚎𝚒𝚗 (@GivingTools) March 30, 2022

At Rethink we set up this template and tools.

For much of my work I was using the Bookdown style and template I worked on here

But I think I am (and we are) moving to Quarto. ... Quarto pages and Quarto books, or whatever they are callled.

Setting up my preferred style and options here - work in progress; repo

Update: the main thing Quarto can't do is 'persist content in the environment between chapters'

Setting up Quarto on your machine and Rstudio

For a Quarto book, to enable building/rendering in Rstudio by button click, you may need to change the settings ... ... Configure build tools, project build tools, 'none'

This seems to switch the build options to include 'preview book' and 'render book'

Adjusting background files, file naming, folder structure,

_quarto.yml essentially replaces content from the bookdown's

• _bookdown.yml ('what content'... now goes in the chapters list) and
• setup options from _output.yml (or from the yaml at the top of index.Rmd

Setting up _quarto.yml

Adjusting 'main.R' to actually build the thing (with code)

Quarto seems to render with its own engine, not something in R (?). Thus, if you want to run it in an R script you need to invoke a system command, I guess, such as

system("quarto render")

Note there are a variety of options, depending on what Quarto thing you want rendered, and how.

But the above command operates outside of Rstudio. It doesn't seem to use any content within the environment you set up previously in main.R. A quick trial of the 'Build', 'Render book' clickable in Rstudio seems to yield the same.

*Thus it seems that all (build, import, define, load package) operations in the general environment need to be made part of each .qmd file that is called, or sourced from an external script in that file.

Otherwise there is a not-great workaround:

save.image(file = "../workspacename.RData") at the bottom of the .qmd files

and load(file = "../workspacename.RData") at the top, plus reloading necessary packages.

.Rmd to .qmd files, syntax adjustments

I think Quarto can and will build Rmd files without an extension change, as part of a book etc. However, renaming it as blahblah.qmd will probably enable additional functionality.

Standard Rmd syntax should also typically work in Quarto.

However...

New convenient syntaxes and nice new features in Quarto

Some syntaxes will be easier to use and worth switching over for readability

Other features are new and great!

Our 'hand-built' content will need adapting

E.g., from the rethinkpriorities package template...

Foldables

::: {.foldable}
Content here

:::

Probably should be replaced with a variant of a collapsable callout (note, warning, etc.)

:::{.callout-note collapse="true"}
## The callout header goes here

This is an example of a 'folded' note callout that can be expanded by the user. You can use `collapse="true"` to collapse it by default or `collapse="false"` to make a collapsible callout that is expanded by default.
:::

Note that inline code doesn't seem to work within a callout.

Hover footnotes

This function should just 'work' if you set the right _quarto.yml option:

format:
  html:
    citations-hover: true
    footnotes-hover: true

Here I told citations and footnotes to hover (and they also go at the bottom... nice!)

Github hosting stuff

Tools and R script for converting between formats

I wrote this parsing function to adjust content in Rmd from the RP/Bookdown/Sleeger format to Quarto. Will continue to update/adjust it. Note that to go from Reinstein bookdown format you may need to first apply the parse_dr_to_ws.R script to first make it bs4.

Some sample code applying this below, applying the parsing scripts in succession, as well as taking some other steps to change extensions etc.

1. List all Rmd and md files that might need adjusting (here, taken from _bookdown.yml ... could try to automate later)

rmd_files <- c("outline_work.Rmd", "present_puzzle.Rmd",  "substitution.Rmd",  "barriers_breakdown.Rmd", "BARRIERS_FUNDAMENTAL.md", "doesimpactmatter.Rmd", "aware-distance.Rmd", "identity.Rmd", "social.Rmd", "BARRIERS_INFO.md", "eval-aversion.Rmd", "impact_analytical.Rmd",  "BARRIERS_JUDGEMENT.md",  "quant-biases.Rmd", "factual.Rmd", "PATH_FORWARD.md", "tools.Rmd", "conclusion-agenda.Rmd", "appendix_tutorial.Rmd", "inertia.Rmd", "references.Rmd")

Grab relevant parsing functions. Here we suppose we first need to go from Reinstein to (Sleegers/Rethink) bs4 bookdown style, applying the parse_dr_to_ws.R ... if not, skip that step.

p_load(rex)
source_url("https://raw.githubusercontent.com/daaronr/dr-rstuff/master/functions/parse_dr_to_ws.R") 
source_url("https://raw.githubusercontent.com/daaronr/dr-rstuff/master/functions/parse_rp_bookdown_to_quarto.R")

Now parsing the Reinstein bookdown Rmds to Willem's bs4 style (skip if already as bs4)

map2(rmd_files, rmd_files,
  ~ dr_to_bs4(here::here("sections", .x), here::here("rmd_rp_style", .y)))

dr_to_bs4("index.Rmd", here::here("index_rp.Rmd"))

Now we move to the Quarto format, parsing and changing names and folder names

# apply all parsing commands and put it into 'chapters' folder
system("mkdir chapters")
map2(rmd_files, rmd_files,
  ~ rp_rmd_to_quarto(here::here("rmd_rp_style", .y), here::here("chapters", .y)))

newName <- sub(".Rmd", ".qmd", here::here("chapters", rmd_files))
file.rename(here::here("chapters", rmd_files), newName)

rp_rmd_to_quarto("index_rp.Rmd", "index.qmd")

##TODO:
# delete remaining .Rmd files to avoid clutter
# index.qmd file needs will need adjusting,
# you need to create a _quarto.yml file

Print a list of .qml files to put in the chapter listing of the _quarto.yml file:

#output to put into chapters list in _quarto.ylm
cat(
  "- index.qmd",
paste("\n- chapters/", str_replace_all(
  rmd_files, ".Rmd", ".qmd"),
  sep="")
)

Things I'm missing/haven't found yet in Quarto

1. Persisting content in the environment between chapters (discussed above)

A possible workaround: could Quarto allow a 'new chapter' (new html page) to be build from the same .qmd file?

Things I did

Solution found, thanks JJ Allaire to

Duplicate chunk forgiveness (the equivalent of options(knitr.duplicate.label = "allow") )

Solution:
Put execute key at top level in _quarto.yml file

execute:
...    
    error: true

error: true will flag errors but continue building

Thanks for sharing! This is really interesting to get feedback like this.

If I may, a few comments below to give some precision and some potential answers

Hope this will help understand better. Thank you very much for sharing this!

---

Quarto seems to render with its own engine, not something in R (?). s, if you want to run it in an R script you need to invoke a system command, I guess,

Yes Quarto does not use R except for computation engines with knitr. However, there is a quarto R package to help calling Quarto from R code when you are not using Terminal or IDE integration. https://cran.r-project.org/web/packages/quarto/index.html
It will call the quarto command for you and allow to create the command line using R arguments in functions. If this can help...

It doesn't seem to use any content within the environment you set up previously in main.R

This will all depend on what you are doing in this main file. Let's discuss it below.

*Thus it seems that all (build, import, define, load package) operations in the general environment need to be made part of each .qmd file that is called, or sourced from an external scriptin that file.

Persisting content in the environment between chapters (discussed above)

This is indeed one of the main difference with bookdown for users that where leverage the Merge-Knit approach.: Everything was happening in the same R session. The Knit and Merge approach is however similar to Quarto (see about approaches in https://bookdown.org/yihui/bookdown/new-session.html) . Usually in the later case, you use a _common.R or _common.Rmd file that you can load in a setup chunk in each the Rmd documents:

```{r, setup, file="_common.R", include = FALSE}
```

There is also the .Rprofile (user or project) that is loaded each times R start that can help setting environments for all R executions.

Both of these should be possible to used with Quarto Book using knitr computation engine.

Tools for caching and loading cache can probably help also with that, like storrr package or R.cache. Also maybe tools like targets would help - I know there are interested to support Quarto document and maybe this would solve such issues.

Quarto pre-render script can also be a solution for some process to do before a rendering.

Duplicate chunk forgiveness (the equivalent of options(knitr.duplicate.label = "allow") )

This option should work but you need to either set it in a project .Rprofile for example, or in each document using a setup chunk.

Thanks @cderv

Yes Quarto does not use R except for computation engines with knitr. However, there is a quarto R package to help calling Quarto from R code when you are not using Terminal or IDE integration. https://cran.r-project.org/web/packages/quarto/index.html
It will call the quarto command for you and allow to create the command line using R arguments in functions. If this can help...

This seems useful; I will try it. It seems better than system("quarto render") ... because everyone's 'system' may be different

Persisting content ...

This is indeed one of the main difference with bookdown for users that where leverage the Merge-Knit approach.: Everything was happening in the same R session. The Knit and Merge approach is however similar to Quarto (see about approaches in https://bookdown.org/yihui/bookdown/new-session.html)

Right, I was always a 'Merge and Knit' guy I guess

Usually in the latter case, you use a _common.R or _common.Rmd file that you can load in a setup chunk in each of the Rmd documents:

But what did this do? I guess the _common.R(md) file is in the root folder and it can have any setup and formatting .options, objects in the environment, loading of packages, etc. But does it need to re-run again for each Rmd or does it somehow 'just persist' (which would save loads of processing time).

Is there a nice template/vignette of this (ideally for quarto)?

There is also the .Rprofile (user or project) that is loaded each times R start that can help setting environments for all R executions.

This seems like something distinct. Would it help me 'preserve the environment across qmd files that are being merged and then knitted (or knitted then merged'?

Both of these should be possible to used with Quarto Book using knitr computation engine.

Thanks. If you have a vignette/template/example, that would be very helpful to me, and perhaps others.

I really appreciate your quick engagement!

But what did this do? I guess the _common.R(md) file is in the root folder and it can have any setup and formatting .options, objects in the environment, loading of packages, etc. But does it need to re-run again for each Rmd or does it somehow 'just persist' (which would save loads of processing time).

The syntax with the file= knitr option means "Read the file content as the chunk content". This is a way to have common setup content for different files. However, it will not persist. Each document is run in its own session, so all chunks into one document are run in one session, but other document don't have access to that. There is no persistence of the environment as built-in feature for now. Maybe this is something this could be discuss to have as an opt-in behavior within Quarto (like saving a R session then reloading the R session for next document rendering).
Currently you would need to do that by adding some chunks for that in your document. No example around though...

Is there a nice template/vignette of this (ideally for quarto)?

Doc for this file= option is there https://bookdown.org/yihui/rmarkdown-cookbook/option-code.html but no specific context of project aournd it.

This seems like something distinct. Would it help me 'preserve the environment across qmd files that are being merged and then knitted (or knitted then merged'?

No Quarto does not have a merged and knitting approach like bookdown. Everything document computation is evaluated on its own session. So data and R config does not persist as this is like executing code is different R Session. That is why I mentioned .Rprofile as this is a the usual way with R to have some content loaded (usually options) each time R is start up in a project.
This reminds me that possible the package startup can help gets things more configurable if need.

Thanks. If you have a vignette/template/example, that would be very helpful to me, and perhaps others.

We don't have specific documentation around that, but I'll keep that in mind.

If you want to open a discussion regarding persistence with the Quarto team, you can post on the discussion board.
There is already this issue quarto-dev/quarto-cli#134 and this discussion quarto-dev/quarto-cli#431 that could be of interest