Environment variables do not reach quarto reliably when rendering

[GitHubGeniusOverlord]

Bug description

Setting environment variable (e.v.) from a python process. It is expected that the e.v. lives on in the subprocesses.
Calling quarto.render() from the python process:
The e.v. is not reliably transfered to the quarto process: It seems to be cached and not always therefore newest version of the e.v. is in use when rendering the quarto document.

Steps to reproduce

render_quarto.py

Here we set the environment variable and call the rendering of the quartofile.qmd

import os
import quarto
from pathlib import Path

env_var_1 = 'variable1'
# print the env_var to a textfile
with open("output.txt", "a") as text_file:
    print("The e.v. before quarto:", file=text_file)
    print(env_var_1, file=text_file)

# Set the environment variable
os.environ['env_var'] = env_var_1

# Render a quarto file in which the e.v. is called
quarto.render(Path(__file__).parent / "quartofile.qmd")

Then, we expect to see the e.v. being used in the quarto document, however, this does not happen reliably: Sometimes, it is an old version of the e.v.. This can be seen, by manipulation env_var_1 and giving it another content, like 'variable2' and then execute the python script again.

quartofile.qmd

import os
# print the env_var to a textfile
with open("output.txt", "a") as text_file:
    print("The e.v. during rendering quarto:", file=text_file)
    print(os.environ['env_var'], file=text_file)

Expected behavior

The environment variable should not be cached. Instead, every change should reliably reach the quarto file, as it is run as a subprocess of the starting python process.
After running four times, with the e.v. set to 'variable1' for the first run, then 'variable2' for the following ones:

Content of output.txt:

The e.v. during rendering quarto:
variable1
The e.v. during rendering quarto:
variable2
The e.v. during rendering quarto:
variable2
The e.v. during rendering quarto:
variable2

Actual behavior

The environment variable is at first set but then it seems like it is being cached and changes arenot reliably transfered. After waiting a while it seems like it is again being refreshed and changes again reach the quarto process.

Content of output.txt:

The e.v. during rendering quarto:
variable1
The e.v. during rendering quarto:
variable1
The e.v. during rendering quarto:
variable1
The e.v. during rendering quarto:
variable1

Your environment

quarto 1.4.553
IDE: Visual Studio Code
Python 3.11.1
System: Windows 11

Quarto check output

Quarto 1.4.553
[>] Checking versions of quarto binary dependencies...
Pandoc version 3.1.11: OK
Dart Sass version 1.69.5: OK
Deno version 1.37.2: OK
[>] Checking versions of quarto dependencies......OK
[>] Checking Quarto installation......OK
Version: 1.4.553
Path: C:\Users\seucker\AppData\Local\Programs\Quarto\bin
CodePage: 1252

[>] Checking tools....................OK
TinyTeX: v2024.01
Chromium: (not installed)

[>] Checking LaTeX....................OK
Using: TinyTex
Path: C:\Users\seucker\AppData\Roaming\TinyTeX\bin\windows
Version: 2023

[>] Checking basic markdown render....OK

[>] Checking Python 3 installation....OK
Version: 3.11.1
Path: c:/Users/seucker/Desktop/Projekte/MigrationAuswertung/.venv/Scripts/python.exe
Jupyter: 5.7.2
Kernels: python3

[>] Checking Jupyter engine render....OK

[>] Checking R installation...........(None)

  Unable to locate an installed version of R.
  Install R from https://cloud.r-project.org/

[remlapmot]

I believe the error here is actually in the Quarto CLI - so the issue should be transferred back to the quarto-cli repo.

For example, I also observe that on re-rendering a document using the jupyter: python3 engine that the value of an environment variable seems stuck and doesn't change even when passed as a different value on a second render. Here's a simple Quarto document.

---
format: html
jupyter: python3
---

```{python}
import os
print(os.getenv('MYVAR'))
```

Which I feel I should be able to render by setting an environment variable in my shell as usual.

MYVAR="Yes" quarto render index.qmd
MYVAR="No" quarto render index.qmd 

But the second (and additional) render never pick up the changed value of the environment variable. So the output stays as.

Quarto 1.6.40.