Native Julia engine not working: "ERROR: Internal julia server error"

[mcanouil]

Description

Using the latest development version and a very simple document:

---
format: html
engine: julia
---

```{julia}
using Plots
plot(sin, 0, 2π)
```

The above produces:

Running [1/1] at line 6:  using Plots
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd:6
ArgumentError: Package Plots not found in current path.
- Run `import Pkg; Pkg.add("Plots")` to install the Plots package.
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1772 [inlined]
  [2] macro expansion
    @ ./lock.jl:267 [inlined]
  [3] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
  [4] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
  [5] invoke_in_world
    @ ./essentials.jl:923 [inlined]
  [6] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
  [7] eval
    @ ./boot.jl:385 [inlined]
  [8] (::Main.var"#14#28"{Module, Expr})()
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:309
  [9] (::Main.IOCapture.var"#4#7"{DataType, Main.var"#14#28"{Module, Expr}, IOContext{Base.PipeEndpoint}, IOContext{Base.PipeEndpoint}, IOStream, Base.PipeEndpoint})()
    @ Main.IOCapture ~/.julia/packages/IOCapture/Rzdxd/src/IOCapture.jl:161
 [10] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:515
 [11] with_logger
    @ ./logging.jl:627 [inlined]
 [12] capture(f::Main.var"#14#28"{Module, Expr}; rethrow::Type, color::Bool, passthrough::Bool, capture_buffer::IOBuffer)
    @ Main.IOCapture ~/.julia/packages/IOCapture/Rzdxd/src/IOCapture.jl:158
 [13] include_str(mod::Module, code::String; file::String, line::Int64)
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:293
 [14] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
 [15] invokelatest
    @ ./essentials.jl:889 [inlined]
 [16] #7
    @ ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:164 [inlined]
 [17] with_inline_display(f::Main.var"#7#24"{String, String, Int64}, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:53
 [18] _render_thunk(thunk::Function, code::String, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:184
 [19] render(code::String, file::String, line::Int64, cell_options::Dict{String, Any})
    @ Main ~/.julia/packages/QuartoNotebookRunner/pl36G/src/worker.jl:161
 [20] top-level scope
    @ none:1

ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:663:13)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async executeJulia (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:511:20)
    at async Object.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:137:16)
    at async renderExecute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:232:25)
    at async renderFileInternal (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:592:35)
    at async renderFiles (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:325:9)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async renderForPreview (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:422:24)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:166:22)

The error is very cryptic as Plots is installed.

I am on macOS (latest).
Using Julia 1.10.2 (installed via Homebrew).

Quarto private environment:

~/Library/Caches/quarto/julia
├── Manifest.toml
├── Project.toml
└── julia_transport.txt

I don't see anything obvious here.

Even the following document does not work:

---
format: html
engine: julia
---

```{julia}
versioninfo()
```

Running [1/1] at line 6:  versioninfo()
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/mcanouil/Projects/quarto/quarto-playground/index.qmd:6
UndefVarError: `versioninfo` not defined
Stacktrace:
 [1] top-level scope
   @ ~/Projects/quarto/quarto-playground/index.qmd:7

ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:663:13)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async executeJulia (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:511:20)
    at async Object.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/julia.ts:137:16)
    at async renderExecute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:232:25)
    at async renderFileInternal (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:592:35)
    at async renderFiles (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:325:9)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async renderForPreview (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:422:24)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/preview/preview.ts:166:22)

While it works via Jupyter

Edit: I upgraded to 1.10.3 but I've got the exact same results.

[mcanouil]

@MichaelHatherly and @jkrumbiegel (cc @cscheid) if you have any pointers here.

[jkrumbiegel]

Two different things going on here:

I assume you have Plots installed in your main environment @v1.10. We have configured QuartoNotebookRunner to use the directory of the notebook as the default environment because this encourages using separate environments for separate projects. If you really want to use your main env from a notebook (which again is not advisable as you're bound to later make changes to it which break your notebook) you can set this in the frontmatter:

julia:
    exeflags: ["--project=@v#.#"]

The versioninfo thing is happening because Julia pulls in some stdlib functions only when it's used in interactive mode (with a REPL). The same thing happens when you directly run some code with the julia binary:

% julia -E 'versioninfo()'
ERROR: UndefVarError: `versioninfo` not defined
Stacktrace:
 [1] top-level scope
   @ none:1

In this case it's the module InteractiveUtils so you'd have to call using InteractiveUtils first:

julia> @which versioninfo
InteractiveUtils

[jkrumbiegel]

perhaps the timeouts aren't quite working as expected?

Could be, there are some tests on the QuartoNotebookRunner side but not integration tests with quarto

[jkrumbiegel]

DO you know how to kill that worker?

sorry, missed that question:

Using --execute-daemon-restart with the quarto render command or setting daemon: false for one invocation under the execute key in the frontmatter will restart the worker.

You cannot kill the main server process unless you find out its pid with --execute-debug and kill it yourself

[mcanouil]

Restarting the daemon did the trick, thanks!

I'll keep that in mind as some users might need this when troubleshooting issues with the Julia engine.

[mcanouil]

I'll probably make a PR to improve the documentation and ping you there for a review.

For the time being:

• Native Julia engine setup and documentation #9561

Thanks for the help!