Option for caching generated Zsh code from external commands

[00dani]

Is your feature request related to a problem? Please describe.

There are many programs that expect to be integrated into your shell by writing something like eval "$(theprogram init)" into your shell initialisation script. The program generates some shell code on the fly to hook itself into the appropriate parts of your shell. Direnv, Starship, Zoxide, and Carapace all recommend this installation method, for example, as do various "version manager" tools like rbenv and Pyenv.

There are also programs like vivid which expect to be called during shell initialisation and have their output saved to a parameter (LS_COLORS in this case), rather than directly evaluated as shell script. Expecting this seems to be less common - even similar tools like dircolors produce executable code that sets LS_COLORS rather than simply a value that needs to be saved to LS_COLORS - but it does happen, and I personally use vivid so I noticed.

However this code-generation approach can be a source of slowdown, because it needs to fork and exec the external program every time your shell starts. Additionally, the Zsh code generated by the external program cannot be zcompiled, since it is always produced on the fly. Since the actual generated code produced by these commands changes relatively rarely, it makes a lot of sense to prebuild it, zcompile it, and source that file on startup instead. Zim doesn't have a built-in way to do this, and I think it would be nice if it did.

Describe the solution you'd like

Inspired by a similar feature in the Znap tool, Zim could provide an additional command inside the zimrc called something like zeval, which would be invoked like this:

zeval direnv 'direnv hook zsh'
zeval zoxide 'zoxide init zsh'

When such a definition is "installed", Zim would run the provided command once, save the result to a new init.zsh file, and zcompile that file. Then on each launch, Zim would simply source that same init.zsh along with everything else.

Scenarios like vivid, where the command output needs to be saved to a parameter rather than simply evaluated, could be supported using a flag like this:

zeval vivid -p LS_COLORS 'vivid generate molokai'

When this is provided, rather than just writing the command's output directly to init.zsh, Zim could generate something like export LS_COLORS="output of command" and write that to init.zsh instead.

Zim could detect when the command in zimrc has been changed (say, from vivid generate molokai to vivid generate gruvbox-dark) and rerun it to produce a new cached result in that case, but otherwise keep using the cached output. Additionally a zimfw subcommand for invalidating cached evals would be useful.

Describe alternatives you've considered

I've already come up with a hacky way to implement this idea with Zim's existing features. It looks like this, in my zimrc:

zeval() {
  zmodule https://git.00dani.me/00dani/null --name zeval-$1 --on-pull "$2 >! init.zsh"
}
zeval-if-installed() {
  (( ${+commands[$1]} )) && zeval "$@"
}

# Installing other zmodules here in the normal way...

zeval-if-installed direnv 'direnv hook zsh'
zeval-if-installed zoxide 'zoxide init zsh'
zeval-if-installed starship 'starship init zsh --print-full-init'
zeval-if-installed vivid 'echo export LS_COLORS=${(qqq)$(vivid generate molokai)}'

zmodule completion
# Carapace calls compdef, so it needs to be sourced after compinit to work properly
zeval-if-installed carapace 'carapace _carapace zsh'

unfunction zeval zeval-if-installed

While this does absolutely work, this setup has a number of shortcomings: it's needlessly cloning an empty Git repository for every command I add, there's no simple way to invalidate the cached script when necessary, and the syntax I need to use for "save to a parameter" commands like vivid is honestly pretty gross!

I could fix that last issue myself by teaching my zeval function about the -p flag I suggested above, but the other problems really need support in Zim itself to be fixed properly.

Outside of my personal hack, I've found there are also plugins like evalcache and zsh-smartcache which implement this idea as a standalone utility. However these approaches have their own shortcomings: they don't support caching of a parameter assignment (so they don't work with vivid) , they run totally independently of your plugin manager so zimfw naturally can't manage them, and they both unconditionally shell out to md5 or md5sum to hash your command so you're still not actually avoiding a fork/exec by using them. Strangely, they also do not attempt to zcompile the cached output. Not sure why.

Additional context

No response

[ericbn]

Hi @00dani. This is something I've been considering for a while. Thank you for the investigation you've done above.

I see another scenario too, besides (1) source the output and (2) set environment variable value to output, which is (3) generate completion from output. But maybe all these cases under (3) can be generalized using (1). For example, this is what we do to initialize the completion for kubectl: https://github.com/zimfw/k/blob/b3567c755a56a34d7de44575f45dcbeedac2ebbd/init.zsh#L1-L8

[ericbn]

I've just released version 1.13.0 with two new zmodule options that make it easier to do what is being proposed here: the --if-command option, and the mkdir tool which creates an empty directory. This is how to use them to implement a zmodule-eval function:

zmodule-eval() {
  local -r ztarget=${2//[^[:alnum:]]/-}.zsh
  zmodule custom-${1} --use mkdir --if-command ${1} \
      --cmd "if [[ ! {}/${ztarget} -nt \${commands[${1}]} ]]; then ${2} >! {}/${ztarget}; zcompile -UR {}/${ztarget}; fi" \
      --source ${ztarget}
}
zmodule-eval starship 'starship init zsh'
unfunction zmodule-eval

The difference of this function with the zeval-if-installed above is that it does the checks during startup, not during the module update: check if the command is installed, check if the generated file is outdated in comparison to the command timestamp, and zcompile the generated file.

This is for example what is generated in the init.zsh file from the example above:

if (( ${+commands[starship]} )); then
  if [[ ! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh -nt ${commands[starship]} ]]; then starship init zsh >! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; zcompile -UR /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; fi
  source /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh
fi

[dannysteenman]

I've just released version 1.13.0 with two new zmodule options that make it easier to do what is being proposed here: the --if-command option, and the mkdir tool which creates an empty directory. This is how to use them to implement a zmodule-eval function:

zmodule-eval() {
  local -r ztarget=${2//[^[:alnum:]]/-}.zsh
  zmodule custom-${1} --use mkdir --if-command ${1} \
      --cmd "if [[ ! {}/${ztarget} -nt \${commands[${1}]} ]]; then ${2} >! {}/${ztarget}; zcompile -UR {}/${ztarget}; fi" \
      --source ${ztarget}
}
zmodule-eval starship 'starship init zsh'
unfunction zmodule-eval

The difference of this function with the zeval-if-installed above is that it does the checks during startup, not during the module update: check if the command is installed, check if the generated file is outdated in comparison to the command timestamp, and zcompile the generated file.

This is for example what is generated in the init.zsh file from the example above:

if (( ${+commands[starship]} )); then
  if [[ ! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh -nt ${commands[starship]} ]]; then starship init zsh >! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; zcompile -UR /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; fi
  source /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh
fi

I've tried adding this eval function to my .zshrc but I get the error: zmodule: Must be called from ~/.zimrc. Am I doing something wrong? I'm interested in using this feature :)

[ericbn]

Hi @dannysteenman. The code above defining, using and undefining zmodule-eval should be in your .zimrc instead.

[dannysteenman]

Thanks for the quick reply @ericbn, I forgot to mention that I've done that as well. However the completion I want to add (in this case for the package orbstack) doesn't activate.

contents of .zimrc:

zmodule-eval() {
    local -r ztarget=${2//[^[:alnum:]]/-}.zsh
    zmodule custom-${1} --use mkdir --if-command ${1} \
        --cmd "if [[ ! {}/${ztarget} -nt \${commands[${1}]} ]]; then ${2} >! {}/${ztarget}; zcompile -UR {}/${ztarget}; fi" \
        --source ${ztarget}
}
zmodule-eval orb 'orb completion zsh'
unfunction zmodule-eval 
#
# Completion
#
zmodule junegunn/fzf --source shell/completion.zsh --source shell/key-bindings.zsh
# Additional completion definitions for Zsh.
zmodule zsh-users/zsh-completions --fpath src
# Enables and configures smart and extensive tab completion.
# completion must be sourced after all modules that add completion definitions.
zmodule completion

#
# Modules that must be initialized last
#
zmodule Aloxaf/fzf-tab
zmodule paulirish/git-open
# Fish-like syntax highlighting for Zsh.
# zsh-users/zsh-syntax-highlighting must be sourced after completion
zmodule zdharma-continuum/fast-syntax-highlighting
# Fish-like history search (up arrow) for Zsh.
# zsh-users/zsh-history-substring-search must be sourced after zsh-users/zsh-syntax-highlighting
zmodule zsh-users/zsh-history-substring-search
# Fish-like autosuggestions for Zsh.
zmodule zsh-users/zsh-autosuggestions

#
# Theme
#
zmodule romkatv/powerlevel10k --use degit --source powerlevel10k.zsh-theme

I've also tried adding the eval commands after the completion modules but unfortunately it didnt help.

normally when I type orb<space><tab> the helper completion should pop up.

it seems the orb completion has been installed:

❯ zimfw list
custom-orb: ~/.zim/modules/custom-orb
fzf: ~/.zim/modules/fzf
zsh-completions: ~/.zim/modules/zsh-completions
completion: ~/.zim/modules/completion
fzf-tab: ~/.zim/modules/fzf-tab
git-open: ~/.zim/modules/git-open
fast-syntax-highlighting: ~/.zim/modules/fast-syntax-highlighting
zsh-history-substring-search: ~/.zim/modules/zsh-history-substring-search
zsh-autosuggestions: ~/.zim/modules/zsh-autosuggestions
powerlevel10k: ~/.zim/modules/powerlevel10k

contents of ~/.zim/modules/custom-orb:

 orb-completion-zsh.zsh   orb-completion-zsh.zsh.zwc

[vinismarques]

I've just released version 1.13.0 with two new zmodule options that make it easier to do what is being proposed here: the --if-command option, and the mkdir tool which creates an empty directory. This is how to use them to implement a zmodule-eval function:

zmodule-eval() {
  local -r ztarget=${2//[^[:alnum:]]/-}.zsh
  zmodule custom-${1} --use mkdir --if-command ${1} \
      --cmd "if [[ ! {}/${ztarget} -nt \${commands[${1}]} ]]; then ${2} >! {}/${ztarget}; zcompile -UR {}/${ztarget}; fi" \
      --source ${ztarget}
}
zmodule-eval starship 'starship init zsh'
unfunction zmodule-eval

The difference of this function with the zeval-if-installed above is that it does the checks during startup, not during the module update: check if the command is installed, check if the generated file is outdated in comparison to the command timestamp, and zcompile the generated file.

This is for example what is generated in the init.zsh file from the example above:

if (( ${+commands[starship]} )); then
  if [[ ! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh -nt ${commands[starship]} ]]; then starship init zsh >! /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; zcompile -UR /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh; fi
  source /path/to/.zim/modules/custom-starship/starship-init-zsh.zsh
fi

Thank you for your great work @ericbn!

I'm trying to get starship to work in this setup you made possible on version 1.13.0, but I am unable to do so. And, to be honest, I'm not entirely sure how much this is better than just using in `.zshrc`, but figured I would give it a try since you made a solution for this.

I installed Starship using the method described in their page curl -sS https://starship.rs/install.sh | sh and then added the same block you suggested here to the .zimrc file. It doesn't seem to be applying all configurations if that makes sense. I can see this got appended to the init.zsh file, and my prompt changes a bit, but not 100% to how it should look like. I have it configured to break a line after every command, for example, and this is not what I see running it this way.

if (( ${+commands[starship]} )); then
  if [[ ! /home/vinicius/.zim/modules/custom-starship/starship-init-zsh.zsh -nt ${commands[starship]} ]]; then starship init zsh >! /home/vinicius/.zim/modules/custom-starship/starship-init-zsh.zsh; zcompile -UR /home/vinicius/.zim/modules/custom-starship/starship-init-zsh.zsh; fi
  source /home/vinicius/.zim/modules/custom-starship/starship-init-zsh.zsh
fi

It does apply all changes (including the line breaks) if I run this code directly on a ZSH terminal. Not sure what I'm missing here. Any tips?]

Got it! What was missing is that when I was running eval "$(starship init zsh)" it was after Conda's initialization, so starship was removing the text that Conda automatically inserts, but when I was running on .zimrc I had the impression that the formatting was a bit off due to the line break missing since Conda was writing text there.

The zmodule-eval works great! Thank you for making this even easier to use.