localrule with --google-lifesciences does not use conda environment

[cademirch]

Snakemake version

6.15.5
Describe the bug

Running snakemake with --use-conda and --google-lifesciences, localrules are not executed in their specified conda environment.
Logs

(snakemake) ubuntu@cade-launchpad:/mnt/snakemake_test$ snakemake --google-lifesciences --default-remote-prefix mybucket --use-conda --google-lifesciences-region us-central1 -j1
Building DAG of jobs...
Unable to retrieve additional files from git. This is not a git repository.
Using shell: /usr/bin/bash
Provided cloud nodes: 1
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
test         1              1              1
total        1              1              1

Select jobs to execute...

[Mon Feb 28 17:47:59 2022]
localrule test:
    output: mybucket/done.txt
    jobid: 0
    resources: tmpdir=/tmp

/usr/bin/bash: samtools: command not found
[Mon Feb 28 17:47:59 2022]
Error in rule test:
    jobid: 0
    output: mybucket/done.txt
    shell:
        samtools --help > mybucket/done.txt
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job test since they might be corrupted:
mybucket/done.txt
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /mnt/snakemake_test/.snakemake/log/2022-02-28T174755.754811.snakemake.log

Minimal example

Snakefile:

localrules: test
rule test:
    output: "done.txt"
    conda: "env.yml"
    shell: "samtools --help > {output}"

env.yml:

channels:
  - bioconda
  - defaults
  - conda-forge
dependencies:
  - samtools>=1.14

Additional context

[nate-d-olson]

I believe this bug is also impacting local rules when using tibanna, #1302

[cademirch]

I looked into this a little bit today. Still don't really know what could be causing this... it seems that here:

snakemake/snakemake/shell.py

Lines 129 to 161 in 2a0230d

	def __new__(
	cls, cmd, *args, iterable=False, read=False, bench_record=None, **kwargs
	):
	if "stepout" in kwargs:
	raise KeyError("Argument stepout is not allowed in shell command.")
	if ON_WINDOWS and not cls.get_executable():
	# If bash is not used on Windows quoting must be handled in a special way
	kwargs["quote_func"] = cmd_exe_quote
	cmd = format(cmd, *args, stepout=2, **kwargs)
	stdout = sp.PIPE if iterable or read else STDOUT
	close_fds = sys.platform != "win32"
	func_context = inspect.currentframe().f_back.f_locals
	if func_context.get(RULEFUNC_CONTEXT_MARKER):
	# If this comes from a rule, we expect certain information to be passed
	# implicitly via the rule func context, which is added here.
	context = func_context
	else:
	# Otherwise, context is just filled via kwargs.
	context = dict()
	# add kwargs to context (overwriting the locals of the caller)
	context.update(kwargs)
	jobid = context.get("jobid")
	if not context.get("is_shell"):
	logger.shellcmd(cmd)
	conda_env = context.get("conda_env", None)

conda_env is None. So for some reason the rule function context does not have conda_env set. Will do some more digging but could use any insight/advice, @johanneskoester

[cademirch]

It seems that the issue is probably earlier than my above post, since Snakemake does not even create the conda env. Which when Snakemake runs successfully, happens first.

[cademirch]

I believe the problem could begin here:

snakemake/snakemake/workflow.py

Lines 967 to 971 in 2a0230d

	if self.use_conda:
	if self.assume_shared_fs:
	dag.create_conda_envs(
	dryrun=dryrun or list_conda_envs or conda_cleanup_envs,
	quiet=list_conda_envs,

self.assume_shared_fs is False - which makes sense because I am using the google lifesciences executor. However, that prevents us from calling dag.create_conda_envs. In dag.create_conda_envs there is a check if the job is local:

snakemake/snakemake/dag.py

Line 283 in 2a0230d

if job.conda_env_spec and (self.workflow.assume_shared_fs or job.is_local)

But that doesn't matter, since we never call the function in this case.

Commenting out the check for self.assume_shared_fs allows the MRE above to run just fine. I'm curious why that check is there to begin with - that would be helpful to know since I'm not sure what unintended consequences there may be in removing it.

Edit: Looking at the git blame for this line its referenced in #1472, so just commenting out this line would probably not be idea. @johanneskoester Would appreciate your feedback/thoughts on this issue.

[cademirch]

@nate-d-olson Could you test this fix on Tibanna? I imagine it should work.

[nate-d-olson]

@cademirch snakemake v > 7 does not work with Tibanna, see #1475, so I am unable to test your proposed fix. When #1475 is fixed I'll test out your fix.

[cademirch]

Still an issue in 7.2.1. @johanneskoester Could you take a look at this? It seems that the check for shared_fs was implemented b/c of issue #1463. Though I'm curious if the check needs to happen since this line:

snakemake/snakemake/dag.py

Line 283 in 2a0230d

if job.conda_env_spec and (self.workflow.assume_shared_fs or job.is_local)

Should solve the issue of creating the same conda env many times? Not sure though.

[johanneskoester]

Thanks a lot. I agree, and have implemented the fix in #1490.