Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: in google live science backend, save multiple logs per rule name and overwrite existing logs #1504

Merged
merged 7 commits into from Mar 23, 2022
Merged

fix: in google live science backend, save multiple logs per rule name and overwrite existing logs #1504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f6f3670
remove check for existing blob so logs overwrite
cademirch Mar 20, 2022
3493f74
add jobid to dest_path for logs
cademirch Mar 20, 2022
4d8143e
add .txt to stderr, stdout, and output to view in browser
cademirch Mar 20, 2022
04af9ec
testing adding .txt to log files
cademirch Mar 21, 2022
84c8940
add content_type arg to upload for txt extension
cademirch Mar 21, 2022
b27f424
change script url
cademirch Mar 21, 2022
fd29e7c
get rid of uneeded print
cademirch Mar 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions snakemake/executors/google_lifesciences.py
View file
Open in desktop
Expand Up @@ -721,13 +721,12 @@ def _generate_log_action(self, job):
"""generate an action to save the pipeline logs to storage."""
# script should be changed to this when added to version control!
# https://raw.githubusercontent.com/snakemake/snakemake/main/snakemake/executors/google_lifesciences_helper.py

# Save logs from /google/logs/output to source/logs in bucket
commands = [
"/bin/bash",
"-c",
"wget -O /gls.py https://raw.githubusercontent.com/snakemake/snakemake/main/snakemake/executors/google_lifesciences_helper.py && chmod +x /gls.py && source activate snakemake || true && python /gls.py save %s /google/logs %s/%s"
% (self.bucket.name, self.gs_logs, job.name),
"wget -O /gls.py https://raw.githubusercontent.com/snakemake/snakemake/main/snakemake/executors/google_lifesciences_helper.py && chmod +x /gls.py && source activate snakemake || true && python /gls.py save %s /google/logs %s/%s/jobid_%s"
% (self.bucket.name, self.gs_logs, job.name, job.jobid),
]

# Always run the action to generate log output
Expand All @@ -738,6 +737,7 @@ def _generate_log_action(self, job):
"labels": self._generate_pipeline_labels(job),
"alwaysRun": True,
}

return action

def _generate_job_action(self, job):
Expand Down
8 changes: 2 additions & 6 deletions snakemake/executors/google_lifesciences_helper.py
View file
Open in desktop
Expand Up @@ -49,19 +49,15 @@ def save_files(bucket_name, source_path, destination_path):

# The relative path of the filename from the source path
relative_path = filename.replace(source_path, "", 1).strip("/")

# The path in storage includes relative path from destination_path
storage_path = os.path.join(destination_path, relative_path)
full_path = os.path.join(bucket_name, storage_path)
print(
"{filename} -> {full_path}".format(filename=filename, full_path=full_path)
)

# Get the blob
blob = bucket.blob(storage_path)
if not blob.exists():
print("Uploading %s to %s" % (filename, full_path))
blob.upload_from_filename(filename)
print("Uploading %s to %s" % (filename, full_path))
blob.upload_from_filename(filename, content_type=".txt")


def get_source_files(source_path):
Expand Down