Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug Cache miss #7248

Open
powelleric opened this issue Nov 7, 2023 · 0 comments
Open

Debug Cache miss #7248

powelleric opened this issue Nov 7, 2023 · 0 comments

Comments

@powelleric
Copy link

powelleric commented Nov 7, 2023 •
edited

Call caching works sometimes for me but not all the time. I find it especially strange when working on a scatter job and some of the scatter jobs get a cache hit but others get a cache miss.

I have queried the METADATA_ENTRY table for the two workflows and all the call cache entries look identical.

Here is my process:

  1. I queried METADATA_ENTRY with this WHERE condition: (WORKFLOW_EXECUTION_UUID ='29791b64-b47a-44ba-aff0-7ab48bc10677' or WORKFLOW_EXECUTION_UUID ='5de042e3-7a03-4c77-8972-f0e4cd010e4b') and CALL_FQN = 'sampleLevelWorkflow_WGS.align' and JOB_SCATTER_INDEX =0
  2. I sort by METADATA_KEY
  3. Then I go down the list and compare the hashes for the two workflows for each METADATA_KEY.

Here is a case where workflow 29791b64 is a restart of 5de042e3. (Workflow 5de042e3 is itself a restart but I don't think that is important here.) I have shown below all the records from METADATA_ENTRY that start with "callCaching" and they all look identical, yet it clearly says it is a "Cache Miss".

Is there anywhere I can see a log message stating exactly which hashes resulted in the cache miss? I have tried to enable LOG_LEVEL=DEBUG but couldn't see it there.

Thanks in advance for your help!

WORKFLOW_EXECUTION_UUID METADATA_KEY METADATA_VALUE
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:result Cache Miss
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:result Cache Miss
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hit false
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hit false
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:runtime attribute:failOnStderr 68934A3E9455FA72420237EB05902327
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:runtime attribute:failOnStderr 68934A3E9455FA72420237EB05902327
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:runtime attribute:docker 4AD3C387725244C1348F252B031B956D
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:runtime attribute:docker 4AD3C387725244C1348F252B031B956D
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:runtime attribute:continueOnReturnCode CFCD208495D565EF66E7DFF9F98764DA
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:runtime attribute:continueOnReturnCode CFCD208495D565EF66E7DFF9F98764DA
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:output expression:File sorted_bam 816FBD6EEA5D806309AA0664E2F2AB86
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:output expression:File sorted_bam 816FBD6EEA5D806309AA0664E2F2AB86
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:output count C4CA4238A0B923820DCC509A6F75849B
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:output count C4CA4238A0B923820DCC509A6F75849B
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String sampleRunID A3F4D40B1A8326642B9C761E2FE16F3E
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String sampleRunID A3F4D40B1A8326642B9C761E2FE16F3E
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String sampleGenomicID 5844E24465D27D94BD0D311D5D189FC9
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String sampleGenomicID 5844E24465D27D94BD0D311D5D189FC9
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String runtime_attr:queueArn 2382ECF7BF8F09D7CD67C24B52AD78BF
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String runtime_attr:queueArn 2382ECF7BF8F09D7CD67C24B52AD78BF
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String runtime_attr:docker A5281F25296D4311ED0C46422719018E
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String runtime_attr:docker A5281F25296D4311ED0C46422719018E
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String default_attr:queueArn 2382ECF7BF8F09D7CD67C24B52AD78BF
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String default_attr:queueArn 2382ECF7BF8F09D7CD67C24B52AD78BF
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:String default_attr:docker A5281F25296D4311ED0C46422719018E
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:String default_attr:docker A5281F25296D4311ED0C46422719018E
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Int runtime_attr:max_retries CFCD208495D565EF66E7DFF9F98764DA
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Int runtime_attr:max_retries CFCD208495D565EF66E7DFF9F98764DA
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Int runtime_attr:cpu_cores C74D97B01EAE257E44AA9D5BADE97BAF
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Int runtime_attr:cpu_cores C74D97B01EAE257E44AA9D5BADE97BAF
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Int numCPUs C74D97B01EAE257E44AA9D5BADE97BAF
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Int numCPUs C74D97B01EAE257E44AA9D5BADE97BAF
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Int default_attr:max_retries CFCD208495D565EF66E7DFF9F98764DA
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Int default_attr:max_retries CFCD208495D565EF66E7DFF9F98764DA
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Int default_attr:cpu_cores C74D97B01EAE257E44AA9D5BADE97BAF
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Int default_attr:cpu_cores C74D97B01EAE257E44AA9D5BADE97BAF
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Float runtime_attr:mem_gb 5BA1DE412E01037F8843D097DCFAF28A
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Float runtime_attr:mem_gb 5BA1DE412E01037F8843D097DCFAF28A
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:Float default_attr:mem_gb 5BA1DE412E01037F8843D097DCFAF28A
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:Float default_attr:mem_gb 5BA1DE412E01037F8843D097DCFAF28A
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:File R2Fastq "9f1cf8859a902eb75202a5c048cd43aa-388"
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:File R2Fastq "9f1cf8859a902eb75202a5c048cd43aa-388"
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input:File R1Fastq "62396abd6b589747ee16034888c9a0b5-381"
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input:File R1Fastq "62396abd6b589747ee16034888c9a0b5-381"
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:input count 9BF31C7FF062936A96D3C8BD1F8F2FF3
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:input count 9BF31C7FF062936A96D3C8BD1F8F2FF3
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:command template 7BCEDB02C5FC300FF83F07417B49229E
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:command template 7BCEDB02C5FC300FF83F07417B49229E
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:hashes:backend name 2267EF43AEF6BB551F414FEC2390F68A
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:hashes:backend name 2267EF43AEF6BB551F414FEC2390F68A
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:effectiveCallCachingMode ReadAndWriteCache
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:effectiveCallCachingMode ReadAndWriteCache
29791b64-b47a-44ba-aff0-7ab48bc10677 callCaching:allowResultReuse true
5de042e3-7a03-4c77-8972-f0e4cd010e4b callCaching:allowResultReuse true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@powelleric