Sprint 9 Task List

[akotlar]

Webapp

• Fix S3 upload - March 13th for PR @akotlar
• Fix search cards to show all data - March 13th for PR @akotlar
• Hide settings wheel in queued file list (it is not currently enabled) - March 11th @akotlar
• Autocomplete field names - March 15th @akotlar
• Fix search phase exception with long query: @akotlar noticed that an aggregations (Heterozygotes per variant was cancelled after it ran for a certain amount of time). If cannot be solved, remove the offending aggregation scripts. https://github.com/bystrogenomics/bystro-web/issues/388 - @akotlar - March 28th
• Finish making AMIs for bystro-web, bystro, and opensearch servers - hard deadline is March 29th.
• (stretch) Add ability to rename genetic submissions in front end, before uploading them, by clicking on the name -@akotlar
• (stretch) Add ability to stitch together datasets.
• (stretch) Add Jupyter notebook launcher
• (stretch) support (refSeq:brca1), which would expand to (refseq.\*:brca1)
• (stretch) Update numerical facets to allow selecting on ranges using slider

Annotation

• Test hg19_v8 database by comparing results, row by row, on trio_trim.vcf.gz between hg19_v8 and dave's instance b10 database - March 15th @akotlar
• Create hg38 database - March 11th @akotlar
• Test hg38 database by comparing results, row by row, on trio_trim.vcf.gz between hg19_v8 and dave's instance b10 database - March 18th @akotlar

Ancestry

• Retrain ancestry model (hg38) with 76k gnomad loadings - @cristinaetrv - March 13th

• Liftover array set - @cristinaetrv - March 13th

• Liftover gnomad loadings - @cristinaetrv - March 14th

• Add assembly to AncestryData, update bystro-web to submit assembly, and use assembly when choosing model (Send assembly version with ancestry job requests #419) - @akotlar - March 13th

• Create ancestry docker container - @akotlar - March 14th

• Add support for choosing best covariate set for ancestry - @akotlar - March 15th

• Gives Thomas new ancestry code to test on healthy aging study - March 20th @cristinaetrv

• Add Ancestry table version that just has IDs and top hit superpops that can be expanded to larger table @akotlar

• Before expanding, top hit for superpops is seen, after expanding format remains as before

• Possible format: Sample ID | Top hit Superpop | Prob(Top hit superpop) --> Expand to see: Number Variants retained| Prob of other Superpops (5 columns) | Prob of Population 1,2,3 ...

• For populations: Instead of adding all 26 populations as something you have to open for each one, add the columns as 'Prob(CEU), Prob(CDX)' and so on for each individual

• Switch missingness to 'variants retained'

Proteomics Data Handling / API

• Finish download of proteomics data - March 12th @dlin30
• Fix frontend upload of proteomic data - March 13th @dlin30
• Add support for somascan upload in bystro webapp and api - March 14th @akotlar

Proteomics Statistics

• Jupyter notebook demonstrating adjusting for batch effects on Adverserial PPCA on simulation data + a real dataset (@akotlar) - March 14th @austinTalbot7241993

Infrastructure

• Make sure EBS scratch disk is very well provisioned, or use instance with 4TB SSD
• Update bystro webapp documentation on how to use bystro, write API documentation, write library documentation - @akotlar - March 28th

PRS

• Add AD GWAS summary statistics suggested by Thomas for C+T PRS - March 15th @cristinaetrv
• Add readme for AD GWAS sum stats - March 18th @cristinaetrv -> Move to sprint 10
• Add batch processing for PRS C+T workflow - March 25th -> Move to sprint 10
• Finish PRS-CS standard way without Langevin Dynamics - @austinTalbot7241993 - March 28th.
• Add support for covariates into PRS - @cristinaetrv - March 25th -> Backlogged for now, move to sprint 11/12

POE

• Mike Epstein was positive on Austin's POE method. Address Mike's simulation suggestions - March 28th

[akotlar]

Updates on //2024-03-29

Alex:

• IBDGC tasks are done
• Somascan support in, but not yet threaded through to bystro-api (partially blocked by proteomics submission PR)
• In process of bug fixing. Need to switch to polling strategy for currently viewed job; we're seeing evidence that our existing socketio implementation is dropping updates)

Dennis: (has been sick)

• Streaming proteomics works, will PR
• Proteomics submission in progress

Austin:

• Spectral alignment - imputation is in
• Harmonizing datasets - in progress - goal is to get an outer join (TMT + somascan), harmonized, so that the data is jointly analyzable
• PRS-CS - on back burner until proteomics is in
• SSPCA - has experimental results on 2 datasets (neuroscience, we beat L1 regularization; so we can argue our merits from purely predictive results, not just generative results as now)
• POE - we're running into: size of parent of origin effects are so tiny relative to variance of our features that gaussian mixture models end up being a poor fit; our estimator will converge almost surely, but that is not so practically relevant, for our sample sizes. So can we do better than that, especially in taming bias which is inflating Type II / false positive error rate.

[akotlar]

S3 Uploads now work, even for very large datasets (e.g. 100GB+ uploads).
bystro webapp documentation is updated, but I have not had a chance to add detailed library documentation

[akotlar]

Semi-automated AMIs have been made and deployed to IBDGC. They need 1 more pass of refinement in order to be fully capable of being taken down / up at will in autoscaling fashion, sprint 10 task