Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimisations required for stage IV simulations #10

Open
wants to merge 239 commits into
base: master
Choose a base branch
from
Open

Optimisations required for stage IV simulations #10

wants to merge 239 commits into from

Conversation

jlvdb
Copy link
Contributor

@jlvdb jlvdb commented Sep 15, 2020

  • Added support for Flagship.
  • Switched from FITS files to memory mapped binary files as intermediate storage for the pipeline.
    • Great performance gains, improved parallelism.
    • Reduced disk space requirements by optimizing how galaxy sample selection information is stored (using bit masks).
  • Improved documentation.
  • Bug fixes.

This restructured pipeline reproduces the KV450 MICE2 mock catalogues and spec-z samples from van den Busch et al. 2020 within the statistical sampling uncertainty (modulo bug fixes).

jlvdb added 30 commits March 25, 2020 21:13
@jlvdb
implemented GAaP-like apertures
9457a81
@jlvdb
implemented Gaussian flux errors (why not earlier)
07f29ef
@jlvdb
implemented GAaP SN aperture correction
9e60ba0
@jlvdb
added missing FWHM conversion
48b07c1
@jlvdb
show mag-magerr plot as contours
d7e8ac2
@jlvdb
adjusted column names in input table
893c962
@jlvdb
renamed new aperture and photometry scripts
703eb04
@jlvdb
added the original versions of the aperture and photometry scripts
7e5baf6
@jlvdb
Create GAaP_documentation.tex
e6b8598 
for a future release/publication of the catalogue
@jlvdb
added a logger
e7b89a4
@jlvdb
converted first half of the pipeline scripts
7cb57a5
@jlvdb
added multiprocessing chunksize
b3d997b
@jlvdb
show progress again
4884241
@jlvdb
switched to BinaryTable backend
3669fc9
@jlvdb
use the attr argument in add_column
80ef540
@jlvdb
changed loop order to improve performance
5aaaef2
@jlvdb
implemented pipeline entry point
7ad7abb
@jlvdb
created testing directory
4ac2c25
@jlvdb
improved the logger and the code documentation
287de81
@jlvdb
added scripts to inspect and export the data storage
ccfc181
@jlvdb
reorganised, select output columns in query, view attributes in info
0375394
@jlvdb
restructured the code
5792151
@jlvdb
added time stamp routines
7efa963
@jlvdb
added wrapper to open the data store
d518553
@jlvdb
truncate fractional seconds in time stamps
870a692
@jlvdb
changed io.py to readwrite.py
639eb45
@jlvdb
added option to list the pipeline call history
19373d6
@jlvdb
added a semi-automatic version tag based on git
91b6580
@jlvdb
added the MICE evolution correction script
9301ab8
@jlvdb
replaced print with sys.stdout.write
dbcd87c
@jlvdb
add the remaining documentation
9d6ca16
@jlvdb
added an updated README file
96f26e3
@jlvdb
added some missing instructions
1c75d4b
@jlvdb
simpliefed the example scripts
4512be8
@jlvdb
fixed an incorrect error message
8bb8736
@jlvdb
bug fixes in the commandline parsing
093322d
@jlvdb
fixed a bug in the variable substitution
ba7670d
@jlvdb
fixed a bug related to writing selected data to stdout
d7fe064
@jlvdb
added an extensive useage example
28ea42b
@jlvdb
added scripts to ingest a single column or drop columns from the data…
99586e1 
… store
@jlvdb
added documentation for new column ingestion script
e82ffaa
@jlvdb
delete the ingested column if the data copying fails
6228261
@jlvdb
add range index to Flagship data
a405037
@jlvdb
changed Flagship index type from u8 to i8 for better FITS compatibilty
6bfecc9
@jlvdb
mmaptable.mathexpression is now an independent package
5d75e3e
@jlvdb
added option to recalculate the check sums
70ee664
@jlvdb
fixe for fitsio converting string to unicode
664f015
@jlvdb
removed the flux flag from the BPZ configuration
034aa6b
@jlvdb
check the input data for invalid values
9bdbfe6
@jlvdb
remove trailing brackets in header when calling GalaxyMock methods fr…
10638da 
…om scripts
@jlvdb
added script to merge back data from image simulations
402f627
@jlvdb
added MmapTable writer and support for attributes
f170cd6
@jlvdb
added tool to create a value-initialized column
84df259
@jlvdb
fixed logging for new tool
08a12bc
@jlvdb
added option to create a unique range index
a9624ff
@jlvdb
create a range index in the test scripts and no longer in prepare_Fla…
8dc4e3b 
…gship
@jlvdb
bug fixes and added simple index based merging method
9ae1410
@jlvdb
fixed a syntax error in the argument parser
e44e6cd
@jlvdb
adjusted example script paths
c7c8b35
@jlvdb
fixed a python/numpy version related issue
389122b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@jlvdb