Skip to content

Project Meeting 2018.11.30

Ben Stabler edited this page Nov 30, 2018 · 11 revisions

Multi-processing

  • Ran 100% sample full scale example in 59 minutes on one Azure Linux machine with 64 processors and 432GB RAM
  • Ran the same example in 2 hours on an Azure Windows box
  • The Linux example includes tweaks to the low level Intel MKL settings used by OpenMP and pandas/numpy
  • We plan to experiment a bit more with the same MKL environment variables settings on Windows to decrease runtimes
  • Cost of the Linux run was $5 and max RAM usage was 242GB (see below)
  • Added three new example configurations, see dev:
    • mp - Multi-processing setup
    • azure - Azure cloud based setup
    • stride - Separate machines setup (but not really ready for use since incomplete)
  • Coding largely complete, just inline docs and some settings clean-up required
  • @RSG still todo:
    • Do some local machine Windows runs with smarter Intel MKL environment variable settings
    • Run a cloud based 128 processor 2TB RAM setup as well
    • Revise user documentation and then do a pull request (and release)
    • Plan to complete by TRB
  • @PSRC and SFCTA to experiment with mp as well
    • PSRC has mp running locally and on AWS; both with Windows

Shadow pricing

  • Started reviewing existing CT-RAMP and DaySim code and draft plan, see Task 3
  • Drafted a rough prototype this week that works in a single process
  • For mp, need to collate results after each iteration to calculate shadow price
  • For the user configuration, a new concept is added - shadow price segment
    • This allows for specifying shadow price segments, worker, elem student, high student, etc. and an expression file to identify it + what size terms to use
    • For geography constraints, these would be handled by adding an expression to the sampling expression file (such as home school district)
  • Will allow for multiple types of convergence - iterations, goodness-of-fit, etc.
  • Planning to complete shadow pricing by TRB as well
Clone this wiki locally