Skip to content

Commit

Permalink
maintenance release (#666)
Browse files Browse the repository at this point in the history
* keep black 2022 style for now

* typo in link

* note about pre-package installer in quick start

* add model_settings to estimator.write_coefficients

* BayDAG Contributions (#657)

* code changes to add joint tour utilities in cdap

* add joint tour frequency composition component

* post process jtfc result to tours

* update joint tour participation

* update #inmtf to tm2 specs

* allow alternative id > 127

* set up testing infrastructure

* cdap test script

* cdap configs

* set up testing infrastructure

* joint tour test script

* joint tour configs

* set up testing infrastructure

* nm tour frequency test script

* nm tour frequency configs

* add jtfc alt table dictionary to yaml

* update jtfc yaml

* jtfc move coef values to coef.csv

* jtfc update preprocessor

* add jtfc alt table dictionary to yaml

* nmtf consolidate all -999 to one coef

* code changes to fix bug in parking location choice model

* Revert "code changes to fix bug in parking location choice model"

This reverts commit 5a38ebb.

* parking location choice bug fix

* move coefficients to csv file

* restore the original mandatory channels

* restore the original mandatory channels

* RSG Phase 7 Development (#49)

* school escorting initial commit

* bundles created

* added pure escort tours after non_mand_sched

* created escorting tours & trips

* integrated escort tours and trips

* Differentiate examples between quality and validity of example models (#571)

* change examples that are no longer tied to an agency to fictional place names

* change name of full example also

* add back missing output dir

* restore test output dir also

* more empty dirs that got lost

* clean up docs

* example_mtc -> prototype_mtc

* Prototype MTC extended

* add all the ignored files

* add test output dirs

* remove superfluous example_test

* prototype_sf

* prototype_arc

* prototype_marin

* move dirs

* psrc

* semcog

* sandag_xborder

* placeholder_sandag

* placeholder_multiple_zone

* no more coveralls

* repair docs

* clean up example names

* black and isort (#578)

* black and isort

* stop emitting output dir

it fails later tests

* trace files in nested directories

because windows

* swap files for xborder model

* repair ARC MP

* downstream model integration

* print checksum even when not used

* add hashes for sandag_xborder_full

* fix dtype in university hack

* fix persons to match tours

* repair ARC

* initial commit of flexible tour/trip ids

* pycodestyle

* black formatting

* Bump version: 1.0.4 → 1.1.0

* adding frequency alternatives configs required for test system

* added additional unit tests

* added setup function for test configs dir

* formatting

* handling missing alts files in examples

* error catching around extension probs

* passing tests

* still passing tests around missing config files

* accounting for missing mtf spec in marin example

* nm tour scheduling passing

* updating stop freq in tours table

* added check for mand tour overlap with pe tours

* num chauffeurs and escortees

* fixed defaults not getting used if file missing

* merging canonical ids from flexible id work

* setting escorting tour and trip ids

* remove unneeded nmtf settings file

* black formatting

* excluding escort trips from trip scheduling

* fixing bug where mand tours not getting assigned correct id

* adding school_escorting to mp model list

* Added mwcog small area

* missed one edit

* adding school escort tour flavors as own category

* reformatting

* updating timetable windows with pure escort tours

* additional logging

* Update README.MD

* call as module

* github actions tests

* adding non-mand destination changes

* pre commit hooks

* pyproject toml

* limit numpy

* docs for mamba instead of conda

* ignore generated files

* add nbmake to test env

* fix dupe line

* fixing bad tdd merge

* repair test multiple zones for github actions

* publish develop docs

* fix docbuild env

* merging tdd alts to all tours

* adding ride share threshold to unvailability for pure escort time overlap

* cleanup

* Update .travis.yml

* fixed testing files

* fixed testing files (again)

* fixed test script again

* mins per time bin

* black formatting

* black formatting

* fixing reindex import bug

* fixing missed import

* replacing trips test table

* inserting default setting if no models in config for tests

* publish docs to branch name

* adding setup function to tests to set configs_dir injectable

* documentation

* updated testing scripts (note shorter travis script for now)

* fixed slash (windows vs. linux testing issue)

* added output folders

* updated travis script to run all tests, should pass

* docs cleaning

* docs re-style

* rebuild

* dynamic versioning docs

* version switcher

* blacken

* fix switcher url

* fix conf

* switcher update

* master to main

* deployment actions

* actions

* build wheel

* fix for testpypi

* blacken

* manual switcher

* branch docs service [makedocs]

* syntax [makedocs]

* travis depends

* checkout v3, fix versioning in docs

* only build develop docs once

* failsafe version

* documentation repairs

* python-simplified-semver

* front cards

* end testing w travis

* add mwcog test to gh-actions

* add mwcog to docs

* fixed origin bug and missing outbound trip to work

* point to data not copy it

* sort dependencies

* req sh 2.2.4

* account for variance across platforms in trip dest est

* copy bike skims for sandag test

* proper cleanup when trips that get removed if schedule failed

* changed failed trip scheduling option for example

* updating regress table

* blacken

* extending nmtf and stop frequency to demonstrate flexible ids

* param on rtol

* fix sandag_2 test files

* fix test file names

* allowing multiple escort tours in the same period

* sandag 3-zone example fix

* fixing expression for missing escort participants sandag 3_zone

* cleanup

* blacken

* Update trip_destination.py

applying valid primary origin default value

* Update trip_destination.py

blacken

* fixing bug for inbound destination for second escortee

* blacken

* Disaggregate Accessibilities (#5)

* initial commit with basic working proto pop gen

* fixed bad dependency range

* fixed vary_on method to vary all with mapped fields after

* added check if already dataframe

* added dataframe check if the table passed is already a dataframe

* working proto pop gen. still debugging mandatory work/school model

* check all table is df

* workplace runs, next testing school

* began setting up to get all logsums, not just chosen ones

* restructured to run as either one-off or model step

* revert to last merge. Removed dataframe check in iterate, no longer needed to modify this file

* extracted logsums, need to inject into pipeline or save output?

* working standalone or step model

* extractable accessibilities for fixed and nonmandatory

* extractable accessibilities for fixed and nonmandatory

* cleanup and add documentation

* resolved duplicates of old 'example' folders

* model uses write_tables functionality

* disaggregate accessibility runs as model step or standalone. Runs as subprocess

* working model test. table initialization not yet working

* added logsum short circuit to get logsums and avoid drawing unneccesary samples

* working model plus major cleanup

* working model plus major cleanup

* override sample size

* fixed trace output files

* skip_choice default to False

* fixed empty logsums bug

* cleanup redundent parameters

* fixed list index instead of set

* coordinated model run files

* added default skip to false

* added multiprocessing line

* began setup for multiprocessing

* fixed sampling problem. Pipeline for multi processing, NOT WORKING YET

* deleted run file, not needed anymore

* update release instructions

* auto-build docs on release

* remind to update switcher

* need checkout

* added helper functions for handling suffix args

* to enable multiprocessing, overhauled pipeline method to include proto_ tables rather than overwriting main tables

* working multiprocess, but requires debugging. fails on other models

* working multiprocess, but requires debugging. fails on other models

* cleanup of mp table registering

* fixed tracing and slicing issue

* removed old 'run' model

* setup example 2 zones

* minimum working 2 zone

* multizone basic working

* fixed buggy settings

* multiprocessing working, but empty final tables

* fixed blank output bug and cleaned up settings to use verbose table names (not just 'disaggregate_accessibility')

* cleanup run scripts a bit

* fixed missing base path for full run

* fixed missing path

* fixed typo

* fixed 1 per taz sampling

* fixing person merge bug and filtering workers

* fixed duplicate tours!

* updated settings for initialize disagg

* removed obsolete file

* cleaned up file and moved initialize to tables/...

* added find nearest node function

* added initialize disaggregate_accessibility to initialize_households

* moved initialize steps to 'tables/disaggregate_accessibility'

* updated settings to include initialize disaggregate accessibilities as part of initialize_households

* moved initialize disagg accessibilities to tables. Created working merge step using naive bayes and 'hard' join

* add mp back in

* PEP formatting revisions

* fixed logsums merge on households instead of persons

* fixed _accessibility suffix

* fixed conflict with persons_merged

* updated yaml to use simpler join method

* PEP formatting fixes

* refreshed example folder from develop

* added missing line at end of file

* black fixes

* black fixes to disaggregate accessibility changes

* fixed missing column pipeline error

* merged disagg accessibilities into mtc_extended and added doc section

* ran black on disaggregate_accessibility.py

* updated dependencies

* removing sklearn imports

* blacken

* add if none catch

* fixed None suffix default

* moved order of get_table(persons_merged) to avoid pulling prematurely in xborder

* tested and cleaned up rng channels

* setup injectable suffixes to allow add_size_table as model step not just function

* removed accessibility output from test

* re blacken py scripts

* fixed tracing typo

* added variable index name suffix to pass optionally

* pipeline housekeeping to cleanup any tables, traceables, or channels that were touched during disagg

* added multiprocess testing

* blacken updates

* updated test scripts to include MP, problem with vehicle model in mp (even without disagg acc?)

* added improved origin sampling, resolved issue with merging with sample <100%. Need to address random seed for origins

* added sci-kit learn to test depends

* fixed person merging error causing pytest fail, uses inject method to ensure consistency

* cleanup comments

* fixed pytest to include accessibility table in regress

* setup for mp test, but needs debugging

* 'blacken'

* 'blacken'

* cleanup example folder

* fixed pipeline NoneType bug for disagg accessibility table

* fixed pytest fail on mp, due to exept:True on mp_simulate

* 'blacken'

* added weighted k-means method

* created run script for multiple sampling scenarios

* blacken changes

* blacken

* fixed copy script

* blacken sampling script

* fixed n_zone when integer

* fixed typo

* update sampling script

* 'blacken'

* fixed replacement sample bug

* updated documentation

* more flexible scenario testing

* blacken

Co-authored-by: Nick Fournier <nick.fournier@rsginc.com>
Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Nick Fournier <99497883+nick-fournier-rsg@users.noreply.github.com>

* Shadow Pricing Enhancements (#7)

* updated scripts to include simulation-based shadow pricing

* blacken

* Updated shadow_pricing.yaml for mtc example

* code cleanup

* more cleanup

* documentation and passing tests

* passing tests

* passing tests

* updated doc on shadow pricing

* 2nd Update model doc on shadow pricing

* more doc update on shadow pricing

* fixing pandas future warning

* blacken

* bug in trying to access shadow price settings when not running shadow pricing

* limiting pandas version

* always updating choices

* testing removal of lognormal for hh vot

* putting hh vot back in

* updating to match sharrow test versions

* raw person table for buffer instead of injectable

* adding segmentation, output by iteration, and external worker removal

* formatting & documentation

Co-authored-by: aletzdy <58451076+aletzdy@users.noreply.github.com>

* updating mtc_extended test files

* no sp for non-work or school, better logging, weighting options

* sample TAZ only if available MAZ when shadow pricing

* adding missed weight column option

* cleaning up comments

Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>
Co-authored-by: Nick Fournier <nick.fournier@rsginc.com>
Co-authored-by: Nick Fournier <99497883+nick-fournier-rsg@users.noreply.github.com>
Co-authored-by: aletzdy <58451076+aletzdy@users.noreply.github.com>

* set up testing infrastructure (#40)

Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>

* BayDAG auto ownership configs and test (#41)

* set up testing infrastructure

* auto ownership test script

* auto ownership configs

* ao move coef values to coef.csv

Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>

* blacken

* more formatting

* blacken

* blacken

* blacken

* BayDAG parking location choice configs and test (#45)

* code changes to fix bug in parking location choice model

* parking location choice configs

* Revert "code changes to fix bug in parking location choice model"

This reverts commit 5a38ebb.

* parking location choice bug fix

* parking location choice config updates

move coefficients to csv file and activitysim variable naming consistency

* parking location choice summary

Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>

* blacken

* cdap bug fixes

* adding sklearn to dependencies

* Update activitysim-dev.yml

fixing bad reference to scikit-learn

* Update activitysim-test.yml

fixing bad reference to scikit-learn

* Update setup.cfg

fixing bad reference to scikit-learn

* Memory Bug Fix

* working with 2k no sp

* optimize nearest zone determination

* Changed data type of participant_id in candidates in joint_tour_participation_candidates() to unsigned 64-bit integer to prevent overflow

* better comments

* fix overflow in jtp participant id

* blacken

* forcing participation

* Trip Scheduling (#51)

* Differentiate examples between quality and validity of example models (#571)

* change examples that are no longer tied to an agency to fictional place names

* change name of full example also

* add back missing output dir

* restore test output dir also

* more empty dirs that got lost

* clean up docs

* example_mtc -> prototype_mtc

* Prototype MTC extended

* add all the ignored files

* add test output dirs

* remove superfluous example_test

* prototype_sf

* prototype_arc

* prototype_marin

* move dirs

* psrc

* semcog

* sandag_xborder

* placeholder_sandag

* placeholder_multiple_zone

* no more coveralls

* repair docs

* clean up example names

* black and isort (#578)

* black and isort

* stop emitting output dir

it fails later tests

* trace files in nested directories

because windows

* swap files for xborder model

* repair ARC MP

* print checksum even when not used

* add hashes for sandag_xborder_full

* fix dtype in university hack

* fix persons to match tours

* repair ARC

* Bump version: 1.0.4 → 1.1.0

* Added mwcog small area

* missed one edit

* reformatting

* Update README.MD

* call as module

* github actions tests

* pre commit hooks

* pyproject toml

* limit numpy

* docs for mamba instead of conda

* ignore generated files

* add nbmake to test env

* fix dupe line

* repair test multiple zones for github actions

* publish develop docs

* fix docbuild env

* Update .travis.yml

* fixed testing files

* fixed testing files (again)

* fixed test script again

* publish docs to branch name

* updated testing scripts (note shorter travis script for now)

* fixed slash (windows vs. linux testing issue)

* added output folders

* updated travis script to run all tests, should pass

* docs cleaning

* docs re-style

* rebuild

* dynamic versioning docs

* version switcher

* blacken

* fix switcher url

* fix conf

* switcher update

* master to main

* deployment actions

* actions

* build wheel

* fix for testpypi

* blacken

* manual switcher

* branch docs service [makedocs]

* syntax [makedocs]

* travis depends

* checkout v3, fix versioning in docs

* only build develop docs once

* failsafe version

* documentation repairs

* python-simplified-semver

* front cards

* end testing w travis

* add mwcog test to gh-actions

* add mwcog to docs

* point to data not copy it

* sort dependencies

* req sh 2.2.4

* account for variance across platforms in trip dest est

* copy bike skims for sandag test

* param on rtol

* fix sandag_2 test files

* fix test file names

* added pre-processor option to trip scheduling

* trip scheduling relative mode initial commit

* moved everything to mwcog example

* adding output analysis notebook

* adding additional segmentation

* testing and documentation

* blacken

* not assuming scheduling mode is set

* fixing merge trip scheduling

* still fixing merge

* reverting regression trips

* fixing bad merge and updating test

Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>

* Trip Scheduling Bug & Trip Mode Choice Annotate (#55)

* other sp methods price updates

* Trip Scheduling into Resident Debug (#54)

* Differentiate examples between quality and validity of example models (#571)

* change examples that are no longer tied to an agency to fictional place names

* change name of full example also

* add back missing output dir

* restore test output dir also

* more empty dirs that got lost

* clean up docs

* example_mtc -> prototype_mtc

* Prototype MTC extended

* add all the ignored files

* add test output dirs

* remove superfluous example_test

* prototype_sf

* prototype_arc

* prototype_marin

* move dirs

* psrc

* semcog

* sandag_xborder

* placeholder_sandag

* placeholder_multiple_zone

* no more coveralls

* repair docs

* clean up example names

* black and isort (#578)

* black and isort

* stop emitting output dir

it fails later tests

* trace files in nested directories

because windows

* swap files for xborder model

* repair ARC MP

* print checksum even when not used

* add hashes for sandag_xborder_full

* fix dtype in university hack

* fix persons to match tours

* repair ARC

* Bump version: 1.0.4 → 1.1.0

* Added mwcog small area

* missed one edit

* reformatting

* Update README.MD

* call as module

* github actions tests

* pre commit hooks

* pyproject toml

* limit numpy

* docs for mamba instead of conda

* ignore generated files

* add nbmake to test env

* fix dupe line

* repair test multiple zones for github actions

* publish develop docs

* fix docbuild env

* Update .travis.yml

* fixed testing files

* fixed testing files (again)

* fixed test script again

* publish docs to branch name

* updated testing scripts (note shorter travis script for now)

* fixed slash (windows vs. linux testing issue)

* added output folders

* updated travis script to run all tests, should pass

* docs cleaning

* docs re-style

* rebuild

* dynamic versioning docs

* version switcher

* blacken

* fix switcher url

* fix conf

* switcher update

* master to main

* deployment actions

* actions

* build wheel

* fix for testpypi

* blacken

* manual switcher

* branch docs service [makedocs]

* syntax [makedocs]

* travis depends

* checkout v3, fix versioning in docs

* only build develop docs once

* failsafe version

* documentation repairs

* python-simplified-semver

* front cards

* end testing w travis

* add mwcog test to gh-actions

* add mwcog to docs

* point to data not copy it

* sort dependencies

* req sh 2.2.4

* account for variance across platforms in trip dest est

* copy bike skims for sandag test

* param on rtol

* fix sandag_2 test files

* fix test file names

* added pre-processor option to trip scheduling

* trip scheduling relative mode initial commit

* moved everything to mwcog example

* adding output analysis notebook

* adding additional segmentation

* testing and documentation

* blacken

* not assuming scheduling mode is set

* fixing merge trip scheduling

* still fixing merge

* reverting regression trips

* fixing bad merge and updating test

Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>

* adding annotate to trip mode choice

* no earliest change if previous trip fail

Co-authored-by: David Hensle <davidh@sandag.org>
Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>

* adding locals_dict to annotate

* chooser cols in final trips table

* BayDAG merge with ActivitySim v1.2  (#56)

* fixed logsums merge on households instead of persons

* code cleanup

* more cleanup

* documentation and passing tests

* passing tests

* fixed _accessibility suffix

* fixed conflict with persons_merged

* updated yaml to use simpler join method

* PEP formatting fixes

* refreshed example folder from develop

* added missing line at end of file

* black fixes

* black fixes to disaggregate accessibility changes

* passing tests

* ext cli arg

* memory sidecar

* restart if resume_after checkpoint is missing

* no pandas 1.5 yet

* stop the sidecar

* updated doc on shadow pricing

* 2nd Update model doc on shadow pricing

* more doc update on shadow pricing

* minor repairs

* skip when household_income is None

* fixed missing column pipeline error

* fixing pandas future warning

* blacken

* merged disagg accessibilities into mtc_extended and added doc section

* ran black on disaggregate_accessibility.py

* bug in trying to access shadow price settings when not running shadow pricing

* limiting pandas version

* updated dependencies

* always updating choices

* removing sklearn imports

* blacken

* add if none catch

* fixed None suffix default

* moved order of get_table(persons_merged) to avoid pulling prematurely in xborder

* testing removal of lognormal for hh vot

* putting hh vot back in

* tested and cleaned up rng channels

* setup injectable suffixes to allow add_size_table as model step not just function

* removed accessibility output from test

* re blacken py scripts

* updating to match sharrow test versions

* fixed tracing typo

* added variable index name suffix to pass optionally

* pipeline housekeeping to cleanup any tables, traceables, or channels that were touched during disagg

* added multiprocess testing

* blacken updates

* updated test scripts to include MP, problem with vehicle model in mp (even without disagg acc?)

* added improved origin sampling, resolved issue with merging with sample <100%. Need to address random seed for origins

* added sci-kit learn to test depends

* fixed person merging error causing pytest fail, uses inject method to ensure consistency

* cleanup comments

* fixed pytest to include accessibility table in regress

* setup for mp test, but needs debugging

* 'blacken'

* 'blacken'

* cleanup example folder

* fixed pipeline NoneType bug for disagg accessibility table

* raw person table for buffer instead of injectable

* add mp test for prototype_mtc_extended

* fixed pytest fail on mp, due to exept:True on mp_simulate

* 'blacken'

* added weighted k-means method

* created run script for multiple sampling scenarios

* blacken changes

* blacken

* fixed copy script

* blacken sampling script

* fixed n_zone when integer

* fixed typo

* Update random.py

* Updated test environment to reflect changes in pandas

* Removed random random seed generation from placeholder_sandage 1-zone test configuration to get example to pass automated test

* update sampling script

* 'blacken'

* fixed replacement sample bug

* adding segmentation, output by iteration, and external worker removal

* updated documentation

* more flexible scenario testing

* blacken

* Added test_random_seed.py

* blacken test_random_seed.py

* Moved random seed generation testing out of prototype_mtc folder and edited core_tests.yaml to call it

* Corrected name of check_outputs()

* Added print statements to debug script

* Blacken test_random_seed.py

* keep alternative attributes in parking location

* move coeff values to coeff csv

* Added pipe character to call that tests the random generator

* Added explicit call for pytest to run test_random_seed.py

* Added print statement at start of script for debugging

* Replaced single quotes with double quotes in print statement

* reformat with Black

* Moved random seed test into activitysim folder

* Edited core_tests.yml to reflect change in location of random seed test

* Renamed tests folder to test to reflect call in core-tests.yml

* formatting & documentation

* Added test_random_seed function for PyTest to read

* Fixed defition of seeds list and removed print statements

* updating github test env

* responding to review comments

* Added line to create output folder as that's not being committed

* create_rng_configs() now copies entire example configs directory instead of just settings.yaml

* Added try except statement when creating the output directory in case it already exists

* additional logging, random seed option, landuse sample weight column

* ensuring TAZ is not selected if no available MAZ

* adding logic to skip external location choice models

* Applied suggested changes

* Removed activitysim\test\random_seed\configs directory to reduce confusion

* Moved random seed test directory to  folder that had already existed

* temporarily constrain pandas #608

* work around psutil runtimeerror

* temp constrain pandas

* add mwcog full example

* review responses

* revisions per Sijia's comments, except for the proto-table-template implementation, still working.

* added template pop option

* blacken

* set back to create_tables to pass test against old tables

* updated proto-pop test data

* resolved bad test data issue

* warn don't fail on duplicate skims

* include sharrow in mwcog example

* cleanup

* zarr digital encoding for mwcog

* include reference in psrc mini

* fow window ids to int64 because windows

* handle expected warnings

* unpin dependencies

* columns cannot be a set

* unpin pandas

* columns cannot be a set

* numeric_only args in groupby.sum

* docs: correct the unit of `chunk_size`.

* change 'arry' to more readable 'sizearray'

* repair tests

* catch FileNotFound

* add check that land use is zero-based when using sharrow

* better docs on skims in shared memory

* better docstrings for skimdataset

* remove old code

* remove junk

* add logging for recoding

* optimize nearest zone determination

* consistent multiprocessing results

* blacken

* updating regression trips

* adding sandag 2-zone test

* blacken

* fix merge error

* allow pandas 1.5 again

* sharrow compatibility with #606

* remove outdated code in comments

* note on tracing limitation

* allow repeating key cols

* fix decoding in MP output

* escortee modes match chauf modes

* updating regression tests

* fixing bad var name in inbound expressions

* add note about `condense_parameters`

* change default condense_parameters to False

* Update .gitignore

Co-authored-by: Sijia Wang <wangsijia0628@gmail.com>

* ignore caches

* remove special install instructions

* remove sharrow_skip's

* Revert "remove sharrow_skip's"

This reverts commit 6b52af2.

* add note about why sharrow_skip is activated

* Update school_escorting.yaml

* Update config.py

* arc trip destination spec skim correction

* update regression trips

* require sharrow 2.5

* fix mtc_ext for sharrow

* fix test consistent with escort mode alignments

* allow sharrow skipping

* remove stray indent

* delete expression values after sharrow testing to make debugging easier

* require sharrow 2.5.2

* recode only once

* testing repairs

* note about why sharrow_skip

* fork mtc_extended testing on shadow pricing and sharrow

* note on sharrow_skip

* size term scaling only for full run

* reordering

* multiprocessing hangs!

* never scale size terms for disagg

* tally pending persons

* tests cleanup

* fix error with orca

* let pipeline tables be dropped

* check on available MAZs, comment cleanup

* work zone to -1 if wfh, removed unneeded extensions

* fix tests for sandag 2zone

* ignore deprecation warning in tests

not clear why this started being a problem suddenly in only the xborder-MP model

* add scikit-learn to formal dependencies

* docs update for installer

* fixing black to 22.12.0

* missed merge conflicts

* fixing merge issues in prototype_semcog

* fixing prototype_arc merge

* fixing placeholder_sandag merge issues

* blacken

* fixing mwcog merge issues

* skims not found during estimation test

---------

Co-authored-by: Nick Fournier <nick.fournier@rsginc.com>
Co-authored-by: Jeff Newman <jeffnewman@camsys.com>
Co-authored-by: aletzdy <58451076+aletzdy@users.noreply.github.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Co-authored-by: Nick Fournier <99497883+nick-fournier-rsg@users.noreply.github.com>
Co-authored-by: Joe Flood <joejimflood@gmail.com>
Co-authored-by: Sijia Wang <wangsijia0628@gmail.com>
Co-authored-by: amarin <17020181+asiripanich@users.noreply.github.com>

* do not require all alts in data file

* int8 to int16 lost in v1.2 merge

---------

Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>
Co-authored-by: Sijia Wang <wangsijia0628@gmail.com>
Co-authored-by: Ashish Kulshrestha <ashish.kulshrestha@wsp.com>
Co-authored-by: Jeffrey Newman <jeffnewman@camsys.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>
Co-authored-by: Nick Fournier <nick.fournier@rsginc.com>
Co-authored-by: Nick Fournier <99497883+nick-fournier-rsg@users.noreply.github.com>
Co-authored-by: aletzdy <58451076+aletzdy@users.noreply.github.com>
Co-authored-by: David Hensle <davidh@sandag.org>
Co-authored-by: JoeJimFlood <joejimflood@gmail.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Co-authored-by: amarin <17020181+asiripanich@users.noreply.github.com>

* docs: apply a minor correction to user guides (#659)

* Trip scheduling logic (#660)

* trip scheduling logic_version

* in testing ignore the warning we just added

* cannot use logic_version 1 with relative scheduling

* Pin Dependencies (#665)

* pinderella

* pandas in setup.cfg deps

---------

Co-authored-by: Bo Wen <bo.wen@translink.ca>
Co-authored-by: David Hensle <51132108+dhensle@users.noreply.github.com>
Co-authored-by: Lisa Zorn <lzorn@bayareametro.gov>
Co-authored-by: Sijia Wang <wangsijia0628@gmail.com>
Co-authored-by: Ashish Kulshrestha <ashish.kulshrestha@wsp.com>
Co-authored-by: Andrew Rohne <andrew@siliconcreek.net>
Co-authored-by: Nick Fournier <nick.fournier@rsginc.com>
Co-authored-by: Nick Fournier <99497883+nick-fournier-rsg@users.noreply.github.com>
Co-authored-by: aletzdy <58451076+aletzdy@users.noreply.github.com>
Co-authored-by: David Hensle <davidh@sandag.org>
Co-authored-by: JoeJimFlood <joejimflood@gmail.com>
Co-authored-by: amarin <17020181+asiripanich@users.noreply.github.com>
  • Loading branch information
13 people committed Apr 17, 2023
1 parent d0776af commit 6a57e12
Show file tree
Hide file tree
Showing 150 changed files with 9,230 additions and 144 deletions.
13 changes: 9 additions & 4 deletions HOW_TO_RELEASE.md
Expand Up @@ -42,15 +42,16 @@

00. Run black to ensure that the codebase passes all style checks.
This check should only take a few seconds. These checks are also done on
Travis and are platform independent, so they should not be necessary to
GitHub Actions and are platform independent, so they should not be necessary to
replicate locally, but are listed here for completeness.
```sh
black --check --diff .
```

00. Run the regular test suite on Windows. Travis tests are done on Linux,
but most users are on Windows, and the test suite should also be run
on Windows to ensure that it works on that platform as well. If you
00. Run the regular test suite on Windows. Most GitHub Actions tests are done on Linux,
Linux (it's faster to start up and run a new clean VM for testing) but most
users are on Windows, and the test suite should also be run on Windows to
ensure that it works on that platform as well. If you
are not preparing this release on Windows, you should be sure to run
at least through this step on a Windows machine before finalizing a
release.
Expand Down Expand Up @@ -135,6 +136,10 @@
```sh
gh release create v1.2.3
```
The process of creating and tagging a release will automatically
trigger various GitHub Actions scripts to build, test, and publish the
new release to PyPI and conda forge, assuming there are no errors.

For a development pre-release, include the `--prerelease` argument.
As the project's policy is that only formally released code is merged
to the main branch, any pre-release should also be built against a
Expand Down
1 change: 1 addition & 0 deletions activitysim/abm/models/__init__.py
Expand Up @@ -20,6 +20,7 @@
joint_tour_frequency,
joint_tour_participation,
joint_tour_scheduling,
joint_tour_frequency_composition,
location_choice,
mandatory_scheduling,
mandatory_tour_frequency,
Expand Down
71 changes: 59 additions & 12 deletions activitysim/abm/models/cdap.py
Expand Up @@ -78,6 +78,17 @@ def cdap_simulate(persons_merged, persons, households, chunk_size, trace_hh_id):
file_name=model_settings["FIXED_RELATIVE_PROPORTIONS_SPEC"]
)

add_joint_tour_utility = model_settings.get("ADD_JOINT_TOUR_UTILITY", False)

if add_joint_tour_utility:
# Rules and coefficients for generating cdap joint tour specs for different household sizes
joint_tour_coefficients_file_name = model_settings.get(
"JOINT_TOUR_COEFFICIENTS", "cdap_joint_tour_coefficients.csv"
)
cdap_joint_tour_coefficients = pd.read_csv(
config.config_file_path(joint_tour_coefficients_file_name), comment="#"
)

persons_merged = persons_merged.to_frame()

# add tour-based chunk_id so we can chunk all trips in tour together
Expand All @@ -101,11 +112,27 @@ def cdap_simulate(persons_merged, persons, households, chunk_size, trace_hh_id):
# (also when multiprocessing locutor might not see all household sizes)
logger.info("Pre-building cdap specs")
for hhsize in range(2, cdap.MAX_HHSIZE + 1):
spec = cdap.build_cdap_spec(cdap_interaction_coefficients, hhsize, cache=True)
spec = cdap.build_cdap_spec(
cdap_interaction_coefficients,
hhsize,
cache=True,
joint_tour_alt=add_joint_tour_utility,
)
if inject.get_injectable("locutor", False):
spec.to_csv(
config.output_file_path("cdap_spec_%s.csv" % hhsize), index=True
)
if add_joint_tour_utility:
# build cdap joint tour spec
# joint_spec_dependency = spec.loc[[c for c in spec.index if c.startswith(('M_p', 'N_p', 'H_p'))]]
joint_spec = cdap.build_cdap_joint_spec(
cdap_joint_tour_coefficients, hhsize, cache=True
)
if inject.get_injectable("locutor", False):
joint_spec.to_csv(
config.output_file_path("cdap_joint_spec_%s.csv" % hhsize),
index=True,
)

if estimator:
estimator.write_model_settings(model_settings, "cdap.yaml")
Expand All @@ -127,17 +154,32 @@ def cdap_simulate(persons_merged, persons, households, chunk_size, trace_hh_id):

logger.info("Running cdap_simulate with %d persons", len(persons_merged.index))

choices = cdap.run_cdap(
persons=persons_merged,
person_type_map=person_type_map,
cdap_indiv_spec=cdap_indiv_spec,
cdap_interaction_coefficients=cdap_interaction_coefficients,
cdap_fixed_relative_proportions=cdap_fixed_relative_proportions,
locals_d=constants,
chunk_size=chunk_size,
trace_hh_id=trace_hh_id,
trace_label=trace_label,
)
if add_joint_tour_utility:
choices, hh_joint = cdap.run_cdap(
persons=persons_merged,
person_type_map=person_type_map,
cdap_indiv_spec=cdap_indiv_spec,
cdap_interaction_coefficients=cdap_interaction_coefficients,
cdap_fixed_relative_proportions=cdap_fixed_relative_proportions,
locals_d=constants,
chunk_size=chunk_size,
trace_hh_id=trace_hh_id,
trace_label=trace_label,
add_joint_tour_utility=add_joint_tour_utility,
)
else:
choices = cdap.run_cdap(
persons=persons_merged,
person_type_map=person_type_map,
cdap_indiv_spec=cdap_indiv_spec,
cdap_interaction_coefficients=cdap_interaction_coefficients,
cdap_fixed_relative_proportions=cdap_fixed_relative_proportions,
locals_d=constants,
chunk_size=chunk_size,
trace_hh_id=trace_hh_id,
trace_label=trace_label,
add_joint_tour_utility=add_joint_tour_utility,
)

if estimator:
estimator.write_choices(choices)
Expand All @@ -161,6 +203,11 @@ def cdap_simulate(persons_merged, persons, households, chunk_size, trace_hh_id):

# - annotate households table
households = households.to_frame()

if add_joint_tour_utility:
hh_joint = hh_joint.reindex(households.index)
households["has_joint_tour"] = hh_joint

expressions.assign_columns(
df=households,
model_settings=model_settings.get("annotate_households"),
Expand Down
204 changes: 204 additions & 0 deletions activitysim/abm/models/joint_tour_frequency_composition.py
@@ -0,0 +1,204 @@
# ActivitySim
# See full license in LICENSE.txt.
import logging

import numpy as np
import pandas as pd
import os
from activitysim.core.interaction_simulate import interaction_simulate

from activitysim.core import simulate
from activitysim.core import tracing
from activitysim.core import pipeline
from activitysim.core import config
from activitysim.core import inject
from activitysim.core import expressions

from .util import estimation

from .util.overlap import hh_time_window_overlap
from .util.tour_frequency import process_joint_tours_frequency_composition

logger = logging.getLogger(__name__)


@inject.step()
def joint_tour_frequency_composition(
households_merged, persons, chunk_size, trace_hh_id
):
"""
This model predicts the frequency and composition of fully joint tours.
"""

trace_label = "joint_tour_frequency_composition"
model_settings_file_name = "joint_tour_frequency_composition.yaml"

model_settings = config.read_model_settings(model_settings_file_name)

alt_tdd = simulate.read_model_alts(
"joint_tour_frequency_composition_alternatives.csv", set_index="alt"
)

# - only interested in households with more than one cdap travel_active person and
# - at least one non-preschooler
households_merged = households_merged.to_frame()
choosers = households_merged[households_merged.participates_in_jtf_model].copy()

# - only interested in persons in choosers households
persons = persons.to_frame()
persons = persons[persons.household_id.isin(choosers.index)]

logger.info("Running %s with %d households", trace_label, len(choosers))

# alt preprocessor
alt_preprocessor_settings = model_settings.get("ALTS_PREPROCESSOR", None)
if alt_preprocessor_settings:

locals_dict = {}

alt_tdd = alt_tdd.copy()

expressions.assign_columns(
df=alt_tdd,
model_settings=alt_preprocessor_settings,
locals_dict=locals_dict,
trace_label=trace_label,
)

# - preprocessor
preprocessor_settings = model_settings.get("preprocessor", None)
if preprocessor_settings:

locals_dict = {
"persons": persons,
"hh_time_window_overlap": hh_time_window_overlap,
}

expressions.assign_columns(
df=choosers,
model_settings=preprocessor_settings,
locals_dict=locals_dict,
trace_label=trace_label,
)

estimator = estimation.manager.begin_estimation("joint_tour_frequency_composition")

model_spec = simulate.read_model_spec(file_name=model_settings["SPEC"])
coefficients_df = simulate.read_model_coefficients(model_settings)
model_spec = simulate.eval_coefficients(model_spec, coefficients_df, estimator)

constants = config.get_model_constants(model_settings)

if estimator:
estimator.write_spec(model_settings)
estimator.write_model_settings(model_settings, model_settings_file_name)
estimator.write_coefficients(coefficients_df, model_settings)
estimator.write_choosers(choosers)
estimator.write_alternatives(alts)

assert choosers.index.name == "household_id"
assert "household_id" not in choosers.columns
choosers["household_id"] = choosers.index

estimator.set_chooser_id(choosers.index.name)

# The choice value 'joint_tour_frequency_composition' assigned by interaction_simulate
# is the index value of the chosen alternative in the alternatives table.
choices = interaction_simulate(
choosers=choosers,
alternatives=alt_tdd,
spec=model_spec,
locals_d=constants,
chunk_size=chunk_size,
trace_label=trace_label,
trace_choice_name=trace_label,
estimator=estimator,
)

if estimator:
estimator.write_choices(choices)
choices = estimator.get_survey_values(
choices, "households", "joint_tour_frequency_composition"
)
estimator.write_override_choices(choices)
estimator.end_estimation()

# add joint tour frequency composition column to household table
households_merged["joint_tour_frequency_composition"] = choices.reindex(
households_merged.index
).fillna(0)

# - create joint_tours based on choices

# - we need a person_id in order to generate the tour index (and for register_traceable_table)
# - but we don't know the tour participants yet
# - so we arbitrarily choose the first person in the household
# - to be point person for the purpose of generating an index and setting origin
temp_point_persons = persons.loc[persons.PNUM == 1]
temp_point_persons["person_id"] = temp_point_persons.index
temp_point_persons = temp_point_persons.set_index("household_id")
temp_point_persons = temp_point_persons[["person_id", "home_zone_id"]]

# create a tours table of tour_category "joint" and different tour_types (e.g. shopping, eat)
# and add the composition column (adults or children or mixed) to the tour

# Choices
# hhid choice
# 11111 1
# 22222 2
# 33333 3

# Alts
# alt purpose1 purpose2 party1 party2 eat shop
# 1 5 0 3 0 1 0
# 2 5 6 1 3 1 1
# 3 6 0 1 0 0 1

# Joint Tours
# hhid type category composition
# 11111 eat joint mixed
# 22222 eat joint adults
# 22222 shop joint mixed
# 33333 shop joint adults

joint_tours = process_joint_tours_frequency_composition(
choices, alt_tdd, temp_point_persons
)

tours = pipeline.extend_table("tours", joint_tours)

tracing.register_traceable_table("tours", joint_tours)
pipeline.get_rn_generator().add_channel("tours", joint_tours)

# we expect there to be an alt with no tours - which we can use to backfill non-travelers
no_tours_alt = 0
households_merged["joint_tour_frequency_composition"] = (
choices.reindex(households_merged.index).fillna(no_tours_alt).astype(str)
)

households_merged["num_hh_joint_tours"] = (
joint_tours.groupby("household_id")
.size()
.reindex(households_merged.index)
.fillna(0)
.astype(np.int8)
)

pipeline.replace_table("households", households_merged)

tracing.print_summary(
"joint_tour_frequency_composition",
households_merged.joint_tour_frequency_composition,
value_counts=True,
)

if trace_hh_id:
tracing.trace_df(
households_merged, label="joint_tour_frequency_composition.households"
)

tracing.trace_df(
joint_tours,
label="joint_tour_frequency_composition.joint_tours",
slicer="household_id",
)

0 comments on commit 6a57e12

Please sign in to comment.