VE Validation UNDER CONSTRUCTION

Validation

This page summarizes additional detail on the validation of VisionEval models. This page is under construction.

More How

Adjustments can be made to specific modules within each package:

Essential...vs. fine tuning [using local PUMS is necessary, PowertrainsAndFuels is good practice, most others are fine-tuning] [can we put more meat in 7.1, classify by importance, show more detail on necessary, and have this be a drill-down detail on another page?] [Can we group these by the concepts, or by the bulleted types in 7.1? "package" organization are obtuse for basic user]

• The VESimHouseholds package processes PUMS data to derive parameters for several of its sub-modules. The default PUMS files in the inst/extdata folder from Oregon should be replaced with data for the area they are modeling. Some adjustments specific to simulated households by Azone include:

  • The average household size (AveHhSize) and proportion of one-person households (Prop1PerHh) can be set in the azone_hhsize_targets.csv file.
  • The relative employment rate by age group by Azone can be specified to match observed differences across a metropolitan area or levels at various points of economic cycles. These changes are coded in the optional azone_relative_employment.csv file. The relative employment rate is relative to the average employment rate for the worker group in the PUMS data (e.g., a value of 0.5 would be entered if the employment rate for 20-29 age group in one Azone was half the employment rate for persons in that age group in the metropolitan area).
  • The prediction of household income as a function of workers by age group and per captia income can be adjusted in the azone_per_cap_inc.csv file. Note that adjustments to the relative employment rate described above will influence affect the distribution of incomes across households.

• The VELandUse package also uses the PUMS data, which can be further adjusted in two ways:

  • The mix of single family versus multi-family households will reflect local patterns if PUMS data for the modeled region are used instead of the default data from Oregon.
  • The proportion of households residing in mixed-use neighborhoods within each Bzone can optionally be set in the bzone_urban-mixed-use_prop.csv file. Such adjustments are subjective, as the definition of "urban mixed-use" neighborhoods derived from the Claritas data in the NHTS are imprecise.

• Most of the parameters in the VEHouseholdVehicles package are self-calibrating. However, the relative driver licensing rate by age group can be coded in the region_hh_driver_adjust_prop.csv file.

• There are few opportunities to adjust the parameters in the VEHouseholdTravel package, as most are derived from the NHTS and PUMS input data. Note that the optionally specified driver licensing rates described above can substantially affect daily household VMT. It is possible to specify the percentage of household Short Trips SOV travel diverted to bicycles in the azone_prop_sov_dvmt_diverted.csv file to better match observed local values.

• The default data inputs in the VEPowertrainsAndFuels package substantially affect modeled fuel consumption and vehicle emissions rates. These default inputs are contained in the inst/extdata folder of the source package. Note that the package needs to be built (installed) from the source package after adjustments have been made in order for the changes to have effect.

• The VETravelPerformance package is self-calibrating. However, the user must provide several estimates used as constraints during that process:

  • Estimates of urbanized area light-duty vehicle and heavy truck VMT (UrbanLdvDvmt and UrbanHvyTrkDvmt, respectively) must be coded in the marea_base_year_dvmt.csv input file.
  • The user must also provide a regional estimate of heavy truck VMT (HvyTrkDvmt) in the region_base_year_dvmt.csv that is consistent with the urbanized area heavy truck VMT estimates.
  • The user should check the basis used for estimating commercial service VMT (ComSvcDvmtGrowthBasis) and heavy truck VMT (HvyTrkDvmtGrowthBasis).

Key Considerations

[Great to have this in 1 place! Group by Concept?]

The experience to date suggest that several model inputs and assumptions can be changed to better match observed local patterns and trends:

• The choice of geographies used in VisionEval can influence validation results. For example, if economic conditions or driver licensing rates vary significantly across the modeled area it might be a good idea to define Azones to reflect those differences.
• PUMS data for the modeled region should be used in place of the Oregon data used in the original development of the model. This is done by simply replacing the PUMS data in the inst/extdata folder with local data, which are then processed by several modules as part of a normal model run. [I think you have to rebuild the package, need to show how or have FHWA do this for you, keep options available for download]
• Care should be taken in choosing the validation targets to match model predictions. VE household travel modules predict household travel regardless of where the travel occurs within the modeled region. Therefore the results should be compared with household survey data which provide similarly defined estimates.
• Care should similarly be taken in comparing the model results with daily roadway VMT data, such as those reported in Highway Statistics reports or imputed from annual HPMS estimates. If such comparisons are made the model estimates of roadway DVMT calculated by the CalculateRoadDvmt module should be used for comparison.[rather than HPMS data that is defined as all vehicles but only those miles on roadways within a specific geography.]
• If modeled DVMT matches validation data but fuel consumption does not (e.g. light-duty vehicle fuel consumption reported in Highway Statistics), the values in ldv_powertrain_characteristics.csv for current and past years can be adjusted to achieve a match. Fuel consumption may not match for several reasons such as:
  • The definition of crossover vehicles as light trucks vs. autos used in the vehicle type model does not match the average fuel consumption characteristics in the ldv_powertrain_characteristics.csv file.
  • The average fuel consumption characteristics in ldv_powertrain_characteristics.csv do not represent real world fuel economy.
• If validation data are available for commercial service vehicles, heavy trucks, and public transit vehicles changes can be made to the respective powertrain characteristics files for those vehicle types to match observed values.
• If the modeled road DVMT estimates match for the base year but not for prior years the modeled household DVMT trends should be checked against road DVMT trends. In particular, check whether reduction of DVMT (or reduction of DVMT growth rate) observed during the Great Recession and increase in DVMT (or increase in the DVMT growth rate) afterwards are reflected in the modeled household DVMT. If not, the values in the following files should be checked and adjusted if warranted:
  • Young persons that moved back in with parents during recession and left afterwards when earning more should be accounted for in azone_hhsize_targets.csv
  • The lower incomes during the recession should be reflected in the azone_per_cap_inc.csv file
  • Lower employment rates, especially in certain age groups, should be reflected in azone_relative_employment.csv
  • The reduction in driver licensing rates among young drivers should be accounted for in the region_hh_driver_adjust_prop.csv file