Error when running matsim's emission contrib #213
Comments
|
Mmmmm, yeah it looks weird. Can you share your modified |
|
Thanks! @syhwawa will share :) I'm wondering about turning off the link type functionality all together (speed and vehicle type being far more dominant factors) |
|
With some effort it is definitely possible. I'm not sure it is negligible though, as it includes the max speed and it is probably important. |
|
Hi, thanks for your reply. This is the modified the package org.eqasim.ile_de_france.emissions;
// this is a modified copy of a private contribs.emissions class
// TODO: will need to be updated after the matsim 14 release to profit from https://github.com/matsim-org/matsim-libs/pull/1859
import java.util.HashMap;
import java.util.Map;
import org.matsim.api.core.v01.network.Link;
import com.google.inject.Provides;
import org.matsim.api.core.v01.network.Network;
import org.matsim.contrib.emissions.EmissionUtils;
/**
* Created by molloyj on 01.12.2017.
*
*
* handled OSM road types:
* motorway,trunk,primary,secondary, tertiary, unclassified,residential,service
* motorway_link, trunk_link,primary_link, secondary_link
* tertiary_link, living_street, pedestrian,track,road
*
* Hbefa categories and respective speeds
* URB/MW-Nat./80 - 130
* URB/MW-City/60 - 110
* URB/Trunk-Nat./70 - 110
* URB/Trunk-City/50 - 90
* URB/Distr/50 - 80
* URB/Local/50 - 60
* URB/Access/30 - 50
*
* Conversions from OSM to hbefa types
* motorway;MW
* primary;Trunk
* secondary;Distr
* tertiary;Local
* residential;Access
* living;Access
*/
public class OsmHbefaMapping {
private static final int MAX_SPEED = 130;
private static final String OSM_HIGHWAY_TAG = "osm:way:highway";
Map<String, Hbefa> hbfeaMap = new HashMap<>();
public static class Hbefa {
String name;
int min;
int max;
public Hbefa(String name, int min, int max) {
this.name = name;
this.min = min;
this.max = max;
}
}
public void addHbefaMappings(Network network) {
for (Link link : network.getLinks().values()) {
String hbefaString = determineHebfaType(link);
if (hbefaString != null) {
EmissionUtils.setHbefaRoadType(link, hbefaString);
}
}
}
@Provides
public static OsmHbefaMapping build() {
OsmHbefaMapping mapping = new OsmHbefaMapping();
mapping.put("motorway-Nat.", new Hbefa("MW-Nat.",80,130));
mapping.put("motorway", new Hbefa("MW-City",60,90));
mapping.put("primary-Nat.", new Hbefa("Trunk-Nat.",80,110));
mapping.put("primary", new Hbefa("Trunk-City",50,80));
mapping.put("trunk", new Hbefa("Trunk-City",50,80));
mapping.put("secondary", new Hbefa("Distr",50,80));
mapping.put("tertiary", new Hbefa("Local",50,60));
mapping.put("residential", new Hbefa("Access",30,50));
mapping.put("service", new Hbefa("Access",30,50));
mapping.put("living", new Hbefa("Access",30,50));
mapping.put("pedestrian", new Hbefa("Access",30,50));
mapping.put("footway", new Hbefa("Access",30,50));
mapping.put("construction", new Hbefa("Access",30,50));
mapping.put("busway", new Hbefa("Access",30,50));
mapping.put("cycleway", new Hbefa("Access",30,50));
return mapping;
}
private void put(String s, Hbefa hbefa) {
hbfeaMap.put(s, hbefa);
}
public String determineHebfaType(Link link) {
String roadType = (String) link.getAttributes().getAttribute(OSM_HIGHWAY_TAG);
String hbefaType = null;
if (roadType != null) {
hbefaType = getHEBFAtype(roadType,link.getFreespeed());
}
return hbefaType;
}
private String getHEBFAtype(String roadType, double freeVelocity) {
String[] ss = roadType.split("_"); //want to remove
String type = ss[0];
//TODO: could make distinction between national and city, based on shapefile, or regions.
double freeVelocity_kmh = freeVelocity * 3.6;
if (type.equals("unclassified") || type.equals("road")) {
if (freeVelocity_kmh <= 50) type = "living";
else if (freeVelocity_kmh == 60) type = "tertiary";
else if (freeVelocity_kmh == 70) type = "secondary";
else if (freeVelocity_kmh <= 90) type = "primary";
else type = "motorway";
}
//specify that if speed > 90 and primary or motorway, then Nat.
if (type.equals("motorway") || type.equals("primary") && freeVelocity_kmh >= 90) {
type += "-Nat.";
}
if (hbfeaMap.get(type) == null) {
// return null;
throw new RuntimeException("'"+ type +"' not in hbefa map");
}
int min_speed = hbfeaMap.get(type).min;
int max_speed = hbfeaMap.get(type).max;
int capped_speed = (int) Math.min(Math.max(min_speed, freeVelocity_kmh), max_speed);
return "URB/" + hbfeaMap.get(type).name + "/" + capped_speed;
}
}This is the missing road type in the network we have added: mapping.put("pedestrian", new Hbefa("Access",30,50));
mapping.put("footway", new Hbefa("Access",30,50));
mapping.put("construction", new Hbefa("Access",30,50));
mapping.put("busway", new Hbefa("Access",30,50));
mapping.put("cycleway", new Hbefa("Access",30,50)); |
|
Ok looks good to me, with your current code, all links should have a call to so private void checkNetworkConsistency() {
var allMatch = scenario.getNetwork().getLinks().values().parallelStream()
.map(link -> link.getAttributes().getAttribute(EmissionUtils.HBEFA_ROAD_TYPE))
.allMatch(roadType -> roadType instanceof String);
if (!allMatch)
throw new RuntimeException("The Emission Contrib expects HbefaRoadTypes to be set as link attributes. " +
"Use EmissionUtils.setHbefaRoadType(link, type) to set HbefaRoadTypes on each link. " +
"Alternatively use an existing mapper such as OsmHbefaMapping, VisumHbefaRoadTypeMapping or VspHbefaRoadTypeMapping to set HbefaRoadTypes on your network.");
}Can you find the links where the That would give us a hint I think. |
|
In the mean time, try to add this to your OsmHbefaMapping osmHbefaMapping = OsmHbefaMapping.build();
Network network = scenario.getNetwork();
// if the network is from pt2matsim it might not have "type" but "osm:way:highway" attribute instead
for (Link link: network.getLinks().values()) {
String roadTypeAttribute = NetworkUtils.getType(link);
String osmRoadTypeAttribute = (String) link.getAttributes().getAttribute("osm:way:highway");
if (StringUtils.isBlank(roadTypeAttribute)) {
if (!StringUtils.isBlank(osmRoadTypeAttribute)) {
NetworkUtils.setType(link, osmRoadTypeAttribute);
}
else { // not a road (railway for example)
NetworkUtils.setType(link, "unclassified");
}
}
// '_link' types are not defined in the OSM mapping, set t undefined
if (NetworkUtils.getType(link).contains("_link")) {
NetworkUtils.setType(link, "unclassified");
}
if (NetworkUtils.getType(link).equals("living_street")) {
NetworkUtils.setType(link, "living");
}
}
osmHbefaMapping.addHbefaMappings(network); |
|
Thanks for your reply. @Nitnelav Yes, I believe the error was due to the network side. When I checked the network file This attribute should be created after running the simulation pipeline? I found the example network file in matsim-lib has this attribute in network(link). Do you think it could relate the OSM data update for IDF in this PR? |
|
Ah yes the attribute is created in memory but is never written to a file, you need to export the network with a NetworkWriter or something like that, after the |
Hi, I gave a try by adding these codes but still encountered the same error: Here is my submitting command: java -Xmx16G -cp /mnt/efs/eqasim-java/ile_de_france/target/ile_de_france-1.3.1.jar:/mnt/efs/ile-de-france/output_vehicles/ile_de_france_run.jar: org.eqasim.ile_de_france.emissions.RunComputeEmissionsEvents --config-path /mnt/efs/ile-de-france/output_vehicles/ile_de_france_config.xml --hbefa-cold-avg /mnt/efs/ile-de-france/data/HBEFA/CML_HBEFA_Factors/EFA_ColdStart_Vehcat_2015_Cold_Avergae.csv --hbefa-hot-avg /mnt/efs/ile-de-france/data/HBEFA/CML_HBEFA_Factors/EFA_HOT_Vehcat_2015_Hot_Average.csv --hbefa-cold-detailed /mnt/efs/ile-de-france/data/HBEFA/CML_HBEFA_Factors/EFA_ColdStart_Subsegm_2015_Cold_Detailed.csv --hbefa-hot-detailed /mnt/efs/ile-de-france/data/HBEFA/CML_HBEFA_Factors/Combined_EFA_HOT_Subsegm_2015_Hot_Detialed.csv
|
Hey @Nitnelav , I gave a try to add osmHbefaMapping.addHbefaMappings(network);
String networkOutputFile = scenario.getConfig().controler().getOutputDirectory() + "output_network_with_emissions_test.xml";
NetworkWriter networkWriter = new NetworkWriter(network);
networkWriter.write(networkOutputFile);It failed with the same error as expected and it does output a network file. The exported the networ has the attribute for the <links capperiod="01:00:00" effectivecellsize="7.5" effectivelanewidth="3.75">
<link id="1" from="1174895929" to="1174895898" length="136.97395144641263" freespeed="8.333333333333334" capacity="600.0" permlanes="1.0" oneway="1" modes="car,car_passenger" >
<attributes>
<attribute name="hbefa_road_type" class="java.lang.String">URB/Access/30</attribute>
<attribute name="osm:way:highway" class="java.lang.String">residential</attribute>
<attribute name="osm:way:id" class="java.lang.Long">101767911</attribute>
<attribute name="osm:way:name" class="java.lang.String">Rue Jean de la Chaize</attribute>
<attribute name="type" class="java.lang.String">residential</attribute>
</attributes>
</link> |
|
Ok perfect, can you find the links where the attribute is missing ? |
|
Yeah, I can see from the created network, a lot of links miss the Is it because the method used to add hbefa_road_type attributes might have been designed or configured to only process certain types of links? I have attached an example below: <link id="pt_IDFM:8963" from="pt_IDFM:8963" to="pt_IDFM:8963" length="85.0" freespeed="Infinity" capacity="9999.0" permlanes="1.0" oneway="1" modes="artificial,stopFacilityLink,bus" >
<attributes>
<attribute name="type" class="java.lang.String">unclassified</attribute>
</attributes>
</link>
<link id="pt_IDFM:8963_374554" from="pt_IDFM:8963" to="1206327421" length="436.8431952802874" freespeed="Infinity" capacity="9999.0" permlanes="1.0" oneway="1" modes="bus,artificial" >
<attributes>
<attribute name="type" class="java.lang.String">unclassified</attribute>
</attributes>
</link>
<link id="pt_IDFM:9312" from="pt_IDFM:9312" to="pt_IDFM:9312" length="20.0" freespeed="20.0" capacity="9999.0" permlanes="1.0" oneway="1" modes="artificial,stopFacilityLink,bus" >
<attributes>
<attribute name="type" class="java.lang.String">unclassified</attribute>
</attributes>
</link>
|
|
mmm still weird, those links are here from the beginning and they never raised any issues... did you update the dependencies to Matsim 15 at some point ? You can try using the code from the matsim repo https://github.com/matsim-org/matsim-libs/blob/master/contribs/emissions/src/main/java/org/matsim/contrib/emissions/OsmHbefaMapping.java |
|
Hi @Nitnelav, thanks for your advice. I believe we are on the matsim version 14.0. We have replaced the Here is the latest error that we met: We are thinking the error is due to the format of our HBEFA dataset. The values we have in the Here are the top five rows of one of the HBEFA inputs According to our understanding, the value I have pasted the HBEFA dataset format what we got: I have also attached the images when we required the HBEFA data: Could you @Nitnelav @neda-git please let us know if our hbefa csv files are like your working dataset or you can see anything weird? Or maybe if you did some post-processing on the HBEFA data to make them compatible with the code? Much appreciate your help and time in advance! |
|
Ok I don't know, at this point I would use the debugger and start placing breakpoints everywhere. I know I did not post process the HBEFA data that's for sure. For the rest it is too much guesswork here. It is tedious to debug a situation through a gihub issues :) |
|
Also, Why did you separate the Passager car from the rest in the detailed export ? |
|
You can find my file headers here : #159 (comment) |
Thanks for your reply. @Nitnelav, Yeah. My colleague helped us to prepare the HBEFA dataset since he had the licence, when he prepared the detailed hot emissions factors in the format of all of the combinations, the outputs were too huge to export so he divided it into two separate files. And I combined these two datasets into one later. I believe it doesn't matter if HBEFA data were extracted as
Thanks for the sharing links. I have compared the header of each table that we had are same compared to the outputs you posted #159 (comment) In terms of the parameters, I am wondering if these extra pollutants in the components part that we required from HBEFA led to the issue. |
|
Hey @Nitnelav, could you please let me know that did you run the emission contribs based on the "actual" matsim outputs instead of test outputs after running the IDF simulation pipelines? I found the it needs the read the The IDF documentation describe after running, you should find the MATSim scenario files in the output folder which doesn't include the any output event file. |
|
Yes I did run it on actual Matsim scenarios, one in Nantes and one in Lyon. The You know you need to launch the Matsim simulation after the Eqasim generated the files right ? https://github.com/eqasim-org/ile-de-france/blob/fb1112d2a7d1817746be84413da584c391059ad1/docs/simulation.md?plain=1#L86C1-L94C33 |
|
@Nitnelav Thanks for your reply. We have tried to use actual IDF simulation outputs to test the emission contribs. I believe the tool can actually run some emission analysis now from what I can tell via the latest error message below: It seems to me the error was induced by some missing values from the provided HBEFA dataset. Our running command is below: Currently, we are using two jar files
When we tried to use the first jar only after complying with the code in eqasim-java repo, it gave us some missing dependencies error. As you suggested, we're trying to use the IDF debugger for the error. Could you please tell us how would you create the jar file after modifying the compute emission script? Thanks. |
|
Well, I don't know, I confused, may be it's because it's Friday and I'm tiered but there is too much unclear informations in your previous message. If you could formulate a clear question or issue we can focus on instead of sending all your console errors here until it works I would appreciate it :) Also, don't hesitate to close issues once you find a solution and open a new one as the next issue arrises. It is going to be easier to read for anyone who is trying to follow these discussions. Finally, it seems like you could benefit from a few tutorials on using Java (using the IntelliJ IDE to debug, create jars etc.). I would suggest you spend a bit of time on that before going further. |
Hi, we are attempting to run the MATSim emissions contribution for the Île-de-France region. We have successfully completed the full simulation and now wish to utilize certain classes from eqasim-java to compute emissions using the MATSim outputs.
However, we encountered an error related to a network consistency check, with the error message as follows
We believe the issue may be due to recent changes in the OSM network update for the IDF, which resulted in inconsistencies with the mapping function.
When I comment out this line
The error message printed out some new road types are not in the hbefa map:
We have incorporated these new road types into OsmHbefaMapping.java. However, when we reran the test, the same error occurred.
Below is our HBEFA dataset, but we now think that the problem is not related to the HBEFA data. The header is the same as the one I found in this source (resolved github issue)
EFA_ColdStart_Vehcat_2015_Cold_Avergae.csv
Case,VehCat,Year,TrafficScenario,Component,RoadCat,AmbientCondPattern,EFA_weighted,EFA_km_weighted,EFA_WTT_weighted,EFA_WTT_km_weighted,EFA_WTW_weighted,EFA_WTW_km_weightedEFA_HOT_Vehcat_2015_Hot_Average.csv
Case,VehCat,Year,TrafficScenario,Component,RoadCat,TrafficSit,Gradient,V_weighted,EFA_weighted,AmbientCondPattern,EFA_WTT_weighted,EFA_WTW_weightedEFA_ColdStart_Subsegm_2015_Cold_Detailed.csv
Case,VehCat,Year,TrafficScenario,Component,RoadCat,AmbientCondPattern,IDSubsegment,Subsegment,Technology,SizeClasse,EmConcept,KM,%OfSubsegment,EFA,EFA_weighted,EFA_km,EFA_km_weighted,EFA_WTT,EFA_WTT_km,EFA_WTW,EFA_WTW_kmEFA_HOT_Subsegm_2015_Hot_Detialed_PC.csv
Case,VehCat,Year,TrafficScenario,Component,RoadCat,TrafficSit,Gradient,IDSubsegment,Subsegment,Technology,SizeClasse,EmConcept,KM,%OfSubsegment,V,V_0%,V_100%,EFA,EFA_0%,EFA_100%,V_weighted,V_weighted_0%,V_weighted_100%,EFA_weighted,EFA_weighted_0%,EFA_weighted_100%,AmbientCondPattern,EFA_WTT,EFA_WTT_0%,EFA_WTT_100%,EFA_WTW,EFA_WTW_0%,EFA_WTW_100%Much appreciate if you have any comments or ideas! @Nitnelav
The text was updated successfully, but these errors were encountered: