IDS Validation Efficiency

[Elobo68]

Hi everyone,

I am trying to create an IDS in order to explain and ease data validation for my model.
It's seems quite slow.

I have a PBS (a classification) in some parameters, with a hierachy.
I have a huge excel file, that explain this hierachy, the IDS aim to translate that PBS in a file that can control it's application.

Explanation of the PBS
Domaine = GXX (G11, G12, etc...)
Specialite = It depends of the "Domaine".
G14-600 is a valid value and G14-599 is invalid, because it's not in my main Excel.

Domaine and Specialite are linked by my main Excel file, but i have two parameters.

Building my IDS
First Specification
Do you have a domaine ? If not, i will not try to test further.

My applicability, is based on tag. In that way, i can only select, IfcElement.

G11 Specification
If you have G14, you are only allowed to have an enumeration in the Specialite
...
...

G99 Specification
If you have G99, you are only allowed to have an enumeration in the Specialite

My question:

It seems that my script is slow and i would like to understand if it's a bad creation of the IDS or a normal way of working.

It tooks 19 seconds to check.
It's only a small part of my check (G1X, not the whole)
It's my smallest model (10Mo), and 820 objects to check.

I put my IDS here.

SampleIDS.zip

My script is quite easy.

Thanks for you light !
And thank you for your amazing IfcOpenShell

[aothms]

I haven't ran the IDS myself, if you can share a model that might help.

But I do see this:

            <applicability>
                <attribute>
                    <name>
                        <simpleValue>Tag</simpleValue>
                    </name>
                </attribute>

Which relates this line comment in the code:

            # Usually, we rely on an entity applicability to give us our first
            # shortlist of elements, as it's the most efficient way to filter
            # elements. If this does not exist, then we have no choice but to
            # check everything.

https://github.com/IfcOpenShell/IfcOpenShell/blob/v0.7.0/src/ifctester/ifctester/ids.py#L206-L212

So yes, your IDS will check quite a bit slower than typical IDS'es because we have no efficient mechanism implemented to filter out the relevant instances.

But it is possible to find the entities that carry a certain attribute by which we could apply the same pre-filtering we have with entity facets. See below:

import ifcopenshell
w = ifcopenshell.ifcopenshell_wrapper
s = w.schema_by_name('IFC4')
def get_entities_with_attribute(name)
  for ent in s.entities(): 
    if any(a.name() == name for a in ent.attributes()):
      yield ent.name()

Are you interested perhaps in tying these two things together in ids.py, with the help from the community if needed?

[Elobo68]

Hi,

Sadly, i can not share the models with the whole planet. I will make a dummy one, with the parameter i need to check in order to share it.

It's already better, but i will try to improve it further.
I did it by hand, but i thinkg the sprirt is here.

I gain 9s, over the checking part.

Next week, i will have some time, to try a proper implementation in the ids.py.

Edit: In the IDS, it could be cool, to only specify IfcElement or IfcBuildingElement, in order to select all the supertype (IfcWall+IfcSlab...)

[aothms]

Edit: In the IDS, it could be cool, to only specify IfcElement or IfcBuildingElement, in order to select all the supertype (IfcWall+IfcSlab...)

See buildingSMART/IDS#146

[Elobo68]

Hi, i have made a few research

I got a few solution:

First One: Optimise Property Facet
I found the bottleneck here, but i haven't understand all part of the code.

With the IfcOpenshell, 0.7, there is a new function to fetch Pset.
It's way faster, but i am wondering if there is a bug in it.

ifcopenshell.util.element.get_pset() is way faster than the one looking for all the pset.

import ifcopenshell
import datetime

my_ifc = ifcopenshell.open("HTBT.ifc")

products = my_ifc.by_type("IfcElement")

Un_Element=products[0]
The_Pset_Name="Contraintes" #I want to check on that Pset, it's a random one

#Method to get all Pset, and only select Texte
t0_AllPset=datetime.datetime.now()
all_Psets=ifcopenshell.util.element.get_psets(Un_Element)
One_Psets=all_Psets[The_Pset_Name]
t1_AllPset=datetime.datetime.now()

#The Faset Method
t0_UnPset=datetime.datetime.now()
GET_PSET=ifcopenshell.util.element.get_pset(Un_Element,name=The_Pset_Name)
t1_UnPset=datetime.datetime.now()


Temp_AllPSet=t1_AllPset-t0_AllPset
Temp_UnPset=t1_UnPset-t0_UnPset


print("Fonction ALl Pset:",Temp_AllPSet,One_Psets)
print("Fonction Get_Pset:",Temp_UnPset,GET_PSET)
print(One_Psets==GET_PSET)`

i have try with other Pset, and with some other one, i don't have all the properties.
I doubt that it's an intended feature.
Shouldn't i get the same response with my both method ?

Second Solution
I can check redundancies with the facet.
We can start the IDS analysing by checking if some facet are used several times, if so we can calculated all the facet at the beginning, only one.
Then, i can check my IDS, with the already calculated one facet.

I still haven't started that one, i will first try to check if i fund redundancies in IDS facet.

Third Solution
I will try to check the fastest Facet, and i will always try the fastest first. In that way, i will reduce the slowest one.

1. Entity
2. Attribute
3. Classification, Material, Partof
4. Property

i need to check if this assumption is good and if that king of hierarchy exist

Elobo

[Moult]

Thank you so much for your investigation. You are right there was a bug in get_pset which now has been fixed 64516a1 and tests added.

I've also now implemented more efficient filtering for all facets, not just entity: c040268 which takes into account your findings and recommendations. Hopefully this should speed things up!