STIX Custom Object issue with references

[priamai]

Hi there,
I am doing some practice to create custom objects to model additional concepts such as a firewall and its actions such as drop/pass to a network object.

This is a very bare bone starting point:

"""The classes found here are how Priam specific objects can be represented as custom STIX objects instead of python dictionaries."""
'''
Inspired by this: https://github.com/mitre-attack/mitreattack-python/tree/master/mitreattack
And this approach: https://schema.ocsf.io/

'''
from stix2 import CustomObject, ExternalReference
from stix2.properties import (
    StringProperty,
    ListProperty,
    TypeProperty,
    IDProperty,
    ReferenceProperty,
    TimestampProperty,
    BooleanProperty,
    IntegerProperty,
    FloatProperty,
    EnumProperty
)


class CustomStixObject(object):
    """Custom STIX object used for PRIAM objects."""

    def get_version(self) -> str:
        """Get the version of the object.

        Returns
        -------
        str
            the object version
        """
        return self.x_priam_version


def StixObjectFactory(data: dict) -> object:
    """Convert STIX 2 content into a STIX object (factory method).

    Parameters
    ----------
    data : dict
        the STIX 2 object content to instantiate, typically
        the result of a stix2 query

    Returns
    -------
    stix2.CustomObject | stix2.v20.sdo._DomainObject
        an instantiated Python STIX object
    """
    stix_type_to_custom_class = {
        "x-priam-sensor": Sensor
    }

    if "type" in data and data["type"] in stix_type_to_custom_class:
        return stix_type_to_custom_class[data["type"]](**data, allow_custom=True)
    return data

@CustomObject(
    "x-priam-sensor",
    [
        # SDO Common Properties
        ("id", IDProperty("x-priam-sensor", spec_version="2.1")),
        ("type", TypeProperty("x-priam-sensor", spec_version="2.1")),
        ("created_by_ref", ReferenceProperty(valid_types="identity", spec_version="2.1")),
        ("created", TimestampProperty(precision="millisecond")),
        ("modified", TimestampProperty(precision="millisecond")),
        ("revoked", BooleanProperty(default=lambda: False)),
        ("name", StringProperty(required=True)),
        ("description", StringProperty()),
        ("x_priam_version", StringProperty()),
        ("fqdn", StringProperty())
    ],
)
class Sensor(CustomStixObject, object):
    """Custom Sensor object of type stix2.CustomObject.

    Custom Properties
    -----------------
    fqdn: str
    """

    pass

@CustomObject(
    "x-priam-action",
    [
        # SDO Common Properties
        ("id", IDProperty("x-priam-action", spec_version="2.1")),
        ("type", TypeProperty("x-priam-action", spec_version="2.1")),
        ("created", TimestampProperty(precision="millisecond")),
        ("modified", TimestampProperty(precision="millisecond")),
        ("revoked", BooleanProperty(default=lambda: False)),
        ("action",EnumProperty(allowed=['allow','block'])),
        ("severity",IntegerProperty(min=0,max=10)),
        ("x_priam_version", StringProperty()),
        # Sensor References
        ("sensor_refs", ListProperty(ReferenceProperty(valid_types="x-priam-sensor"))),
    ],
)
class Action(CustomStixObject, object):
    """Custom Sensor object of type stix2.CustomObject.

    Custom Properties
    -----------------
    fqdn: str
    """

    pass

I then try the following:

    def test_sensor_action(self):

        sensor = priamstix.Sensor(name='edr', description='an EDR sensor', fqdn='falcon.contoso.com')
        action = priamstix.Action(action='allow',sensor_refs=[sensor.id],object_marking_refs=TLP_AMBER)
        print(action.serialize(indent=4))

But I am getting this error:

self = <[AttributeError("'Action' object has no attribute '_inner'") raised in repr()] Action object at 0x7f786b52eeb0>
prop_name = 'sensor_refs'
prop = <stix2.properties.ListProperty object at 0x7f786b5921f0>
kwargs = {'action': 'allow', 'id': 'x-priam-action--c198f0de-de48-4206-aa6b-7e49f755c424', 'revoked': False, 'sensor_refs': ['x-priam-sensor--1d43338c-d60a-4a63-b9c0-e3981c074284'], ...}
allow_custom = False

    def _check_property(self, prop_name, prop, kwargs, allow_custom):
        if prop_name not in kwargs:
            if hasattr(prop, 'default'):
                value = prop.default()
                if value == NOW:
                    value = self.__now
                kwargs[prop_name] = value
    
        has_custom = False
        if prop_name in kwargs:
            try:
>               kwargs[prop_name], has_custom = prop.clean(
                    kwargs[prop_name], allow_custom,
                )

../venv/lib/python3.8/site-packages/stix2/base.py:50: 

Is this happening because I am not allowed to create list of references to custom objects?

[chisholm]

It will help if you post code that's runnable as-is. After "fixing" it with regard to imports and some other misc things, I get output:

{
    "type": "x-priam-action",
    "spec_version": "2.1",
    "id": "x-priam-action--70f0a1fd-00a4-45f2-8fef-246ddaf8b1c9",
    "action": "allow",
    "sensor_refs": [
        "x-priam-sensor--73076d2d-c193-4e98-8793-f1dac7966891"
    ],
    "object_marking_refs": [
        "marking-definition--f88d31f6-486f-44da-b317-01333bde0b82"
    ]
}

So, it works. I tried it with stix2 3.0.1 from pypi.

Some notes though:

• Your "factory" function is unnecessary: it duplicates functionality the stix2 library already has. One of the things the @CustomObject decorator does is register your custom class with the library's internal registry. You don't need to make another registry yourself. After having registered it, if you want to look up the class according to a STIX type, you can use the stix2.registry.class_for_type() function.
• You may not need to look up the class anyway. If you have a string, dict, or file-like object, you can use the stix2.parse() function to look up the class and instantiate it for you.
• Don't define common properties in your custom classes (e.g. "id", "type", "created", "modified", etc). The decorator defines many of them for you, and it may even override some of yours (and vice versa).
• "Custom" STIX objects, properties, etc are deprecated in STIX 2.1. I encourage you to use the current STIX mechanism for extending the data model, which is the extension. To do this, you create an extension-definition object, and then refer to it in your extensions.

Here is a sample implementing a part of your code using the extension mechanism:

import stix2
from stix2.properties import StringProperty
from stix2.v21.vocab import EXTENSION_TYPE_NEW_SDO
import sys


# Those who use your extension definition will use its ID to refer to it, so
# maybe it is better to fix this ID here instead of randomly regenerating it
# in each run of this script.
PRIAM_SENSOR_EXT_ID="extension-definition--638dd8ff-083a-4e42-adba-702080ceb4a0"


my_ext = stix2.ExtensionDefinition(
    id=PRIAM_SENSOR_EXT_ID,
    created_by_ref="identity--87c30030-82c5-4e7c-a253-db6d9782ce3a",
    name="priam-sensor",
    schema="My priam sensor object",
    version="1.0",
    extension_types=[EXTENSION_TYPE_NEW_SDO]
)


@stix2.CustomObject(
    "priam-sensor",
    [
        ("name", StringProperty(required=True)),
        ("description", StringProperty()),
        ("priam_version", StringProperty()),
        ("fqdn", StringProperty())
    ],
    extension_name=PRIAM_SENSOR_EXT_ID
)
class Sensor:
    """Custom Sensor object of type stix2.CustomObject.

    Custom Properties
    -----------------
    fqdn: str
    """

    pass


# Show the extension definition too!
my_ext.fp_serialize(sys.stdout, pretty=True)
print()
sensor = Sensor(name='edr', description='an EDR sensor', fqdn='falcon.contoso.com')
sensor.fp_serialize(sys.stdout, pretty=True)

There is some documentation of the stix2 extension API here.

Sample output:

{
    "type": "extension-definition",
    "spec_version": "2.1",
    "id": "extension-definition--638dd8ff-083a-4e42-adba-702080ceb4a0",
    "created_by_ref": "identity--87c30030-82c5-4e7c-a253-db6d9782ce3a",
    "created": "2023-02-15T02:26:02.161753Z",
    "modified": "2023-02-15T02:26:02.161753Z",
    "name": "priam-sensor",
    "schema": "My priam sensor object",
    "version": "1.0",
    "extension_types": [
        "new-sdo"
    ]
}
{
    "type": "priam-sensor",
    "spec_version": "2.1",
    "id": "priam-sensor--5957492d-740e-49e6-b54e-5c6269d3c718",
    "created": "2023-02-15T02:26:02.2192Z",
    "modified": "2023-02-15T02:26:02.2192Z",
    "name": "edr",
    "description": "an EDR sensor",
    "fqdn": "falcon.contoso.com",
    "extensions": {
        "extension-definition--638dd8ff-083a-4e42-adba-702080ceb4a0": {
            "extension_type": "new-sdo"
        }
    }
}

[priamai]

Hello Chris,
just a few points: I assumed att&ck was a good approach this is why I copied their structure but maybe they wrote that many years ago before it got deprecated? My error gets triggered with the sensor_refs property, the sensor is created just fine.
Anyway you are right let me isolate the code and publish it somewhere, so it's easier to debug.

[priamai]

Hi there,
I followed your approach and now is working as expected, example here on Colab.

Also looking at the example here would be nice to see how to defined a custom SRO.

[chisholm]

For custom SROs, there is an is_sdo parameter in the CustomObject decorator. So you could do @CustomObject("foo",{...properties...}, extension_name="ext--...", is_sdo=False). It just switches the extension type to new-sro. I wonder how old that documentation is... it still mentions "STIX 2.1 CS 03", but it is an OASIS standard now. It could stand improvement.

For your "weird" cases at the end of your notebook, notice that the resulting objects have two extensions: one is an instance of the registered extension class, and the other is a plain dict. The library is designed to cope with lack of registration. It will pass through unregistered objects and extensions as they are (as a dict), if it can't find a class to use. It needs to be able to deal with unanticipated object types, where the registry isn't set up with custom classes beforehand. Having multiple extensions of type "new-sdo" in the same object does seem... weird though. Maybe the library should warn about that.

[priamai]

Hello,
hmmm but if I try to register one extension like this and use that for all the 3 objects (2 SDO and 1 SRO), it triggers a duplication error.

DuplicateRegistrationError: A(n) Extension with type 'extension-definition--638dd8ff-083a-4e42-adba-702080ceb4a0' already exists and cannot be registered again

[priamai]

Sorry just to be clear my question is if is weird to have on extension for each object (in this case I will need 3 extensions), then how should I approach the problem?

[clenk]

Using the extension_name parameter will create and register an extension with that id as part of creating the custom object. So the code registers the extension when it defines Sensor and then can't register the extension when defining Action because it's already defined. Maybe the code in this library that handles this registration should check if it's already been registered...

Another way to use the same extension in each object would be to register the extension yourself first, and then pass in the extensions dictionary when instantiating your class. For example:

@stix2.v21.CustomExtension(
    PRIAM_EXT_ID, [
        ('extension_type', EnumProperty(
            allowed=[EXTENSION_TYPE_NEW_SDO,EXTENSION_TYPE_NEW_SRO], required=True)
        )
    ],
)
class PriamExtension:
    pass

...

sensor_example = Sensor(
    name='foo',
    extensions={
        PRIAM_EXT_ID: {
            'extension_type': EXTENSION_TYPE_NEW_SDO,
        }
    }
)
print(sensor_example)

Out of curiosity, what new SRO are you creating? Is it something that cannot be modeled by the existing Relationship or Sighting object? You could create a property-extension that adds properties to one of those.

[chisholm]

Hello, hmmm but if I try to register one extension like this and use that for all the 3 objects (2 SDO and 1 SRO), it triggers a duplication error.

DuplicateRegistrationError: A(n) Extension with type 'extension-definition--638dd8ff-083a-4e42-adba-702080ceb4a0' already exists and cannot be registered again

That code doesn't register one extension, it tries to register three with the same ID. Perhaps you were thinking of the creation of the extension-definition as a kind of "registration", and then the references to its ID when creating the custom objects as "using" it? That's not what's actually happening.

What is actually registered is a custom class which corresponds to the object which must appear in the extensions property value, for a particular extension. I.e. a class is registered which handles usages of the extension definition. The extension definition itself needs no registration. In fact you don't have to create the extension-definition object at all, to be able to use it. I did in my example to illustrate the parts of the design of the extension mechanism. That definition should exist in some publicly accessible place (or at least accessible to whoever will need to understand your extension), but its creation and storage can be completely separate from its usages.

With regard to the STIX 2.1 spec and how it defines extensions: you tried to use the same extension-definition for two different SDOs. I don't see any specific discussion of that in the spec, but I don't think that's what was intended. I think a hybrid extension-definition can only represent one extension of each type. So if your extension-definition defines the priam-sensor SDO, it can't also define the priam-action SDO, but it could define the priam-generate SRO. I think you'll need to divvy up your extensions into multiple extension-definitions.

With regard to the stix2 library: when you use the CustomObject decorator to register your custom SDO class and an extension class, the latter class (which is auto-created) does not support usage in a hybrid manner. The auto-created class defines an extension_type property which will enforce a single fixed value. Hybrid usage requires allowing multiple values. The bookkeeping also does not currently support registering multiple classes under the same extension-definition ID. Library design might need some improvement here. Issue #560 is about this. clenk has a clever workaround above, though I think it would not work for toplevel property extensions. I think it will also be more complicated for a hybrid including a new object (new-sdo, new-sco, and new-sro) and property-extension, since you can't express the necessary property co-constraints in a simple declarative way like that. I think you'd need to override _check_object_constraints() like many library classes do, to make additional checks (I have not actually tried this).

Anyway, I think the library can support what you need, depending on how complex your needs get!

@clenk: The SRO extension he was creating is at the bottom of the notebook, underneath the exception stacktrace (priam-generate). It's a custom object with some ref properties. @priamai: I agree that a plain relationship SRO with a custom relationship_type (e.g. "generates") would make for wider compatibility with your content, since STIX tools already know how to deal with that type of SRO. As clenk says, you could define an extension to add additional properties to relationship SROs if necessary. But if the inherent directionality of that kind of relationship or any other semantics don't quite align, you could create your own.

[priamai]

Hi,
thanks for detailed clarification, I guess my confusion in the interpretation was the plural usage extension_types

ext_priam = stix2.ExtensionDefinition(
    id=PRIAM_EXT_ID,
    created_by_ref="identity--87c30030-82c5-4e7c-a253-db6d9782ce3a",
    name="priam-sensor",
    schema="My priam sensor object",
    version="1.0",
    extension_types=[EXTENSION_TYPE_NEW_SDO,EXTENSION_TYPE_NEW_SRO]
)

would this even be possible considering all that has been said? In other words, should we change the extension_types to extension_type a single value? Would be nice to see a counter example.

With regards to SRO, yes I totally get your point I could re-use some of the original ones and add a few properties.

I forgot to ask to the previous question, the reason for this extension is to be able to model cyber sources such as ips,av,edr,firewalls generating alerts associated to observables.
The intention is to provide more context to SOC teams and can be eventually linked to CACAO and/or OpenC2.

Let me know if this is something of general interest maybe we can plan a proper official extension?
Cheers.

[rpiazza]

On page 213 of the spec, there is an example extension definition that has multiple extension types.

I'm not sure that python-stix2 supports this - a question for @chisholm and/or @clenk

[chisholm]

Stix2 library APIs for STIX objects basically mirror the spec-defined properties, and I don't think that's likely to change.

Regarding your question, the spec allows a single extension-definition with new-sdo and new-sro extension types, but I think it does not allow one to bring an arbitrary collection of object types under the umbrella of a single extension-definition. There are five defined extension types, which means a single extension-definition can be used in at most five different ways. Only three of those extension types (usages) are for new STIX object types. So a single extension-definition can define at most three new STIX object types. There are limits as to what you can do with a single extension-definition, and those limits are not due to the stix2 library.

So perhaps the first thing to do is design spec-compliant extension-definition(s) which satisfy your needs, and then to ask whether the stix2 library allows you do work with them adequately, as an implementation. I think you will need a separate extension-definition per SDO. You could lump in the SRO with one of the SDO extensions, but it seems arbitrary which one that should be, so maybe the SRO gets its own extension definition as well?

@rpiazza maybe there is a mailing list with a wider user base where he could present these ideas, and gauge general interest?

[priamai]

Hi @chisholm we are releasing a project soon, will be easier to talk over that once is ready. Thanks for the help.

[rpiazza]

I think having only one extension type per extension definition should be a best practice.

I think the idea behind having multiple types in one extension definition was to indicate that the extension definition had multiple parts, but were basically one extension "idea". I'm not sure if that is important or adds anything useful.

[brettforbes]

Hi,

Question: Is it possible to have an Extension Definition that subclasses an existing object?

We are working with OCA IoB and they have defined a custom object "Behaviour". I have spoken to them and suggested there is some conceptual overlap with "Indicator". Is it possible to have an Extension Definition that effectively does this?

class Behaviour(Indicator):

This approach of extending existing objects, but with new names, is similar to the ATT&CK one, and is pretty useful. Can you advise whether this is one of the five Extension Definition methods (I dont think so), and/or whether it may/can appear in a future Stix release as a feature?

Thank you

[chisholm]

I don't think there's any notion of "subclass" or "subtype" in the spec, at least among object types. None of the extension types capture that. Maybe the nearest you could get is devising a relationship of some kind between a behavior and indicator, e.g. an SRO or embedded relationship. Of course, that would only enable relating instances of the "behavior" and "indicator" concepts, not the concepts themselves.

I can't speak to getting it into the spec, or similar data model extensions (the latter might require more information about the behavior type anyway).

[rpiazza]

There is a concept of "subtypes" in the specification. They were the "original" extensions - see the File SCO
Have you seen the extension policy document? Right now it is a draft, but discusses various ways to create extensions, including "new" subtypes. See if that has what you are looking for.

https://docs.google.com/document/d/1cGAQy93KuYZAgYUbzSomU_WIeDSUP4H7OVwbaBX5Szc

[chisholm]

Well that's interesting, I had not seen that. Current spec only says (section 3.10) the purpose of predefined extensions is "defining coherent sets of properties beyond the base". Maybe sometimes the intent is to create a subtype, but not always? I think intent is not captured in the STIX content. It's just something to keep in mind, and will guide how the extension gets absorbed into the spec, if that should happen.

My takeaway is that "subtypes", at least as far as they're discussed right now, are not separate object types. You'd create a property-extension type of extension definition which includes properties specific to your subtype. That would get converted to a predefined extension on the "supertype" object if it is added to the spec.

[rpiazza]

@chisholm - Yes - I was using the term "subtype" informally. Predefined extensions were a OO shortcut - I don't think the intent was every anything else. The new extension definition facility usually does not create "subtypes" - it is redefining the extended object type. When an extension definition gets absorbed into the spec - the old definition of the object type is replaced.

However, it is possible to use the extension definition facility to create new "predefined" extensions - assuming they get absorbed. Once again see the google doc for more details.