-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[client] Improve worker behavior to handle complex bundle (#629) #630
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know how to test a complex bundle with errors in it to test the use case but locally tested normal csv mappers and import data works correctly
@@ -198,7 +233,7 @@ def import_bundle_from_json( | |||
json_data: Union[str, bytes], | |||
update: bool = False, | |||
types: List = None, | |||
retry_number: int = None, | |||
work_id: str = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there is a call in test (test_stix_crud.py) that has not been updated and still calls with a number :
bundles_sent = api_stix.import_bundle_from_json(split_bundle, False, None, 0)
pycti/utils/opencti_stix2.py
Outdated
@@ -208,16 +243,13 @@ def import_bundle_from_json( | |||
:type update: bool, optional | |||
:param types: list of stix2 types, defaults to None | |||
:type types: list, optional | |||
:param retry_number retry_number: int, optional |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
retry_number is not a param anymore
nb_deps = elem["nb_deps"] | ||
if elem["type"] == "relationship": | ||
source_ref = ( | ||
ids_equivalence.get(elem["source_ref"], elem["source_ref"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really understand what is ids_equivalence since it will always return elem["source_ref"], why not use directly elem["source_ref"] ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ids_equivalence doesnt always return elem["source_ref"].
It will return the equivalence in the dictionnary or elem["source_ref"] if not found
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how can it be different ? get method returns a default value if not found, but it doesn't store anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed the grouping part from the PR for now. It will be reintroduce in a dedicated one.
target_source = id_alias.get(source_ref) | ||
target_target = id_alias.get(target_ref) | ||
|
||
target = "generated-" + str(uuid.uuid4()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is "generated-" + str(uuid.uuid4()) ? an ID ? it's the first time I see "generated" as an ID prefix, do we handle it somewhere else in the code ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its a generic id only used to group information. This id will not be propagated as only values will remains after the processing (grouped_elements.values())
b652a6a
to
b340570
Compare
d587136
to
c126eb7
Compare
See #629