Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[client] Improve worker behavior to handle complex bundle (#629) #630

Merged
merged 11 commits into from
May 25, 2024

Conversation

richard-julien
Copy link
Member

See #629

@richard-julien richard-julien linked an issue May 1, 2024 that may be closed by this pull request
Copy link
Member

@lndrtrbn lndrtrbn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how to test a complex bundle with errors in it to test the use case but locally tested normal csv mappers and import data works correctly

@@ -198,7 +233,7 @@ def import_bundle_from_json(
json_data: Union[str, bytes],
update: bool = False,
types: List = None,
retry_number: int = None,
work_id: str = None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a call in test (test_stix_crud.py) that has not been updated and still calls with a number :
bundles_sent = api_stix.import_bundle_from_json(split_bundle, False, None, 0)

@@ -208,16 +243,13 @@ def import_bundle_from_json(
:type update: bool, optional
:param types: list of stix2 types, defaults to None
:type types: list, optional
:param retry_number retry_number: int, optional
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retry_number is not a param anymore

nb_deps = elem["nb_deps"]
if elem["type"] == "relationship":
source_ref = (
ids_equivalence.get(elem["source_ref"], elem["source_ref"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand what is ids_equivalence since it will always return elem["source_ref"], why not use directly elem["source_ref"] ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ids_equivalence doesnt always return elem["source_ref"].
It will return the equivalence in the dictionnary or elem["source_ref"] if not found

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how can it be different ? get method returns a default value if not found, but it doesn't store anything.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the grouping part from the PR for now. It will be reintroduce in a dedicated one.

target_source = id_alias.get(source_ref)
target_target = id_alias.get(target_ref)

target = "generated-" + str(uuid.uuid4())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is "generated-" + str(uuid.uuid4()) ? an ID ? it's the first time I see "generated" as an ID prefix, do we handle it somewhere else in the code ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its a generic id only used to group information. This id will not be propagated as only values will remains after the processing (grouped_elements.values())

@richard-julien richard-julien merged commit da56f6c into master May 25, 2024
6 checks passed
@richard-julien richard-julien deleted the opencti/issue/6844 branch May 25, 2024 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve worker behavior to handle complex bundle
3 participants