Loss of filename due to deduplication.

[ytreister]

Suppose I have two PEs e1.exe and e2.exe and when scanned with a worker plugin that analyzes PEs produce an identical extracted payload foo.dll, but the name of the DLL is foo.dll for e1.exe and bar.dll for e2.exe. The stoQ deduplication logic would prevent the same file from being analyzed twice. Suppose I have a file_details worker plugin that pulls out the filename. When all the processing is done, the file_details plugin results will only have one of the names (the first one it encounters foo.dll). Looking closely at the stoQ response object for the DLL, I would see that the extracted_from contain two references (to both files e1.exe and e2.exe). The analyzed result makes it seem as if e1.exe contains a child file named foo.dll and e2.exe also contains a child file named foo.dll, instead of a file named bar.dll which is the actual name of the file.

In order to not lose the filename data in this scenario, I propose the following change to stoQ:
Payload and Extracted Payload objects should have two new properties:

• filename: str
• also_known_as List

The filename string can be optionally added to payloads and extracted payloads added to the queue and could default to None. The also_know_as will be populated when stoQ performs deduplication logic. When it encounters a payload that has already been seen before, it would append the filename if the filename is not None to this list.

[ytreister]

@brbickel @mgeide Please let me know your thoughts on this...