-
|
I have an S3 bucket with 3+ million files in it. I want to build a channel that parses each of the files (understanding that this can take a long time to run) but I cannot get through even polling every message in the bucket without the channel crashing my server due to memory maxing out. Is it possible to not poll every message or is there a better design for this task that anyone has ideas for? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
You will have to use a tool that supports pagination. e.g. use the java SDK to call "list objects". You will likely need to use the SDK to get the objects as well. |
Beta Was this translation helpful? Give feedback.
-
|
I'd note you can also do this via rest calls but if you don't want to get hung up writing java in Mirth, that would be with http senders. |
Beta Was this translation helpful? Give feedback.
You will have to use a tool that supports pagination. e.g. use the java SDK to call "list objects". You will likely need to use the SDK to get the objects as well.