-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Connector-V2] Supports the transfer of any file #6826
base: dev
Are you sure you want to change the base?
Conversation
aa7a3ff
to
b20a0ab
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cute cat ?
does this means we can use this feature copy file between different file systems? |
yep. |
while ((readSize = inputStream.read(buffer)) != -1) { | ||
if (readSize != maxSize) { | ||
buffer = Arrays.copyOf(buffer, readSize); | ||
} | ||
SeaTunnelRow row = new SeaTunnelRow(new Object[] {buffer, relativePath, partIndex}); | ||
buffer = new byte[1024]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this while loop, a new buffer is reallocated after each read:
buffer = new byte[1024]
It might be better to reuse an existing buffer:
Arrays.fill(buffer, (byte) 0)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the suggestion, the original version is actually exactly as you described. But buffer
is an object and will eventually be used by the sink writer. If we reuse it in advance, the sink will not be able to get the correct data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your reply, it is cool to read and write any file!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Purpose of this pull request
This PR add new file format
binary
. With this format, SeaTunnel can transfer any file, such as compressed packages, pictures, etc. We can move files between different environments.There are a demo of copy file from local to local.
Does this PR introduce any user-facing change?
no
How was this patch tested?
add new test.
Check list
New License Guide
release-note
.