Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add folder check for load job #18473

Open
wants to merge 2 commits into
base: master-2.x
Choose a base branch
from

Conversation

Haoning-Sun
Copy link
Contributor

@Haoning-Sun Haoning-Sun commented Dec 20, 2023

What changes are proposed in this pull request?

Add folder check for load job.

Why are the changes needed?

Adding 'partial-listing' when loading a file instead of a directory will cause the loading task to fail, as it results in duplicate blocks of the file being loaded.

./bin/alluxio fs load /test.txt --submit --partial-listing
./bin/alluxio fs load /test.txt  --progress
Progress for loading path '/test.txt':
	Settings:	bandwidth: unlimited	verify: false
	Job State: FAILED (alluxio.exception.runtime.InternalRuntimeException: Job failed because it's not healthy.)
	Files Processed: 5
	Bytes Loaded: 2783.32MB
	Throughput: 927.77MB/s
	Block load failure rate: 5.02%
	Files Failed: 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant