Bulk Loading Part 1: Transaction-Based User Interface #11343

kakaiu · 2024-04-26T22:27:08Z

Design principles to make it simple and avoid error-prone:

Bulk loading is managed based on KeyRange. The bulk loading UI must guarantee that a bulk loading task is running on at most one KeyRange, and a KeyRange can have at most one bulk loading task at a time.
Users can never overwrite a key range that is running a bulk loading task, unless the user has manually cancelled it.
Even for a key range which bulk loading task has been completed, users cannot start a new task overlapping the range unless users have explicitly removed the old task on the range.
When users input a range to remove or cancel bulk loading tasks, the system must cancel all tasks intersecting the input range.
Each transaction issued by users can only include one of the three types of the operation: new task, check status, cancel task.

User interface:
Say, tr is a readwrite transaction. Each transaction can only include one of the three following types of operation:

New task: tr->set("\xff\xff/bulk_loading/task/", bulkLoadStateValue(range, path)); wait(tr->commit()); Succeed if no existing bulk loading task (in metadata) intersecting the input task range; Otherwise, throwing an error.
Check status: wait(tr->getRange("\xff\xff/bulk_loading/status/a", "\xff\xff/bulk_loading/status/b")); Return all tasks's status intersecting the input range.
Cancel task: tr->clear("\xff\xff/bulk_loading/cancel/a", "\xff\xff/bulk_loading/cancel/b"); wait(tr->commit()); Cancel all tasks intersecting the input range.

Those operations to the special key space are mapped to read/write operation on "\xff/bulkLoad/" system key space, which in turn to control the actual bulk loading in FDB internal.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

The PR has a description, explaining both the problem and the solution.
The description mentions which forms of testing were done and the testing seems reasonable.
Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci · 2024-04-26T22:31:10Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 192b0b1
Duration 0:03:53
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:31:14Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 192b0b1
Duration 0:03:54
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:31:18Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 192b0b1
Duration 0:03:56
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:31:20Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 192b0b1
Duration 0:04:03
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-04-26T22:37:56Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 192b0b1
Duration 0:10:37
Result: ❌ FAILED
Error: reference not found for primary source and source version 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:38:15Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 192b0b1
Duration 0:10:53
Result: ❌ FAILED
Error: reference not found for primary source and source version 192b0b19d7031687fb7a3f65c0e86919cccd5b7d
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:55:46Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 4831a38
Duration 0:24:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T22:58:37Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 42f624b
Duration 0:24:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T23:00:32Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 4831a38
Duration 0:29:46
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T23:04:52Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 42f624b
Duration 0:31:03
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T23:07:36Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 4831a38
Duration 0:36:47
Result: ❌ FAILED
Error: Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-04-26T23:10:08Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 4831a38
Duration 0:39:22
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-26T23:10:39Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 42f624b
Duration 0:36:49
Result: ❌ FAILED
Error: Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-04-26T23:12:04Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 42f624b
Duration 0:38:16
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:02:38Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 50c9bae
Duration 0:22:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:14:28Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 50c9bae
Duration 0:34:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:17:10Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 50c9bae
Duration 0:37:16
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:21:23Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: b9bdb9b
Duration 0:22:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:26:19Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 50c9bae
Duration 0:46:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:27:42Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: b9bdb9b
Duration 0:29:12
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:32:49Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 50c9bae
Duration 0:52:57
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:35:56Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: b9bdb9b
Duration 0:37:28
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:38:07Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 116780b
Duration 0:21:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:40:41Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 50c9bae
Duration 1:00:48
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-04-30T23:50:10Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 116780b
Duration 0:33:51
Result: ❌ FAILED
Error: Error while executing command: mvn install:install-file --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -Dfile=packages/fdb-java-${FDB_VERSION}-SNAPSHOT.jar -DgroupId=org.foundationdb -DartifactId=fdb-java -Dversion=${FDB_VERSION}-SNAPSHOT -Dpackaging=jar -DgeneratePom=true. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-04-30T23:51:10Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 116780b
Duration 0:34:54
Result: ❌ FAILED
Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:52:08Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 116780b
Duration 0:35:51
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-04-30T23:53:22Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 116780b
Duration 0:37:07
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-01T00:00:14Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: b9bdb9b
Duration 1:01:44
Result: ❌ FAILED
Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack

New task: tr->set("\xff\xff/bulk_loading/task/", bulkLoadStateValue(range, path));

We don't want to read the key range from the SST file?

wait(tr->commit()); Succeed if no existing bulk loading task (in metadata) intersecting the input task range; Otherwise, throwing an error.

Will there be a check of the SST content on transaction commit to make sure it is wholesome? What if the SST does not exist? Will we get an error here?

What if the path-to-sst has keys that are not inside the specified range? Will we get an error on commit?

Cancel task: tr->clear("\xff\xff/bulk_loading/cancel/a", "\xff\xff/bulk_loading/cancel/b"); wait(tr->commit()); Cancel all tasks intersecting the input range.

This will cancel bulk loads that have 'succeeded' too? (It's a little confusing here around how successful bulk loads are handled.... we need to clear their task? Does above 'cancel' the successful bulk loads?)

Users can never overwrite a key range that is running a bulk loading task, unless the user has manually cancelled it.

What happens if the user does this? Is there protection? Will user get an error?

saintstack

A few notes...

saintstack · 2024-05-01T00:09:33Z

fdbclient/SpecialKeySpace.actor.cpp

@@ -74,6 +78,7 @@ std::unordered_map<SpecialKeySpace::MODULE, KeyRange> SpecialKeySpace::moduleToB
 	{ SpecialKeySpace::MODULE::ACTOR_PROFILER_CONF,
 	  KeyRangeRef("\xff\xff/actor_profiler_conf/"_sr, "\xff\xff/actor_profiler_conf0"_sr) },
 	{ SpecialKeySpace::MODULE::CLUSTERID, singleKeyRange("\xff\xff/cluster_id"_sr) },
+	{ SpecialKeySpace::MODULE::BULKLOADING, KeyRangeRef("\xff\xff/bulk_loading/"_sr, "\xff\xff/bulk_loading0"_sr) },


I think comment on what the content of this area in the special key space will contain will help.... here or probably better down in SystemData.

saintstack · 2024-05-01T00:11:12Z

fdbclient/SpecialKeySpace.actor.cpp

+	    range.withPrefix(SpecialKeySpace::getModuleRange(SpecialKeySpace::MODULE::BULKLOADING).begin);
+	auto ranges = ryw->getSpecialKeySpaceWriteMap().containedRanges(rangeToCheck);
+	for (auto iter = ranges.begin(); iter != ranges.end(); ++iter) {
+		if (iter->value().first && iter->value().second.present()) {


Don't need 'present' on value().first?

saintstack · 2024-05-01T00:16:04Z

fdbclient/include/fdbclient/BulkLoading.h

+	ShardedRocksDB = 3,
+};
+
+struct BulkLoadState {


Comments? Perhaps paste the PR design principles in here? How to use this API?

saintstack · 2024-05-01T00:18:03Z

fdbclient/SpecialKeySpace.actor.cpp

+Future<Optional<std::string>> BulkLoadCancelImpl::commit(ReadYourWritesTransaction* ryw) {
+	return BulkLoadingCancelCommitActor(ryw, getKeyRange());
+}
+


Is it possible to write unit tests for this range 'math'; i.e. checking inside range with prefix, w/o prefix etc.

foundationdb-ci · 2024-05-02T18:12:30Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: ef6d45c
Duration 0:21:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-02T18:26:11Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: ef6d45c
Duration 0:35:09
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-02T18:37:43Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: ef6d45c
Duration 0:46:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-02T18:37:49Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: ef6d45c
Duration 0:46:49
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-02T18:45:26Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: ef6d45c
Duration 0:54:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-05-02T18:51:19Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: ef6d45c
Duration 1:00:19
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-03T01:58:38Z

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 4ed75cc
Duration 0:21:34
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-03T02:10:26Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 4ed75cc
Duration 0:33:24
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-03T02:22:10Z

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 4ed75cc
Duration 0:45:05
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-03T02:24:20Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 4ed75cc
Duration 0:47:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2024-05-03T02:31:21Z

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 4ed75cc
Duration 0:54:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2024-05-03T02:39:55Z

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 4ed75cc
Duration 1:02:50
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

jzhou77 · 2024-05-13T17:05:42Z

flow/include/flow/error_definitions.h

+ERROR( bulkload_add_task_input_error, 1086, "Bulk loading add task input is error" )
+ERROR( bulkload_cancel_task_input_error, 1087, "Bulk loading cancel task input is error" )


For these 3 errors, is it possible to consolidate into one error type? The caller with the context should be able to figure out which task has failed.

jzhou77 · 2024-05-15T03:45:33Z

fdbclient/SpecialKeySpace.actor.cpp

+	ASSERT(bulkLoadState.isValid());
+	KeyRangeRef bulkLoadRange = bulkLoadState.range;
+	ASSERT(!bulkLoadRange.empty());
+	if (bulkLoadRange.begin >= normalKeys.end || bulkLoadRange.end >= normalKeys.end) {


can be simplified as

Suggested change

if (bulkLoadRange.begin >= normalKeys.end || bulkLoadRange.end >= normalKeys.end) {

if (bulkLoadRange.end >= normalKeys.end) {

kakaiu requested review from liquid-helium and jzhou77 April 26, 2024 22:27

kakaiu force-pushed the bulk-loading-framework branch from 192b0b1 to 4831a38 Compare April 26, 2024 22:30

kakaiu force-pushed the bulk-loading-framework branch from 4831a38 to 42f624b Compare April 26, 2024 22:33

kakaiu requested a review from saintstack April 29, 2024 17:55

kakaiu changed the title ~~Transaction-Based Bulk Loading User Interface~~ Bulk Loading 1: Transaction-Based User Interface Apr 30, 2024

kakaiu changed the title ~~Bulk Loading 1: Transaction-Based User Interface~~ Bulk Loading Part 1: Transaction-Based User Interface Apr 30, 2024

saintstack reviewed May 1, 2024

View reviewed changes

kakaiu added 4 commits May 2, 2024 10:50

bulk loading interface

093df69

clear any range intersecting the input range to cancel

3785e04

add more test

d4b5d0d

add more test

ef6d45c

kakaiu force-pushed the bulk-loading-framework branch from 116780b to ef6d45c Compare May 2, 2024 17:50

add bulk load mode

4ed75cc

jzhou77 reviewed May 15, 2024

View reviewed changes

		ERROR( bulkload_add_task_input_error, 1086, "Bulk loading add task input is error" )
		ERROR( bulkload_cancel_task_input_error, 1087, "Bulk loading cancel task input is error" )

	if (bulkLoadRange.begin >= normalKeys.end \|\| bulkLoadRange.end >= normalKeys.end) {
	if (bulkLoadRange.end >= normalKeys.end) {

Bulk Loading Part 1: Transaction-Based User Interface #11343

Are you sure you want to change the base?

Bulk Loading Part 1: Transaction-Based User Interface #11343

Conversation

kakaiu commented Apr 26, 2024 • edited

Code-Reviewer Section

For Release-Branches

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Apr 26, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented Apr 30, 2024

Result of foundationdb-pr-clang on Linux CentOS 7

foundationdb-ci commented May 1, 2024

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

saintstack left a comment

Choose a reason for hiding this comment

saintstack left a comment

Choose a reason for hiding this comment

saintstack May 1, 2024

Choose a reason for hiding this comment

saintstack May 1, 2024

Choose a reason for hiding this comment

saintstack May 1, 2024

Choose a reason for hiding this comment

saintstack May 1, 2024

Choose a reason for hiding this comment

foundationdb-ci commented May 2, 2024

Result of foundationdb-pr-clang-ide on Linux CentOS 7

foundationdb-ci commented May 2, 2024

kakaiu commented Apr 26, 2024 •

edited