Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat]Rocksdb backend #121

Open
wants to merge 67 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
d6a137c
RocksDB hashtable backend for Tensorflow Recommenders Addons.
bashimao Jul 17, 2021
0b48160
Fix major bugs and establish proper compiler toolchain.
bashimao Jul 18, 2021
6eb84c4
Only create if not in read_only mode.
bashimao Jul 18, 2021
e8cbf37
Decouple cleaning from creating a column family.
bashimao Jul 18, 2021
9cdaec7
Correct pending linker issues.
bashimao Jul 20, 2021
42aff13
Add op definitions for RocksDB.
bashimao Jul 20, 2021
222fe26
Need to include op-definitions in compiler directives.
bashimao Jul 20, 2021
052c2e2
Automatically import RocksDB table object upon module load.
bashimao Jul 20, 2021
96856d7
Add interface code to properly transfer arguments to the C++ implemen…
bashimao Jul 20, 2021
06738d2
Typo...
bashimao Jul 20, 2021
23a0b0e
Fix opening non-existent database bug.
bashimao Jul 20, 2021
b55ac03
Merge branch 'tensorflow:master' into rocksdb-backend
bashimao Jul 20, 2021
c2ae3c3
Fix typo.
bashimao Jul 20, 2021
159c4c3
Add connection pool functionality to allow access multiple tables in …
bashimao Jul 22, 2021
51e8389
Fix insert bug. Add configurable size function, two modes: 1) through…
bashimao Jul 22, 2021
e8558e1
Merge branch 'rocksdb-backend' of github.com:bashimao/recommenders-ad…
bashimao Jul 22, 2021
4c0d09f
Fix up all known issues, add registry functionalities.
bashimao Jul 23, 2021
6287344
Make DBWrapper thread-safe.
bashimao Jul 23, 2021
101275c
Switch to reader/writer lock for performance.
bashimao Jul 23, 2021
ccc7b58
Allow settign export_path from Python.
bashimao Jul 23, 2021
d9e8680
Extract the default value's shape.
bashimao Jul 23, 2021
ed8f58a
Improved support for multi-dimensional tensors.
bashimao Jul 24, 2021
15c7953
Improve multi-threading stability. Added limit for export to pass tes…
bashimao Jul 24, 2021
2fb2dec
Fix typo.
bashimao Jul 24, 2021
2dff086
Now passing all but 2 tests.
bashimao Jul 24, 2021
c7828ee
Loop condition was wrong.
bashimao Jul 24, 2021
43fe98e
Pointer value instead of pointed to value written to file.
bashimao Jul 24, 2021
1de6a0a
Protect against divide by zero.
bashimao Jul 24, 2021
626ce3a
Add Import/Export lock to avoid overlapping transactions - despite re…
bashimao Jul 24, 2021
972c4f9
Remove export lock idea. Instead make the mutex absolute. We have a p…
bashimao Jul 24, 2021
e6d52d9
Passing all relevant checks.
bashimao Jul 24, 2021
c770dc0
Reformatted cc-/h-files to conform to Google-style.
bashimao Aug 10, 2021
62600bd
Tick up copyright disclaimer version number.
bashimao Aug 10, 2021
c8ce053
Reformat with yapf.
bashimao Aug 20, 2021
ff725b2
Tick down rules_foreign_cc package version according to Heka's recomm…
bashimao Aug 22, 2021
eff7a39
Merge branch 'master' into rocksdb-backend
bashimao Aug 30, 2021
de7fb53
Reformat python code.
bashimao Aug 30, 2021
82816b0
Merge branch 'master' into rocksdb-backend
bashimao Oct 17, 2021
804efb9
Code cosmetics
bashimao Oct 19, 2021
e116424
Fix Python format according to proposed coding standards.
bashimao Oct 24, 2021
36bd30a
Manually ran clang format. Not sure why it did not work before.
bashimao Oct 24, 2021
bbeca4a
Merge branch 'master' into rocksdb-backend
rhdong Jan 5, 2022
05d6d2a
Merge branch 'master' into rocksdb-backend
bashimao Jan 23, 2022
45a748b
Fix up Bazel rules for 0.6 foreign_cc.
bashimao Jan 23, 2022
567e92d
Somehow these libs are no longer needed?!
bashimao Jan 26, 2022
dbcd36b
Resolve most wiggles that prevent compiling ant testing.
bashimao Jan 26, 2022
ddbb499
Temporary fix for library search path.
bashimao Jan 26, 2022
bff04bb
Apply yapf.
bashimao Jan 27, 2022
08c83e4
Apply Clang Format.
bashimao Jan 27, 2022
fc7ac16
Change to a more commonly used path that has symlinks.
bashimao Jan 27, 2022
f6bd10d
Add dependencies to ci images.
bashimao Jan 27, 2022
46be6bd
Fix syntax error. ;-)
bashimao Jan 27, 2022
ecdc998
Merge branch 'master' into rocksdb-backend
bashimao Mar 27, 2022
0011199
Now included in base image. https://github.com/tensorflow/recommender…
bashimao Mar 27, 2022
fbd18bc
Simplify a couple of things.
bashimao Mar 27, 2022
b32139d
Can process true/false. So why is it 0/1?
bashimao Mar 27, 2022
15714b3
Update C++ portion of RocksDB implementation to accommodate interface…
bashimao Mar 27, 2022
9811e82
Merge branch 'tensorflow:master' into rocksdb-backend
bashimao May 26, 2022
db166a3
Merge branch 'master' into rocksdb-backend
bashimao Jun 12, 2022
366638d
Allow using pre-built rocksdb version.
bashimao Jun 12, 2022
a1bb85f
Merge branch 'master' into rocksdb-backend
bashimao Aug 15, 2022
778f244
Fix up bazel environment to use pre-compiled rocksdb.
bashimao Aug 15, 2022
f86ec70
Implement MemoryUsed API.
bashimao Aug 15, 2022
eaa6219
Bugfixes and implement FindWithExists API.
bashimao Aug 16, 2022
4ebd8ac
Minor bugfix.
bashimao Aug 16, 2022
4a86ac2
Add minor error check.
bashimao Aug 16, 2022
53b732d
Accumulation API. Dummy implementation.
bashimao Aug 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 16 additions & 0 deletions WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,22 @@ http_archive(
url = "https://github.com/sewenew/redis-plus-plus/archive/refs/tags/1.2.3.zip",
)

# Enable to build RocksDB from source.
# http_archive(
# name = "rocksdb",
# build_file = "//build_deps/toolchains/rocksdb:rocksdb.BUILD",
# sha256 = "2df8f34a44eda182e22cf84dee7a14f17f55d305ff79c06fb3cd1e5f8831e00d",
# strip_prefix = "rocksdb-6.22.1",
# urls = [
# "https://github.com/facebook/rocksdb/archive/refs/tags/v6.22.1.tar.gz",
# ],
# )
new_local_repository(
name = "rocksdb",
build_file = "//build_deps/toolchains/rocksdb:rocksdb.BUILD",
path = "/usr/local"
)

tf_configure(
name = "local_config_tf",
)
Expand Down
Empty file.
33 changes: 33 additions & 0 deletions build_deps/toolchains/rocksdb/rocksdb.BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
load("@rules_foreign_cc//foreign_cc:defs.bzl", "make")

package(
default_visibility = ["//visibility:public"],
)

licenses(["notice"]) # BSD

filegroup(
name = "all_srcs",
srcs = glob(["**"]),
visibility = ["//visibility:public"],
)

# Enable this to compile RocksDB from source instead.
#make(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just remove the block

# name = "rocksdb",
# args = [
# "EXTRA_CXXFLAGS=\"-fPIC -D_GLIBCXX_USE_CXX11_ABI=0\"",
# "-j6",
# ],
# targets = ["static_lib", "install-static"],
# lib_source = "@rocksdb//:all_srcs",
# out_static_libs = ["librocksdb.a"],
#)

cc_library(
name = "rocksdb",
srcs = ["lib/librocksdb.a"],
includes = ["./include"],
hdrs = glob(["rocksdb/*.h"]),
visibility = ["//visibility:public"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
'RedisTable',
'RedisTableConfig',
'RedisTableCreator',
'RocksDBTable',
'RocksDBTableConfig',
'RocksDBTableCreator',
'Variable',
'TrainableWrapper',
'DynamicEmbeddingOptimizer',
Expand Down Expand Up @@ -50,11 +53,15 @@
CuckooHashTableCreator,
RedisTableConfig,
RedisTableCreator,
RocksDBTableConfig,
RocksDBTableCreator,
)
from tensorflow_recommenders_addons.dynamic_embedding.python.ops.cuckoo_hashtable_ops import (
CuckooHashTable,)
from tensorflow_recommenders_addons.dynamic_embedding.python.ops.redis_table_ops import (
RedisTable,)
from tensorflow_recommenders_addons.dynamic_embedding.python.ops.rocksdb_table_ops import (
RocksDBTable,)
from tensorflow_recommenders_addons.dynamic_embedding.python.ops.dynamic_embedding_ops import (
embedding_lookup,)
from tensorflow_recommenders_addons.dynamic_embedding.python.ops.dynamic_embedding_ops import (
Expand Down
24 changes: 24 additions & 0 deletions tensorflow_recommenders_addons/dynamic_embedding/core/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,30 @@ custom_op_library(
],
)

custom_op_library(
name = "_rocksdb_table_ops.so",
srcs = [
"kernels/rocksdb_table_op.cc",
"kernels/rocksdb_table_op.h",
"ops/rocksdb_table_ops.cc",
"utils/types.h",
"utils/utils.h",
],
# Hack: To allow allow locating <rocksdb/xyz.h>.
includes = [
".",
],
linkopts = [
"-L/usr/local/lib",
"-lbz2",
"-llz4",
"-lzstd",
],
deps = [
"@rocksdb",
],
)

custom_op_library(
name = "_math_ops.so",
srcs = [
Expand Down