DiStash

DiStash is a disaggregated transactional key-value store for data across pools of different storage medium such as Dynamic Random Access Memory (DRAM), Solid State Disk (SSD), Hard Disk Drive (HDD), or Non-Volatile Memory (NVM). An application may configure a storage pool as either durable or ephemeral (transient cache). DiStash simplifies the application logic by preventing undesirable race conditions that may cause copies of data across different pools of storage to diverge.

DiStash is built using FoundationDB. To learn more about FoundationDB, visit foundationdb.org

Authors: Yiming Gao (gaoyimin@usc.edu), Hieu Nguyen (hieunguyen@ebay.com), Jun Li (junli5@ebay.com), and Shahram Ghandeharizadeh (shahram@usc.edu)

Features

DiStash is a research prototype that supports:

• Pools of stashes. A stash is a storage medium such as Dynamic Random Access Memory (DRAM), Solid State Disk (SSD), Hard Disk Drive (HDD), or Non-Volatile Memory (NVM).
• A single transaction to read and write different copies of one or more key-value pair across the different pools of stashes.
• Pools of stashes as either ephemeral (cache) or durable storage. In ephemeral mode, once the storage capacity of a stash is exhausted, DiStash uses LRU to evict data.
• DiStash may be used to implement cache augmented data stores, front-end load balancing, and hierarchical storage structures.

Getting Started

Compiling from source

A build of DiStash requires at least these dependencies:

1. Install cmake Version 3.13 or higher CMake, on a MacBook Pro: brew install cmake
2. Install Mono, on a MacBook Pro: download pkg file and install
3. Install Ninja (optional, but recommended), on a MacBook Pro: brew install ninja

We also recomment installing:

1. openssl, on a MacBook Pro: brew install openssl
2. lz4, on a Macbook Pro: brew install lz4
3. sphinx-doc, on a Macbook Pro: brew install sphinx-doc
4. jinja2-cli, on a Macbook Pro: brew install jinja2-cli
5. pip install jinja2

If compiling for local development, please set -DUSE_WERROR=ON in cmake. Our CI compiles with -Werror on, so this way you'll find out about compiler warnings that break the build earlier.

Once you have your dependencies, you can run cmake and then build:

1. Check out this repository.
2. Create a build directory (you can have the build directory anywhere you like).
3. cd <PATH_TO_BUILD_DIRECTORY>
4. cmake -G Ninja <PATH_TO_FOUNDATIONDB_DIRECTORY>
5. ninja # If this crashes it probably ran out of memory. Try ninja -j1

Language Bindings

The language bindings that are supported by cmake will have a corresponding README.md file in the corresponding bindings/lang directory.

Generally, cmake will build all language bindings for which it can find all necessary dependencies. After each successful cmake run, cmake will tell you which language bindings it is going to build.

Generating compile_commands.json

CMake can build a compilation database for you. However, the default generated one is not too useful as it operates on the generated files. When running make, the build system will create another compile_commands.json file in the source directory. This can than be used for tools like CCLS, CQuery, etc. This way you can get code-completion and code navigation in flow. It is not yet perfect (it will show a few errors) but we are constantly working on improving the development experience.

CMake will not produce a compile_commands.json, you must pass -DCMAKE_EXPORT_COMPILE_COMMANDS=ON. This also enables the target processed_compile_commands, which rewrites compile_commands.json to describe the actor compiler source file, not the post-processed output files, and places the output file in the source directory. This file should then be picked up automatically by any tooling.

Note that if building inside of the foundationdb/build docker image, the resulting paths will still be incorrect and require manual fixing. One will wish to re-run cmake with -DCMAKE_EXPORT_COMPILE_COMMANDS=OFF to prevent it from reverting the manual changes.

Using IDEs

CMake has built in support for a number of popular IDEs. However, because flow files are precompiled with the actor compiler, an IDE will not be very useful as a user will only be presented with the generated code - which is not what she wants to edit and get IDE features for.

The good news is, that it is possible to generate project files for editing flow with a supported IDE. There is a CMake option called OPEN_FOR_IDE which will generate a project which can be opened in an IDE for editing. You won't be able to build this project, but you will be able to edit the files and get most edit and navigation features your IDE supports.

For example, if you want to use XCode to make changes to FoundationDB you can create a XCode-project with the following command:

cmake -G Xcode -DOPEN_FOR_IDE=ON <FDB_SOURCE_DIRECTORY>

You should create a second build-directory which you will use for building and debugging.

FreeBSD

1. Check out this repo on your server.

2. Install compile-time dependencies from ports.

3. (Optional) Use tmpfs & ccache for significantly faster repeat builds

4. (Optional) Install a JDK for Java Bindings. FoundationDB currently builds with Java 8.

5. Navigate to the directory where you checked out the foundationdb repo.

6. Build from source.

   sudo pkg install -r FreeBSD \
       shells/bash devel/cmake devel/ninja devel/ccache  \
       lang/mono lang/python3 \
       devel/boost-libs devel/libeio \
       security/openssl
   mkdir .build && cd .build
   cmake -G Ninja \
       -DUSE_CCACHE=on \
       -DUSE_DTRACE=off \
       ..
   ninja -j 10
   # run fast tests
   ctest -L fast
   # run all tests
   ctest --output-on-failure -v

CloudLab

Create an instance with the latest versin of Ubuntu (24 as of May 8, 2024). Bring it up to date:

sudo apt update

Install cmake on it.

sudo apt install cmake

Follow the FDB installation guide and verify the version of cmake is 3.13 or higher.

Proceed to install the other required libraries and packages. Install Ninja:

sudo apt-get install ninja-build

Install jemalloc:

sudo apt install libjemalloc-dev

Install ssl:

sudo apt install libssl-dev

Install compression technique lz4:

sudo apt install liblz4-dev

Install compression technique python sphinx:

sudo apt install python3-sphinx

Install mono by following instructions at :

https://www.mono-project.com/download/stable/#dwonload-lin

Create the build directory, change into that directory and run:

cmake -G Ninja ../foundationdb_cache

Now, build FDB:

ninja

eBay Data Center

See https://github.com/eBay-USC/fdb_deploy/tree/fdb_cache2 and follow its instructions.

Linux

There are no special requirements for Linux. A docker image can be pulled from foundationdb/build that has all of FoundationDB's dependencies pre-installed, and is what the CI uses to build and test PRs.

cmake -G Ninja <FDB_SOURCE_DIR>
ninja
cpack -G DEB

For RPM simply replace DEB with RPM.

MacOS

The build under MacOS will work the same way as on Linux. To get boost and ninja you can use Homebrew:

brew install llvm
brew install lld

Verify that the following paths are set:

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/llvm/lib" 
export CPPFLAGS="-I/opt/homebrew/opt/llvm/include" 

To generate a installable package,first compile:

cmake -G Ninja \
      -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang \
      -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \
      -DCMAKE_AR=/opt/homebrew/opt/llvm/bin/llvm-ar \
      -DCMAKE_RANLIB=/opt/homebrew/opt/llvm/bin/llvm-ranlib \
      -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=/opt/homebrew/bin/ld64.lld" \
      -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=/opt/homebrew/bin/ld64.lld" \
<PATH_TO_FOUNDATIONDB_SOURCE>

From the build directory, compile using the following command:

ninja

From the build directory, generate the package:

$SRCDIR/packaging/osx/buildpkg.sh . $SRCDIR

Prior to installing the newly compiled package, remove an existing FDB installation:

sudo /usr/local/foundationdb/uninstall-FoundationDB.sh

Remove its data and configuration files:

sudo rm -rf /usr/local/foundationdb /usr/local/etc/foundationdb

Use the finder to browse the directory that contains the newly generated package:

cd build/packages

Double click the FoundationDB package to install.

Citations

Y. Gao, H. Nguyen, J. Li, and S. Ghandeharizadeh. DiStash: A Disaggregated Multi-Stash Transactional Key-Value Store. In the Seventeenth TPC Technology Conference on Performance Evaluation & Benchmarking, co-located with VLDB 2025, London, UK, September 1, 2025.

BibTex:

@inproceedings{distash2025, 
author = {Gao, Yiming and Nguyen, Hieu and Li, Jun and Ghandeharizadeh, Shahram}, 
title = {DiStash: A Disaggregated Multi-Stash Transactional Key-Value Store}, 
year = {2025}, 
abstract = {A stash is a storage medium such as Dynamic Random Access Memory (DRAM), Solid State Disk (SSD), Hard Disk Drive (HDD), or Non-Volatile Memory (NVM).  This paper presents a disaggregated transactional key-value (KV) store, DiStash, that governs KVs cross pools of stash types.  It enables an application to use a single transaction to read and write different copies of one or more key-value pair across the different pools of stashes.  It simplifies the application logic by (a) preventing undesirable race conditions that may cause copies of data across different stash pools to reflect different values and/or (b) failures that may result in loss of key-value pairs. A configuration of DiStash may use a pool of stashes as either ephemeral or durable storage.  The application dictates whether the content of its participating stashes are inclusive (replicated) or exclusive (tiered).  We implement a DiStash by extending FoundationDB.  We quantify the tradeoffs with its design decisions using microbenchmarks and eBay’s production workload.}, 
booktitle = {Proceedings of the Seventeenth TPC Technology Conference on Performance Evaluation \& Benchmarking}, 
location = {London, England}, 
series = {TPCTC '25} 
}