Try DuckDB

[keller-mark]

Towards #1824

Background

Change List

TODO: re-implement minimally on fresh branch, with minimal changes needed to support useSql hook (e.g. in plugin views)

TODO: For proof of concept

• Create table to store obsEmbedding X/Y coordinates plus obs IDs in database
• Insert obsEmbedding data into database the first time it is requested from data-hooks.
• Query DB for obsEmbedding table any time it is requested from data-hooks.
• Pass Arrow table directly into DeckGL without modification - see https://github.com/geoarrow/deck.gl-layers
• Benchmark initial load time and network request differences?
• Use lazy database instantiation with https://www.npmjs.com/package/lazy-duckdb-react
• Reduce bundle size below 50 MB to ensure compatibility with vitessce-python and unpkg
  • Use Observable approach of hosting DuckDB WASM files on CDN/object store that we have control over
• For tables which contain Arrow dictionaries, insert only the "codes" part and keep the "categories" part in JS
  • Other option: https://duckdb.org/docs/sql/functions/nested#list_positionlist-element
  • Other option: when inserting dictionary, first create table for categories, then reference those using integer foreign key column when inserting codes

Notes

• Had to manually change MIME type of .wasm files pushed to S3 during ./scripts/push-demos.sh from binary/octet-stream to application/wasm. TODO: do this via AWS CLI within push-demos script

Checklist

• Ensure PR works with all demos on the dev.vitessce.io homepage
• Open (draft) PR's into vitessce-python and vitessce-r if this is a release PR
• Documentation added or updated

[github-actions]

Size Change: +122 MB (+936%) 🆘

Total Size: 135 MB

Filename	Size	Change
./packages/main/prod/dist/index-********.js	129 MB	+122 MB (+1685%)	🆘

ℹ️ View Unchanged

Filename	Size
./packages/main/prod/dist/blosc-********.js	612 kB
./packages/main/prod/dist/browser-********.js	16.2 kB
./packages/main/prod/dist/chunk-INHXZS53-********.js	558 B
./packages/main/prod/dist/deflate-********.js	243 B
./packages/main/prod/dist/gzip-********.js	693 B
./packages/main/prod/dist/hglib-********.js	4.33 MB
./packages/main/prod/dist/index.min.js	902 B
./packages/main/prod/dist/jpeg-********.js	15.3 kB
./packages/main/prod/dist/lerc-********.js	47.2 kB
./packages/main/prod/dist/lz4-********.js	43.9 kB
./packages/main/prod/dist/lzw-********.js	2.1 kB
./packages/main/prod/dist/packbits-********.js	576 B
./packages/main/prod/dist/pako.esm-********.js	68.6 kB
./packages/main/prod/dist/raw-********.js	168 B
./packages/main/prod/dist/webimage-********.js	836 B
./packages/main/prod/dist/zlib-********.js	695 B
./packages/main/prod/dist/zstd-********.js	643 kB

_{compressed-size-action}

[keller-mark]

Notes on apache arrow:

How to set (inefficiently) a single string value to repeat for every row

export function repeatString(val, numRows) {
  return arrow.vectorFromArray(
    Array.from({ length: numRows }).fill(val),
    new arrow.Dictionary(new arrow.Utf8, new arrow.Int32),
  )
}

let arrowTable = // something
arrowTable = arrowTable.assign(arrow.makeTable({
  myColumnName: repeatString(valueToRepeat, arrowTable.numRows)
}))