v1.3.0-RC1

Changelog

[v1.3.0-RC1] - 2022-05-19

Bug Fixes

• cmd/immuclient: Do not crash on login prompt
• embedded/sql: selector resolution using valuesRowReader
• embedded/sql: continue stmt execution on handler after changing db in use
• embedded/sql: increase auto_increment pk once per row
• embedded/sql: typo in error message
• embedded/sql: adjust named parameter parsing
• github: Run sonarcloud code analysis after cove coverate
• pkg/database: avoid silent returns when the scan limit is reached
• pkg/database: Fix detection of incorrect revision numbers
• pkg/database: Correctly interpret negative revision for getAt

Changes

• Dockerfile: Add EXPOSE 5432 and IMMUDB_PGSQL_SERVER to all immudb images
• README.md: Switch to github badge
• build: Update the RELEASING.md documentation
• cmd/immuclient: Remove unnecessary sleep for set commands
• cmd/immuclient: Move history command to a separate file
• cmd/immuclient: Extract separate immuclient options
• embedded/sql: validate current database as first step
• embedded/sql: check tx range edge cases
• embedded/sql: not showing unexistent db name as part of error message
• embedded/sql: fully non-transactional db creation and selection
• embedded/sql: re-include ttimestamp conversions in tx periods
• embedded/sql: de-duplicate error handling
• embedded/sql: database selection without multidb handler is still transactional
• embedded/sql: database selection as non-transactional
• embedded/sql: postpone period evaluation so to support parameters type inference
• embedded/sql: param substitution in functional datasource
• embedded/sql: functional-style catalog queries
• embedded/sql: quoted identifiers
• embedded/sql: ensure db selection is the last operation
• embedded/sql: unit testing db selection
• embedded/sql: include short database selection stmt
• embedded/sql: ensure context propagation with multiple txs
• embedded/sql: implicit time expression
• embedded/sql: functional catalog api
• embedded/sql: non-functional catalog access
• embedded/sql: wip grammar extensions to enrich temporal queries
• embedded/sql: sql tx with context
• embedded/sql: multi-db handler
• embedded/sql: detailed error messages
• embedded/store: minor refactoring time-based tx lookup
• github: Speedup push github actions
• grpc: Extend Scan API with endKey, inclusiveSeek, inclusiveEnd
• pkg/api: extend database creation response to indicate db already existed
• pkg/database: minor error renaming
• pkg/database: Rename getAt to getAtTx
• pkg/database: provide query parameters during resolution
• pkg/database: Add missing copyright header in scan_test.go
• pkg/database: contextual sql tx
• pkg/database: set multi-db handler after db initialization
• pkg/database: Improved checking of KeyRequest constraints
• pkg/database: databases catalog query yet unsupported
• pkg/database: maintain MaxKeyScanLimit for backward compatibility
• pkg/integration: Add SQL verify tests after ALTER TABLE
• pkg/integration: Cleanup and restructure SQL tests
• pkg/server: upgrade database method signature
• pkg/server: contextual sql tx

Features

• Add revision number when getting DB entries
• Calculate revision number when scanning key history
• api/schema: Add revision-based option to key query
• cmd/immuclient: Add restore operation
• cmd/immuclient: Add revision numbers when looking up key history
• cmd/immuclient: Add support for revision-based get in immuclient
• cmd/immuclient: Better error messages for invalid revision for restore command
• embedded/sql: create database if not exists
• embedded/sql: queries with temporal ranges
• embedded/sql: temporal row ranges
• embedded/sql: catalog queries
• embedded/sql: WIP - UNION operator
• embedded/sql: Implement ALTER TABLE ADD COLUMN
• embedded/store: time-based tx lookup
• embedded/store: ranged key update reading
• pkg/client: Add revision-based get request on the go client
• pkg/database: Add revision-based get request on the GRPC level
• pkg/server: support database creation from sql
• pkg/server: support database selection from sql stmt

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v1.3.0-RC1-darwin-amd64	cbdabc2bae86a489744cd51ca035a3994129b1a25470985957c21b4dab227a57
immudb-v1.3.0-RC1-darwin-arm64	a80ce4c541a1d122ce0a9ddf3648cf57e969a27f109451eddd68f8d05d30f8f3
immudb-v1.3.0-RC1-freebsd-amd64	d81651387ccdd18fca470e77faa1fcdc5c4568ca5e45c2c336cec03006fa890d
immudb-v1.3.0-RC1-linux-amd64	5abe683d0147b69c37eaa0ee14d688c02aee9be7d5583c711bbcf64066add226
immudb-v1.3.0-RC1-linux-amd64-static	7bed6d577981f8fa67b9a845c8a0451961a670ffcf23698a97ffbc35d9fbe61b
immudb-v1.3.0-RC1-linux-arm64	eedc7fbe102ead28c3801229063ac80f5e57fe7c60b5eb621fe0f4b8889b482e
immudb-v1.3.0-RC1-linux-s390x	4a8ef792a658d3003b84b57d29292ec490dee1fb430e598342d41cae0c9db92b
immudb-v1.3.0-RC1-windows-amd64.exe	7c453663e3aa01985a86e60de24e08532fd2c383b874105a0f158452d33101b4

Immuclient Binaries

File	SHA256
immuclient-v1.3.0-RC1-darwin-amd64	fe0606d638bbf80e1b20b74252952d71880e03e9b1042a4b9823453f4624d40c
immuclient-v1.3.0-RC1-darwin-arm64	d847e0022f748c58772cf5721dad8c86de55d94cb897811a64d90e41a177e3a7
immuclient-v1.3.0-RC1-freebsd-amd64	7b9f3b71c90bcd2e0cea3a1bddd89450fe821eec8613262ad028caf281c50959
immuclient-v1.3.0-RC1-linux-amd64	348dd55bf2d0c3eca0c899f950f2d86afca7ce0a595f975a0c00434bb9397e9e
immuclient-v1.3.0-RC1-linux-amd64-static	b53a25a03e8a63f5fb2421d70e29d6b10d602d102a04284b8aa5339b1ef213a1
immuclient-v1.3.0-RC1-linux-arm64	cccd2453b11316103a125d37b8ea2731fcff777b3a3aaf3b9454b0ec024b939c
immuclient-v1.3.0-RC1-linux-s390x	30cdec8ab0c1a3c896f47b5c098241578caf5fa807d4649006fbda3ddb89f0e9
immuclient-v1.3.0-RC1-windows-amd64.exe	2c8f3454528c3060f3be860060cec39feff8d7dc42bd3e5f226983e89a4038ec

Immuadmin Binaries

File	SHA256
immuadmin-v1.3.0-RC1-darwin-amd64	44b6f682f690b915020cf46f891068416639d03e46e1d0ac69ba146606984031
immuadmin-v1.3.0-RC1-darwin-arm64	ea8bba35520c6f573102a5019d6f5d227f828d61a0868e0fbef381eb4f5e7c1d
immuadmin-v1.3.0-RC1-freebsd-amd64	0521ff358e99c0106742e5445a5044d13812131c61309e40ad820a9f41d8f87c
immuadmin-v1.3.0-RC1-linux-amd64	a3cef49bfff27103d4943cdf7601232a1851c98bbf8d50e741048e57422b2eb6
immuadmin-v1.3.0-RC1-linux-amd64-static	28b42d2a858ac93181bb9d66358c19d3be711fc44c98881786cf2ad6079e5314
immuadmin-v1.3.0-RC1-linux-arm64	b28a1c183107c239156840dab0e275eea5304175f0108736d7d40c963de496f7
immuadmin-v1.3.0-RC1-linux-s390x	12817e873188d4a8b56...

v1.2.4

immudb v1.2.4

This is a small bugfix immudb release aimed at fixing issues found in the previous 1.2.3 version.

• fix issue in btree node split that would cause panics for workloads using very large and small keys at the same time
• add new option to transaction scan to avoid dereferencing reference keys - this option can be used to avoid unnecessary IO reads when reading all entries within a single transaction
• fix issue with online compaction in scenarios with large read workloads
• fix automatic relogin in immuclient binary
• release process improvements
• improvements to podman compatibility (contributed by Benjamin Gentil)

Changelog

[v1.2.4] - 2022-04-28

Bug Fixes

• Dockerfile: Fix HOME variable for podman
• cmd/immuclient: upgrade not logged in error handling
• embedded/tbtree: create nodes with the right number of children
• embedded/tbtree: Better logging in btree flush
• embedded/tbtree: Fix cleanupPercentage in SnapshotSince call
• embedded/tbtree: ensure node split is evaluated
• embedded/tbtree: split into multiple nodes
• github/push: Fix notarization of binaries
• pkg/auth: Clarify comments about token injection
• pkg/auth: Do not send duplicated authorization header
• pkg/server: include db name in flush index result

Changes

• CHANGELOG.md: remove bogus liist tag entry
• build/RELEASING.md: Update releasing docs
• cmd/immuclient: include db name when printing current state
• embedded/store: index settings validations
• embedded/tbtree: ensure node size is consistent with key and value sizes
• embedded/tbtree: rename function that calculates node size lower bound
• github: Use smaller 5-days retention for master builds
• github: Update github workflow on master / version push
• github: Update github action versions
• github/push: Build docker images after tests
• github/push: Build, test and notarize for release/v* branches
• github/push: Calcualte sha256 checksums for binaries in github
• github/push: Add quick test linux-amd64 binaries
• github/push: Add quick test for Mac x64 binaries
• github/push: Add quick test for linux-arm64 binaries through qemu
• github/push: Add quick test for linux-s390x binaries through qemu
• github/push: Run stress test before notarizing binaries
• pkg/api: txbyid with keepReferencesUnresolved option
• tools/testing: Add stress tool
• tools/testing: Add randomized key length mode for stress test tool

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v1.2.4-darwin-amd64	ee43bff63c87eb29a981ec366203092933bf1d13b791d62e26c7fe36637f9f31
immudb-v1.2.4-darwin-arm64	c00b37c6cd154247b2c58c76517ddf74fbbbe7abddade138c3b78b855fc605a8
immudb-v1.2.4-freebsd-amd64	a9e3187226b5daea62ea94d836851a910db957ea06c7cc0dcb0a81802676b2ce
immudb-v1.2.4-linux-amd64	3689e4d8bca0ed6c0785e3ec82169d07e35988534e68741c4f0335c7efb5eb73
immudb-v1.2.4-linux-amd64-static	eb9436b1dbebdc4002d459c24b56396ae3c832b7648cba2bd394cd853f956f0b
immudb-v1.2.4-linux-arm64	f915c394643bade7d0744ccc2a835080d991f3fdb97f8e20781c5c1f99751734
immudb-v1.2.4-linux-s390x	b5cb02e62b99c701aece553caa1e4cdd741e4d65697d67c1d57a93e74cb9f65b
immudb-v1.2.4-windows-amd64.exe	ff1ca6fb6ca0d8b32a429c3466a3099b136d43fc346d9f41511dd0ec623188fe

Immuclient Binaries

File	SHA256
immuclient-v1.2.4-darwin-amd64	389fbb52ada28027a943c27bebe7c962b2af47d1a1d9e79a8d47a17494790d50
immuclient-v1.2.4-darwin-arm64	d0c376392703eeaa1822a0e35530c23f3be871dd471e30ece8398d1402a8ef9f
immuclient-v1.2.4-freebsd-amd64	47d38ece837e558fd99bc5375b61082eb427ab69688da30fc200d098532c6ace
immuclient-v1.2.4-linux-amd64	afaa1345dbd345a32bb18c53ecfcff3d1da804c3a57c4dee88f52c81de46d0ac
immuclient-v1.2.4-linux-amd64-static	1157c8ff346510d6ce072a50818933d07d802df3e166d1090ebb446ef436451f
immuclient-v1.2.4-linux-arm64	054b7383a6b9b09937c26fb5493a90e2c8eeb41095807d2bcba396e8a0fc370a
immuclient-v1.2.4-linux-s390x	9b7d127817c43f9c81dff72b4abe4f1cb4155b5b728ab4ea9b1f59e9e6b45b48
immuclient-v1.2.4-windows-amd64.exe	5324f22567bf7b190936ab0205a03a25360c84b478f83cf0863d93c9f4fa9053

Immuadmin Binaries

File	SHA256
immuadmin-v1.2.4-darwin-amd64	4992a91a70148892f7677a1f35a1f74b00a113dae21247ea8919bb27fa496ba9
immuadmin-v1.2.4-darwin-arm64	8677d51e433304393db6b622e8496f7039c6f35c1c213c06a86464f253a28940
immuadmin-v1.2.4-freebsd-amd64	82c5e50b555b8e2dafa4868837762ff631bb56cfe1f1afd3e622bf4628d2eaca
immuadmin-v1.2.4-linux-amd64	41858833eab18d6379146d5d15c041c6ee001a3fff9cd5dcf086ad89c6b13664
immuadmin-v1.2.4-linux-amd64-static	8637c12c5a3aafe57431c602e45e327b06d000071b8cee1097bf58de62a679a0
immuadmin-v1.2.4-linux-arm64	9ae5928100431589a22d659dd49255195e16a52819ac580b8eabe84745b4d126
immuadmin-v1.2.4-linux-s390x	3dff9d772a21d5c25b77e7437ef7f8984905801050c3fbe776c55533a0e22d18
immuadmin-v1.2.4-windows-amd64.exe	2e1ac9b1d333e23121c497fea3e81e24e197550d621c682399c699f874e60c6c

v1.2.4-RC1

Changelog

[v1.2.4-RC1] - 2022-04-27

Bug Fixes

• Dockerfile: Fix HOME variable for podman
• cmd/immuclient: upgrade not logged in error handling
• embedded/tbtree: create nodes with the right number of children
• embedded/tbtree: Better logging in btree flush
• embedded/tbtree: Fix cleanupPercentage in SnapshotSince call
• embedded/tbtree: ensure node split is evaluated
• embedded/tbtree: split into multiple nodes
• github/push: Fix notarization of binaries
• pkg/auth: Clarify comments about token injection
• pkg/auth: Do not send duplicated authorization header
• pkg/server: include db name in flush index result

Changes

• CHANGELOG.md: remove bogus liist tag entry
• build/RELEASING.md: Update releasing docs
• cmd/immuclient: include db name when printing current state
• embedded/store: index settings validations
• embedded/tbtree: ensure node size is consistent with key and value sizes
• embedded/tbtree: rename function that calculates node size lower bound
• github: Use smaller 5-days retention for master builds
• github: Update github workflow on master / version push
• github: Update github action versions
• github/push: Build docker images after tests
• github/push: Build, test and notarize for release/v* branches
• github/push: Calcualte sha256 checksums for binaries in github
• github/push: Add quick test linux-amd64 binaries
• github/push: Add quick test for Mac x64 binaries
• github/push: Add quick test for linux-arm64 binaries through qemu
• github/push: Add quick test for linux-s390x binaries through qemu
• github/push: Run stress test before notarizing binaries
• pkg/api: txbyid with keepReferencesUnresolved option
• tools/testing: Add stress tool
• tools/testing: Add randomized key length mode for stress test tool

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v1.2.4-RC1-darwin-amd64	1e5e4ccc150acd679b1d16d9b228dc04e9021976c515ce6d947151d5c7207098
immudb-v1.2.4-RC1-darwin-arm64	2bdd12cadd094400a29fdceb6dc2706932dbb5806cb8a3a5e70ae27de5aaafbf
immudb-v1.2.4-RC1-freebsd-amd64	025b1b7993ba2b267e695d13c01f4485d900420b0475679ab17f33002f8849ab
immudb-v1.2.4-RC1-linux-amd64	7007babf447e117d8b3e4d9d67ae75fa40f52880657be09ea8139007082aba8a
immudb-v1.2.4-RC1-linux-amd64-static	46f10bef27b04c0727d142cd6f51ab826db00a8f18223e8d13cc210f016e36df
immudb-v1.2.4-RC1-linux-arm64	48b6def03a92214e8ac95549ceac07e1c5dc231a70b88de1fbfefc49de2db9dc
immudb-v1.2.4-RC1-linux-s390x	306f6c652ba4c247572a23efe576b377bad072f0445d12e4259cb4f85f60afbc
immudb-v1.2.4-RC1-windows-amd64.exe	17362575f6a2f245512ccc2faea7710c0dd4c011c874ca931a7ef047ac7713e8

Immuclient Binaries

File	SHA256
immuclient-v1.2.4-RC1-darwin-amd64	121b9bb08e25b964cd405362337d2b03174f823e3fa3c880452662a33e69dee9
immuclient-v1.2.4-RC1-darwin-arm64	cf8862b57a76adecd9f24980106d0080f8cfbbd8c0ae82364906dffec044af35
immuclient-v1.2.4-RC1-freebsd-amd64	3fb24fa6c7feff3fdbc83b9783bfa2afc1fa9580f3e48072b6adaccb1c7f60fd
immuclient-v1.2.4-RC1-linux-amd64	5d54afb6be8930d948356e740186e17b5ba35e14378488b4478fe979ea90419c
immuclient-v1.2.4-RC1-linux-amd64-static	7624eb69fbce57d11122341aae5f4b41218c9f44919ec55f99c1bb03ae7f4d5f
immuclient-v1.2.4-RC1-linux-arm64	1ba38a8193c178fa29f304b88c288565d9acca2ab2a21bd0407b2e6420360aa4
immuclient-v1.2.4-RC1-linux-s390x	31b7cd8bfd50f8d9279bd49ca5e3680f703d4ea1b43b9dd47edd949bf359344b
immuclient-v1.2.4-RC1-windows-amd64.exe	70b6af3acda3c2b3236c2f890a88d9bd89a77d767e0af5ad4991481bfb72d21d

Immuadmin Binaries

File	SHA256
immuadmin-v1.2.4-RC1-darwin-amd64	3d8fb0f8065ce4d29f9e3aa5c167fc6f9e3c0163ba3db427891b529b24ff009b
immuadmin-v1.2.4-RC1-darwin-arm64	117f0319e7511b329d3e1466a18de9c0de4ecd5cad9dc81f4935824fa82c9692
immuadmin-v1.2.4-RC1-freebsd-amd64	c6e3bf10286c5ee0ef40425fb69239edfe354adf33d1d2b48ed1934df636e6d5
immuadmin-v1.2.4-RC1-linux-amd64	eaf830c2cc97a4d89462ac93b1da05c5797ec72481aba9676c4cf5b15acd83af
immuadmin-v1.2.4-RC1-linux-amd64-static	0f749d9d9965b4eda0991b814211d575fda2f4d20462f7c9f8660ff840ee2571
immuadmin-v1.2.4-RC1-linux-arm64	509fadd4053343c7e3960465243822e6dd49ec35bb94e8f32f15d81173ed6157
immuadmin-v1.2.4-RC1-linux-s390x	061f365eca4e8b3d672a0711621d8894c3c49c0bbb1eeaf4dcc34eaaf112b57f
immuadmin-v1.2.4-RC1-windows-amd64.exe	537be6fe451928493a650b4dd24fccb2e1c4eb364f3c3cf4b0c1d6f7165d6171

v1.2.3

immudb 1.2.3

immudb v1.2.3

• Introduction of instrumentation panel via Prometheus for enhanced monitoring
• Tons of optimizations and simplified long-run maintenance functionality (self-compacting index)
• Run-time database loading/unloading
• Physical database deletion

Fixes:

• improved error handling when webconsole can't be started
• fix sorted set scans in descending order
• fix displayed name of signing key environment variable
• fixes for session handling

Improvements:

• much improved prometheus metrics related to internal btree
  • immudb_btree_cache_size, immudb_btree_cache_hit, immudb_btree_cache_miss, immudb_btree_cache_evict - for monitoring btree node cache usage
  • immudb_btree_depth, immudb_btree_inner_node_entries, immudb_btree_leaf_node_entries for monitoring btree depth and internal data distribution
  • immudb_btree_flushed_nodes_last_cycle, immudb_btree_flushed_nodes_total, immudb_btree_flushed_entries_last_cycle, immudb_btree_flushed_entries_total, immudb_btree_compacted_nodes_last_cycle, immudb_btree_compacted_nodes_total, immudb_btree_compacted_entries_last_cycle, immudb_btree_compacted_entries_total - for monitoring flush and compaction progress
  • immudb_btree_nodes_data_begin, immudb_btree_nodes_data_end for monitoring new online compaction
• SQL syntax improvements (support for <> as an alternative to !=)
• Many btree improvements:
  • faster btree insertions:
    • reduction of insertion time by 22% in our test setup
  • much better protection against unclean shutdowns (minimize the risk of full index rebuild)
  • smaller IO pressure by reducing the number of sync operations
  • more balanced distribution of entries in btree nodes
  • reduced node sizes
    • reduction of disk space occupied by index by 22% in our test setup
  • online index compaction - https://docs.immudb.io/1.2.3/develop/management.html#index-cleaning
• Non-indexable entries
• More tests and log messages
• Many database options are now configurable through the GRPC api, list of options: https://docs.immudb.io/master/develop/management.html#configuration-options
• Improved database health check endpoint
• Compatibility mode with 1.1 proofs - https://docs.immudb.io/1.2.3/operations/backwards-compatibility.html
• Conditional KV writes - https://docs.immudb.io/1.2.3/develop/reading.html#conditional-writes
• On-demand database load / unload and deletion - https://docs.immudb.io/1.2.3/develop/management.html#database-loading-unloading

Important Notes:

Updating to this version will cause regeneration of the btree index. For large databases this may take some time.

Changelog

[v1.2.3] - 2022-04-14

Bug Fixes

• cmd/immuadmin: simplify logging when flushing and compacting current db
• pkg/database: return key not found when resolving a deleted entry
• pkg/database: Return correct error for verifiedGet on deleted entries
• CI/CD: Golang compiler is not needed for building docker images
• CI/CD: Use CAS for notarization
• embedded/store: Ensure up-to-date index on constrained writes
• embedded/store: Fix early precondition checks
• embedded/tbtree: copy-on-write when increasing root ts
• immudb: Fix the name of signing key env var
• pkg: Fix tests after recent changes in API
• pkg/api: Remove unused Sync field from IndexOptions
• pkg/api: typo in kv metadata message
• pkg/api/schema: Use correct id for preconditions in SetRequest
• pkg/auth: Avoid unguarded read from user tokens map
• pkg/client: Adopt to EncodeReference changes
• pkg/client: Prevent updates with incorrect database settings
• pkg/client: Use correct response for UpdateDatabaseV2
• pkg/client/errors: Ensure FromErrors works with ImmuError instance
• pkg/client/errors: Update the list of error codes
• pkg/database: Better handling of invalid constraints
• pkg/database: Improve test coverage for KV constraints
• pkg/database: automatically set max score if not specified in desc order
• pkg/errors: Correct GRPC error mapping for precondition failure
• pkg/server: Use buffered channel for catching OS signals
• pkg/server: adjust time to millis convertion
• pkg/server: ensure sessions locks get released
• pkg/server: override default settings with existent values
• pkg/server: typo in log message
• tools/monitoring: Update grafana dashboards

Changes

• Fix typo in a comment
• Rename Constraints to Preconditions
• cleanup percentage as float value
• Update copyright to 2022
• Dockerfile: Improve dockerfile builds
• build: improve release instructions (#1100)
• cmd/immuadmin: add safety flag in delete database command
• cmd/immuclient: health command description
• embedded/ahtree: fix error message
• embedded/appendable: appendable checksum calculation
• embedded/appendable: return io.EOF if there are not enough data for checksum calculation
• embedded/appendable: discard capability
• embedded/appendable: fix typo in error message
• embedded/store: syncThld at store options
• embedded/store: Add missing Copyright header
• embedded/store: declare constants for all the options
• embedded/store: parametrized index write buffer size
• embedded/store: add synced setting in index options
• embedded/store: sync aht when syncing the store
• embedded/store: verbose data corruption error
• embedded/store: index one tx per iteration
• embedded/store: use store layer for constraint validations
• embedded/store: constraint validations with deletion and expiration support
• embedded/store: do not skip expired entries when indexing
• embedded/store: verbose logging during compaction
• embedded/store: skip expired entries during indexing
• embedded/store: improve compaction logging
• embedded/store/options: Simplify validation tests
• embedded/tbree: only insert nodes in cache when they were mutated
• embedded/tbree: use shared writeOpts
• embedded/tbree: remove obsolete property
• embedded/tbree: bump index version
• embedded/tbtree: minOffset only for non-mutated nodes
• embedded/tbtree: Add metrics for index data size
• embedded/tbtree: reduce allocations when flushing
• embedded/tbtree: use double for min offset calculation
• embedded/tbtree: data discarding with opened and older snapshots
• embedded/tbtree: improve snapshot loading and discarding
• embedded/tbtree: synced flush if cleanup percentage is greater than zero
• embedded/tbtree: Add more internal metrics
• embedded/tbtree: reduce allocs during flush
• embedded/tbtree: ensure current snapshot is synced for compaction
• embedded/tbtree: validate input kv pairs before sorting
• embedded/tbtree: Use KV entries count for sync threshold
• embedded/tbtree: no cache update during compaction reads
• embedded/tbtree: discard unreferenced data when flushing index
• embedded/tbtree: discard unreferenced data
• embedded/tbtree: middle node split
• embedded/tbtree: min offset handling
• embedded/tbtree: validate compaction target path
• embedded/tbtree: positive compaction threshold
• embedded/tbtree: discard unreferenced data after sync
• embedded/tbtree: ensure sync on gracefully closing
• embedded/tbtree: checksum-based snapshot consistency validation
• embedded/tbtree: self-healing index
• embedded/tbtree: set initial offsets during initialization
• embedded/tbtree: validate data-format version
• embedded/tbtree: Extend buckets for child node count histogram
• embedded/tbtree: reduce fixed records length
• embedded/tbtree: open-ranged nodes
• embedded/tbtree: wip reduce node size
• embedded/tbtree: use binary search during key lookups
• embedded/tbtree: fully replace sync with syncThld
• embedded/tbtree: rebase non-indexed on kv syncthreshold
• embedded/tbtree: explicit error validation before loading
• embedded/tbtree: sort kv pairs in bulkInsert
• makefile: fix cas sign instructions
• metrics: Add better flush / compaction metrics for btree
• pkg/api: prepare flushindex endpoint for future extensions
• pkg/api: use entries spec in verified and scan tx endpoints
• pkg/api: parametrized index cleanup percentage
• pkg/api: non-indexable entries
• pkg/api: use nullable prefix in db settings message
• pkg/api: add synced param to flushindex endpoint
• pkg/api: change proto schema toward db loading/unloading
• pkg/api: uniform v2 endpoints
• pkg/api: db loading and unloading
• pkg/client: optional client connection
• pkg/client: synced flushing to enable physical data deletion
• pkg/client: use txRequest in TxByIDWithSpec method
• pkg/database: tx entries excluded by default if non-null spec is provided
• pkg/database: optional tx value resolution
• pkg/database: remove db name from options
• pkg/database: use shared tx holder when resolving tx entries
• pkg/database: parameters to resolve references at tx
• pkg/integration: integrate non-indexed into grpc apis
• pkg/server: Dump used db options when loading databases
• pkg/server: replication options for systemdb and defaultdb
• pkg/server: use syncThreshold
• pkg/server: use previous store default values
• pkg/server: endpoint to retrieve settings of selected database
• pkg/server: log web-console error on boot
• pkg/server: start/stop replicator when loading/unloading db
• pkg/server: expose max opened files for btree indexing
• pkg/server: minor changes
• **pkg/server:*...

v1.2.3-RC1

Changelog

[v1.2.3-RC1] - 2022-04-13

Bug Fixes

• CI/CD: Golang compiler is not needed for building docker images
• CI/CD: Use CAS for notarization
• embedded/store: Ensure up-to-date index on constrained writes
• embedded/store: Fix early precondition checks
• embedded/tbtree: copy-on-write when increasing root ts
• immudb: Fix the name of signing key env var
• pkg: Fix tests after recent changes in API
• pkg/api: Remove unused Sync field from IndexOptions
• pkg/api: typo in kv metadata message
• pkg/api/schema: Use correct id for preconditions in SetRequest
• pkg/auth: Avoid unguarded read from user tokens map
• pkg/client: Adopt to EncodeReference changes
• pkg/client: Prevent updates with incorrect database settings
• pkg/client: Use correct response for UpdateDatabaseV2
• pkg/client/errors: Ensure FromErrors works with ImmuError instance
• pkg/client/errors: Update the list of error codes
• pkg/database: Better handling of invalid constraints
• pkg/database: Improve test coverage for KV constraints
• pkg/database: automatically set max score if not specified in desc order
• pkg/errors: Correct GRPC error mapping for precondition failure
• pkg/server: Use buffered channel for catching OS signals
• pkg/server: adjust time to millis convertion
• pkg/server: ensure sessions locks get released
• pkg/server: override default settings with existent values
• pkg/server: typo in log message
• tools/monitoring: Update grafana dashboards

Changes

• Fix typo in a comment
• Rename Constraints to Preconditions
• cleanup percentage as float value
• Update copyright to 2022
• Dockerfile: Improve dockerfile builds
• build: improve release instructions (#1100)
• cmd/immuadmin: add safety flag in delete database command
• cmd/immuclient: health command description
• embedded/ahtree: fix error message
• embedded/appendable: appendable checksum calculation
• embedded/appendable: return io.EOF if there are not enough data for checksum calculation
• embedded/appendable: discard capability
• embedded/appendable: fix typo in error message
• embedded/store: syncThld at store options
• embedded/store: Add missing Copyright header
• embedded/store: declare constants for all the options
• embedded/store: parametrized index write buffer size
• embedded/store: add synced setting in index options
• embedded/store: sync aht when syncing the store
• embedded/store: verbose data corruption error
• embedded/store: index one tx per iteration
• embedded/store: use store layer for constraint validations
• embedded/store: constraint validations with deletion and expiration support
• embedded/store: do not skip expired entries when indexing
• embedded/store: verbose logging during compaction
• embedded/store: skip expired entries during indexing
• embedded/store: improve compaction logging
• embedded/store/options: Simplify validation tests
• embedded/tbree: only insert nodes in cache when they were mutated
• embedded/tbree: use shared writeOpts
• embedded/tbree: remove obsolete property
• embedded/tbree: bump index version
• embedded/tbtree: minOffset only for non-mutated nodes
• embedded/tbtree: Add metrics for index data size
• embedded/tbtree: reduce allocations when flushing
• embedded/tbtree: use double for min offset calculation
• embedded/tbtree: data discarding with opened and older snapshots
• embedded/tbtree: improve snapshot loading and discarding
• embedded/tbtree: synced flush if cleanup percentage is greater than zero
• embedded/tbtree: Add more internal metrics
• embedded/tbtree: reduce allocs during flush
• embedded/tbtree: ensure current snapshot is synced for compaction
• embedded/tbtree: validate input kv pairs before sorting
• embedded/tbtree: Use KV entries count for sync threshold
• embedded/tbtree: no cache update during compaction reads
• embedded/tbtree: discard unreferenced data when flushing index
• embedded/tbtree: discard unreferenced data
• embedded/tbtree: middle node split
• embedded/tbtree: min offset handling
• embedded/tbtree: validate compaction target path
• embedded/tbtree: positive compaction threshold
• embedded/tbtree: discard unreferenced data after sync
• embedded/tbtree: ensure sync on gracefully closing
• embedded/tbtree: checksum-based snapshot consistency validation
• embedded/tbtree: self-healing index
• embedded/tbtree: set initial offsets during initialization
• embedded/tbtree: validate data-format version
• embedded/tbtree: Extend buckets for child node count histogram
• embedded/tbtree: reduce fixed records length
• embedded/tbtree: open-ranged nodes
• embedded/tbtree: wip reduce node size
• embedded/tbtree: use binary search during key lookups
• embedded/tbtree: fully replace sync with syncThld
• embedded/tbtree: rebase non-indexed on kv syncthreshold
• embedded/tbtree: explicit error validation before loading
• embedded/tbtree: sort kv pairs in bulkInsert
• makefile: fix cas sign instructions
• metrics: Add better flush / compaction metrics for btree
• pkg/api: prepare flushindex endpoint for future extensions
• pkg/api: use entries spec in verified and scan tx endpoints
• pkg/api: parametrized index cleanup percentage
• pkg/api: non-indexable entries
• pkg/api: use nullable prefix in db settings message
• pkg/api: add synced param to flushindex endpoint
• pkg/api: change proto schema toward db loading/unloading
• pkg/api: uniform v2 endpoints
• pkg/api: db loading and unloading
• pkg/client: optional client connection
• pkg/client: synced flushing to enable physical data deletion
• pkg/client: use txRequest in TxByIDWithSpec method
• pkg/database: tx entries excluded by default if non-null spec is provided
• pkg/database: optional tx value resolution
• pkg/database: remove db name from options
• pkg/database: use shared tx holder when resolving tx entries
• pkg/database: parameters to resolve references at tx
• pkg/integration: integrate non-indexed into grpc apis
• pkg/server: Dump used db options when loading databases
• pkg/server: replication options for systemdb and defaultdb
• pkg/server: use syncThreshold
• pkg/server: use previous store default values
• pkg/server: endpoint to retrieve settings of selected database
• pkg/server: log web-console error on boot
• pkg/server: start/stop replicator when loading/unloading db
• pkg/server: expose max opened files for btree indexing
• pkg/server: minor changes
• pkg/server: synced db runtime operations
• pkg/server: expose flush index endpoint
• pkg/server: increase default max number of active snapshots
• pkg/server: tolerate failed user-created db loading
• pkg/server: convert time to milliseconds
• pkg/serverr: validate request in deprecated database creation endpoint
• stats: Add btree cache prometheus stats

Features

• Entries-independent constraints in GRPC api
• Improved API for database creation and update
• Move KV write constraints to OngoingTX member
• Improved validation of kv constraints
• KV: Add constrained KV writes for ExecAll operation
• KV: Do not create unnecessary snapshots when checking KV constraints
• KV: Move constraints validation to OngoingTx
• KV: Add constrained KV writes for Set operation
• KV: Add constrained KV writes for Reference operation
• embedded/cache: dynamic cache resizing
• embedded/store: Fail to write metadata if proof version does not support it
• embedded/store: Add max header version used during writes
• embedded/store: non-indexable entries
• embedded/store: Add tests for generation of entries with metadata
• embedded/store: Allow changing tx header value using GRPC api.
• embedded/tbtree: decouple flush and sync by introducing syncThreshold attribute
• immuadmin: Allow changing proof compatibility from immuadmin
• kv: Update grpc protocol with KV set constraints
• pkg/api: delete database endpoint
• pkg/api: tx api with entry filtering capabilities
• pkg/client: new method to fetch tx entries in a single call
• pkg/database: Updated GRPC protocol for constrained writes
• pkg/database: Update code for new constrained write protocol
• pkg/database: add noWait attribute in get request
• pkg/server: database health endpoint
• pkg/server: support database loading/unloading
• pkg/server: new endpoint databaseSettings
• pkg/server: expose all database settings
• tools/monitoring: Add immudb Grafana dashboard
• tools/monitoring: added datasource selection, added instance selection, labels include instance, fixed calculations

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v1.2.3-RC1-darwin-amd64	c2a0d651602862d4113adaf43708175cb9979e042e0705ade5481a773f214088
immudb-v1.2.3-RC1-darwin-arm64	cfb782e84be293935adfc5c32d6bd1d617e8c8f2356b250c3753bd2db0347a0e
immudb-v1.2.3-RC1-freebsd-amd64	72fbc59b9386d3dc1840210c7813854f9f18e00631378f8e83ad525955e95f10
[immudb-v1.2.3-RC1-...

1.2.2

All binaries are notarized and can now be authenticated against the Community Attestation Service with SignerID immudb@codenotary.com.

After downloading the CAS client simply run cas authenticate binaryname --signerID immudb@codenotary.com

Changelog

[v1.2.2] - 2022-01-17

Bug Fixes

• registering connection in order to make possible conn recycling
• Dockerfile: Add ca-certificates.crt file to immudb image
• client/file_cache: Fix storing immudb state in file cache
• embedded/immustore: Avoid deadlock when acquire vLog lock
• embedded/sql: max key len validations
• embedded/sql: consider not null flag is on for auto incremental column
• pkg/server: validate if db is not replica then other replication attributes are not set
• pkg/stdlib: fix last insert id generation

Changes

• create code of conduct markdown file (#1051)
• cmd/immuclient: return actual login error
• embedded/sql: wip client provided auto-incremental values
• embedded/sql: add first and last insert pks retrivial methods
• embedded/sql: change column constraints ordering
• metrics: Add indexer metrics
• metrics: Add more s3-related metrics
• pkg/database: pre-validation of duplicated entries in execAll operation
• pkg/database: temporarily disable execall validations
• pkg/database: instantiate tx holder only in safe mode
• pkg/database: self-contained noWait execAll
• pkg/database: descriptive error messages
• pkg/replication: delay replication after failure
• pkg/stdlib: clean connection registration and leftovers

Features

• hot backup and restore
• embedded/sql: support for basic insert conflict handling
• s3: Add support for AWS V4 signatures

Downloads

Docker image
https://hub.docker.com/r/codenotary/immudb

Immudb Binaries

File	SHA256
immudb-v1.2.2-darwin-amd64	641853c5fc3c091f5e744f29282921e7c4463eeae76f169cda9d88f787f01b6d
immudb-v1.2.2-darwin-arm64	b3c867513a1682d2b4c927eaab919eced899fd5e8bca07e8e94415478e36d3a9
immudb-v1.2.2-freebsd-amd64	81c5e00e50595358e4ae1e3a6fbf32ba9aa4a75136c40f1291d57568579ba44d
immudb-v1.2.2-linux-amd64	efffe43ae1a95cfa1acd340221621a63b16fb5be2a79728e9d893d22f27f116a
immudb-v1.2.2-linux-amd64-static	93e49b7952eca6b5cdb8b58fe49fee7f841c49b74187492fa93f11d8aea0c8d6
immudb-v1.2.2-linux-arm64	f6fa5f3895731299fe4d0aec77e4313146c306df406365548fe880a9057f1cbf
immudb-v1.2.2-linux-s390x	16fa90bd42b736e40e241a90fe42eb3f4c6d849c7603f9fe71c815bcac141245
immudb-v1.2.2-windows-amd64.exe	42e49263ea0285958996e4965a60db46231d682d0f2bcb550a82dd4d639a9af6

Immuclient Binaries

File	SHA256
immuclient-v1.2.2-darwin-amd64	e57512dcba9f5cbf072bf7a53f2d06dce9d8106b06534b665583359c0f82a9e4
immuclient-v1.2.2-darwin-arm64	7ad3bee87b5438884111b4efe14a1736ea1afaa1a36c069efa69c7deafb052e4
immuclient-v1.2.2-freebsd-amd64	4e869bb48f9d2fde61584892fd7bcf779f997c90cd2d073777191b70718b0865
immuclient-v1.2.2-linux-amd64	c3361d4bea4ed52484138aef3d2e2a92bea824ab5c26ed7c26a3665c000d6519
immuclient-v1.2.2-linux-amd64-static	3e72fbba186a50432dab6827cc40e02d08915ff84b891bb85b534717b9754d10
immuclient-v1.2.2-linux-arm64	b57a4996b099b0c96b7d5f3883d7db6295b24799e43795466fe718cc709695ea
immuclient-v1.2.2-linux-s390x	bf9130d4b4ce2d2c74adf03d40f66d2c669e8a8bbeed3ee6898b63a8f2736511
immuclient-v1.2.2-windows-amd64.exe	f140e8fe37f84e4c45122b0291442edd11157bdce0a8b8cbd9f12a336571ace2

Immuadmin Binaries

File	SHA256
immuadmin-v1.2.2-darwin-amd64	5d181bc75c0bd60fd7f377e33d6b9f08e0ddb85266893e14436b7f26a0683a73
immuadmin-v1.2.2-darwin-arm64	4714c15e7e08b22b00e50501112bea7b61f03fccfce3f14eafaf14fd6ed5a5e7
immuadmin-v1.2.2-freebsd-amd64	a7f2b5566f2b691c3a36fc7d348a0f38e52a288becda05d329152eab4a39bce1
immuadmin-v1.2.2-linux-amd64	6fb772d83b19324940bb4b76d1a8c08d62dd1ae03d9d54df3600ac9d808d7cd6
immuadmin-v1.2.2-linux-amd64-static	79d707b0b63d8df1c543dce7d20fa77082ebf0773d4fc8d275d4efc41eb32a42
immuadmin-v1.2.2-linux-arm64	d0dbaad3fbe743a3f5ce0417df5d365b1be8142cdb1e4c732675bc7f5d6faed4
immuadmin-v1.2.2-linux-s390x	93e9e6bd009072b493c442703dbc5451777fedbde00314f1452443c5b84e9aa2
immuadmin-v1.2.2-windows-amd64.exe	636d76033514c7813a8be72ec3089ffe445c951aa9f238b254990bc0062c1220

v1.2.1 (ACID support, GDPR, Golang stdlib support and improved SQL compatibility)

Immudb 1.2.1 highlights

This release of immudb comes with many new features and improvements. The most important ones are highlighted below.

Improved transaction support

Starting with this version of immudb, SQL clients can now use true SQL transactions. It is now possible to start a transaction, perform multiple read and write operations and then commit or rollback changes. Before that each transaction had to be executed within one command sent to the server.

Logical deletion and expiration

KV and SQL records can now be deleted. This was enabled by introduction of metadata attached to entries that can store information about deletion or expiration. Such deleted entries are no longer available when trying to read KV or SQL entries but one can still lookup the history of a key or perform an SQL statement using time travel to fetch historical values before deletion.

In adddition to deletion, KV entries also support setting data expiration. Once expired, values of KV entries can no longer be retrieved. Expiration is a much stronger concept than deletion and once expired, immudb will protect against reading of the value for given key even if the history of an entry is retrieved.

Much improved SQL support

Due to introduction of metadata, immudb now supports the DELETE statement. Deleted rows do not appear in standard SQL queries anymore, however one can still query them using time travel feature.

This release also add support for a standard UPDATE statement that enables modifying only some columns of a row, before that the only way to update an existing entry was with the UPSERT statement that required values for all columns and could only be used with a primary key.

An improvement to the UPSERT allows updating rows for tables that have secondary indexes.

There are many more smaller improvements in the SQL engine for this release such as: indexes for columns with NULL values, native TIMESTAMP type, SELECT DISTINCT, IS [NOT] NULL, IN, basic CAST, LIKE NOT. We also improved SQL compatibility for table aliases and COUNT function.

Important notes

Breaking changes on the SQL layer

In order to support nullable columns, a breaking change for the on-disk format was introduced. After upgrading to immudb 1.2.1, the old data will not be available through the SQL interface even though immudb still contains the low-level KV representation of those entries. If you need to migrate the old data to the new format, please contact us on our discord channel.

Breaking changes in the SDK

One of the strengths of immudb is that immudb can prove to the client that the record was immutably stored in the database and that it was not tampered with. Verification of such proof is done on the client side.

Introduction of the entry metadata requires a slightly modified version of the proof that also needs to consider such metadata. An updated version of the SDK will be needed to correctly verify proofs for new entries.

There are also many changes in the API related to new transaction support. Because transaction requires server-side state, the client can now operate in a context of a server-side session. There are also many new commands related to transaction support.

Avoiding downtime during the upgrade

To avoid downtime during the upgrade, components should be upgraded in the following order:

• immudb clients
• immudb follower replica instances
• immudb main database

Changelog

[v1.2.1] - 2021-12-13

Bug Fixes

• protobuf: Fix compatibility with 1.1 version
• database: Internal consistency check on data reads
• database/meta: Do not crash on history with deleted items
• pkg/database: history skipping not found entries

Changes

• cmd/immuadmin/command: add super user login hint
• embedded/sql: use sql standard escaping with single quotes
• embedded/sql: support for escaped strings
• embedded/store: prevent value reading of expired entries
• embedded/store: fix expiration error declaration
• embedded/store: dedicated expiration error
• embedded/store: improve metadata serialization/deserialization methods
• embedded/store: validations during metadata deserialization
• embedded/store: return data corrupted error when deserialization cannot proceed
• embedded/store: use fixed time during the lifespan of a tx
• embedded/store: easily extendable meta attributes
• embedded/store: mandatory expiration filter
• embedded/store: reduce attribute code size
• pkg/auth: require admin permission to export and replicate txs
• pkg/integration: remove useless compilation tag on tests
• pkg/server: deprecate GetAuth and WithAuth
• pkg/server/sessions: session max inactivity time set to 3m and minor stat collecting fix
• pkg/server/sessions: tuning sessions params
• pkg/server/sessions: session timeout set to 2 min

Features

• embedded/store: logical entries expiration
• pkg/api: logical entries expiration
• pkg/client: expirable set

[v1.2.0-RC1] - 2021-12-07

Bug Fixes

• Update jaswdr/faker to v1.4.3 to fix build on 32-bit systems
• Makefile: Fix building immudb for specific os/arch
• Makefile: Use correct version of the grpc-gateway package
• embedded/sql: ensure determinism and no value overlaps distinct rows
• embedded/sql: fix inserting calculated null values
• embedded/sql: correct max key length validation based on specified col max length
• embedded/sql: fix rollback stmt
• embedded/sql: normalize parameters with lower case identifiers
• embedded/sql: param substitution in LIKE expression
• embedded/sql: Use correct statement for subquery
• embedded/sql: Do not modify value returned by colsBySelector
• embedded/sql: Fix SELECT * when joining with subquery
• embedded/sql: distinct row reader with limit argument
• embedded/store: release lock when tx has a conflict
• embedded/store: read conflict validation
• embedded/store: typo in error message
• pkg/auth: Fix password tests
• pkg/client: fix database name saving on token service
• pkg/database: sql exec on provided tx
• pkg/server: fix keep alive session interceptor
• testing: using pointers for job channels
• webconsole: Fix html of the default missing page.

Changes

• refining sdk client constructor and add readOnly tx guard
• Update build/RELEASING.md documentation.
• fix more tests
• decoupled token service
• token is handled internally by sdk. Remove useless code
• remove token service from client options and fix tests
• cmd/immuadmin/command: fix immuadmin token name on client creation
• cmd/immuclient: deleteKeys functioin and updates after metadata-related changes
• cmd/immuclient: temporary disable displaying hash in non-verified methods
• embeddded/tbtree: leverage snapshot id to identify it's the current unflushed one
• embedded/multierr: minor code simplification
• embedded/sql: set INNER as default join type
• embedded/sql: minor update after rebasing
• embedded/sql: Alter index key prefixes
• embedded/sql: wip rw transactions
• embedded/sql: remove opt_unique rule to ensure proper error message
• embedded/sql: minor code simplification
• embedded/sql: kept last snapshot open
• embedded/sql: Simplify row_reader key selection
• embedded/sql: Better error messages when (up|in)serting data
• embedded/sql: standardized datasource aliasing
• embedded/sql: wip sqlTx
• embedded/sql: de-duplicate tx attributes using tx header struct
• embedded/sql: fix nullable values handling
• embedded/sql: rollback token
• embedded/sql: set parsing verbose mode when instantiating sql engine
• embedded/sql: unsafe snapshot without flushing
• embedded/sql: reusable index entries and ignore deleted index entries
• embedded/sql: bound stmt execution to a single sqltx
• embedded/sql: delay index sync until fetching row by its pk
• embedded/sql: changes on tx closing
• embedded/sql: leverage metadata for logical deletion
• embedded/sql: use order type in scanSpecs
• embedded/sql: cancel non-closed tx
• embedded/sql: expose Cancel method
• embedded/sql: wip interactive sqltx
• embedded/sql: limit row reader
• embedded/sql: return map with last inserted pks
• embedded/sql: use int type for limit arg
• embedded/sql: defer execution of onClose callback
• embedded/sql: sql engine options and validations
• embedded/sql: standard count(*)
• embedded/sql: ddl stmts not counted in updatedRows
• embedded/sql: method to return sql catalog
• embedded/sql: non-thread safe tx
• embedded/sql: use current db from ongoing tx
• embedded/sql: postponing short-circuit evaluation for safetiness
• embedded/sql: wip sql tx preparation
• embedded/store: entryDigest calculation including key len
• embedded/store: conservative read conflict validation
• embedded/store: non-thread safe ongoing tx
• embedded/store: wip tx header versioning
• embedded/store: expose ExistKeyWithPrefix in OngoingTx
• embedded/store: set header version at commit time
• embedded/store: set tx as closed even on failed attempts
• embedded/store: strengthen tx validations
• embedded/store: GetWith method accepting filters
• embedded/store: handle watchersHub closing error
• embedded/store: remove currentShapshot method
• embedded/store: tx header version validations and increased max number of entries
• embedded/store: threadsafe tx
• embedded/store: filter out entries when filter evals to t...

v1.2.0-RC1

release: v1.2.0-RC1

Signed-off-by: Michele Meloni <cleaversdev@gmail.com>

v1.1.0

Immudb 1.1.0 highlights

This release of immudb comes with many new features and improvements. The most important ones are highlighted below.

Follower replication

Immudb follower replication is now stable. This enables configuring a cluster of immudb nodes where follower replicas replicate transactions committed on the leader node.

S3 storage backend

S3 storage backend is now marked stable. With the S3 backend, immudb automatically uploads the data into an S3-compatible server reducing the amount of data stored locally.

Much improved SQL backend

Following new features were added to the SQL backend:

• Multi-column indexes
• Multi-column primary keys
• Unique indexes
• Support for negative values in INTEGER columns and calculations using negative values
• Huge improvements of query optimization with indexes
• AUTO_INCREMENT primary key

Improved docker images

New release comes with changes to how our docker images are built.

The main codenotary/immudb image is now built using a scratch base to reduce the image size and increase its security.

There are two more versions of the immudb image - one based on debian (codenotary/immudb:1.1.0-bulllseye-small) and one based on almalinux (codenotary/immudb:1.1.0-almalinux-8-minimal).

Important notes

Breaking changes on the SQL layer

This release is using completely new format for storing SQL data. This change was necessary to support new features but it is not compatible with the old format.
After upgrading to immudb 1.1.0, the old data will not be available through the SQL interface even though immudb still contains the low-level KV representation of those entries. If you need to migrate the old data to the new format, please contact us on our discord channel.

Go version support

Starting with the next release our official builds and docker images will be compiled and tested using go 1.17. In order to support older systems where go 1.17 is not easily available, we are providing statically-linked binaries. We also ensure that binaries can be manually compiled with older go versions down to 1.13 (excluding tests).

Changelog

[v1.1.0] - 2021-09-22

Bug Fixes

• Minor updates to build/RELEASING.md
• Update Dockerfile.alma maintainer field
• CI: Fix building and releasing almalinux images
• Dockerfile: Fix compiling version information in docker images
• Dockerfile.rndpass: Fix building rndpass docker image
• embedded/sql: suffix endKey when scan over all entries
• embedded/sql: in-mem catalog rollback and syncing fixes
• embedded/sql: exclude length from maxKey
• embedded/sql: return error when joint table doest not exist
• embedded/sql: support edge case of table with just an auto-increment column
• embedded/sql: limit auto-increment to single-column pks
• embedded/sql: take into account table aliasing during range calculations
• embedded/sql: improve error handling and parameters validation
• embedded/sql: adjust selector ranges calculation
• embedded/sql: consider boolean type in maxKeyVal
• embedded/sql: set type any to nil parameters
• embedded/sql: fix table aliasing with implicit selectors
• embedded/sql: enforce ordering by grouping column
• embedded/store: fix constraint condition
• embedded/store: error handling when setting offset fails
• pkg: improve signature verification during audit
• pkg/stdlib: fix driver connection releasing

Changes

• Update RELEASING.md with documentation step.
• remove wip warning for fully implemented features
• Add documentation badge to README.md
• Add documentation link at the beginning of README.md
• Add documentation link to command line help outputs
• Update codenotary maintainer info
• Remove experimental S3 warning from README
• CI: Build almalinux-based immudb image
• CI: Explicitly require bash in gh action building docker images
• CI: Use buildkit when building docker images
• Dockerfile: Build a debian-based image for immudb next to the scratch one
• Dockerfile: Use scratch as a base for immudb image
• Dockerfile: Remove unused IMMUDB_DBNAME env var
• Dockerfile: Update base docker images
• Makefile: More explicit webconsole version
• Makefile: Add darwin/amd64 target
• build.md: Add info about removing webconsole/dist folder
• cmd/immuadmin: remove replication flag shortcut
• cmd/immuadmin: parse all db flags when preparing settings
• cmd/immuadmin: improve flag description and rollback args spec
• cmd/immuclient: display number of updated rows as result of sql exec
• cmd/immudb: use common replication prefix
• docker: Update generation of docker tags
• embedded: leverage kv constraint to enforce upsert over auto-incremental pk requires row to already exist
• embedded/multierr: enhace multi error implementation
• embedded/sql: fix primary key supported types error message
• embedded/sql: get rid of limited joint implementation
• embedded/sql: ignore null values when encoding row
• embedded/sql: include constraint only when insert occurs without auto_incremental pk
• embedded/sql: wip scan optimizations based on query condition and sorting
• embedded/sql: partial progress on selector range calculation
• embedded/sql: partial progress on selector range calculation
• embedded/sql: expose primary key index id
• embedded/sql: leverage endKey to optimize indexing scanning
• embedded/sql: minor code refactoring
• embedded/sql: mark catalog as mutated when using auto incremental pk
• embedded/sql: fix max key length validation
• embedded/sql: catalog loading requires up to date data store indexing
• embedded/sql: optional parenthesis when specifying single-column index
• embedded/sql: disable TIMESTAMP data-type
• embedded/sql: move index selection closer to data source in query statements
• embedded/sql: optimize integer key mapping
• embedded/sql: use plain big-endian encoding for integer values
• embedded/sql: include support for int64 parameters
• embedded/sql: minor refactoring to simplify code
• embedded/sql: minor code simplification
• embedded/sql: use int64 as value holder for INTEGER type
• embedded/sql: add further validations when encoding values as keys
• embedded/sql: move index spec closer to ds
• embedded/sql: reserve byte to support multi-ordered indexes
• embedded/sql: index prefix function
• embedded/sql: use Cols as a replacement for ColsByID
• embedded/sql: changed identifiers length in catalog
• embedded/sql: validate non-null pk when decoding index entry
• embedded/sql: limit upsert to tables without secondary indexes
• embedded/sql: remove join constraints
• embedded/sql: convert unmapIndexedRow into unmapRow with optional indexed value
• embedded/tbtree: typo in log message
• embedded/tbtree: compaction doesn't need snapshots to be closed
• embedded/tbtree: return kv copies
• embedded/tbtree: adjust seekKey based on prefix even when a value is set
• embedded/tools: update sql stress tool with exec summary
• pkg/api: changed db identifiers type
• pkg/api: use fresh id in proto message
• pkg/api: use a map for holding latest auto-incremental pks
• pkg/api: use int64 as value holder for INTEGER type
• pkg/api: use follower naming for replication credentials
• pkg/api: include updated rows and last inserted pks in sql exec result
• pkg/api: delete deprecated clean operation
• pkg/client: changed db identifiers type
• pkg/client: move unit testing to integration package to avoid circular references
• pkg/database: warn about data migration needed
• pkg/database: minor refactoring coding conventions
• pkg/database: remove active replication options from database
• pkg/database: create sql db instance if not present
• pkg/database: minor adjustments based on multi-column indexing
• pkg/database: update integration to exec summary
• pkg/database: display as unique if there is a single-column index
• pkg/database: include updated rows and last inserted pks in sql exec result
• pkg/database: warn about data migration needed
• pkg/database: minor renaming after rebase
• pkg/pgsql/server: adds pgsql server maxMsgSize 32MB limit
• pkg/pgsql/server: add a guard on payload message len
• pkg/replication: handle disconnection only within a single thread
• pkg/replication: use new context for each client connection
• pkg/replication: use info log level for network failures
• pkg/server: followers management
• pkg/server: validate replication settings
• pkg/server: nil tlsConfig on default options
• pkg/server: change max concurrency per database to 30
• pkg/server: changed default db file size and make it customizable at db creation time
• pkg/server: use replica wording
• pkg/stdLib: implementing golang standard sql interfaces
• pkg/stdlib: remove pinger interface implementation and increase code coverage
• pkg/stdlib: increase code coverage and fix blob results scan
• pkg/stdlib: immuclient options identifier(uri) is used to retrieve cached connections
• pkg/stdlib: simplified and hardened uri handling

Features

• Dockerfile for almalinux based image
• cmd/immuadmin: add replication flags
• cmd/immuadmin: add flag to exclude commit time
• embedded/multierr: implement stardard error Is & As methods
• embedded/sql: wip unique multi-column indexes
• embedded/sql: inner join with joint table and subqueries
• embedded/sql: towards more powerful joins
• embedded/sql: value expressions in row spe...

v1.0.5

Changelog

[v1.0.5] - 2021-08-02

Bug Fixes

• bind psql port to the same IP address as grpc interface (#867)
• Update crypto, sys dependencies
• consistent reads of recently written data
• embedded/ahtree: fix the full revert corner case
• embedded/store: Truncate aht before commit
• embedded/store: Don't fail to open on corrupted commit log
• embedded/store: revert change so to prevent nil assigments
• embedded/store: handle missing error case during commit phase
• embedded/store: use reserved concurrency slot for indexing
• embedded/tbtree: ensure clog is the last one being synced
• embedded/tbtree: garbage-less nodes log
• embedded/tbtree: use padding to ensure stored snapshots are named following lex order
• embedded/tbtree: flush logs containing compacted index
• embedded/tbtree: ensure proper data flushing and syncing
• pkg/client/auditor: fix and enhance state signature verification
• pkg/pgsql/server: fix boolean and blob extended query handling
• pkg/pgsql/server: hardened bind message parsing and fix leftovers
• pkg/pgsql/server/fmessages: use a variable size reader to parse fe messages
• pkg/server: initialize db settings if not present
• pkg/server: lock userdata map read
• s3: Use remote storage for index
• s3: Use remote storage for new databases
• sql/engine: Harden DecodeValue
• store/indexer: Ensure indexer state lock is always unlocked

Changes

• Better logging when opening databases
• increased coverage handling failure branches (#861)
• remove unused interceptors and add missing error code prefixes
• move sqlutils package to schema
• group user methods in a dedicated file
• Update dependencies
• appendable: Expose validation functions of appendable options
• appendable/multiapp: Introduce appendableLRUCache
• appendable/multiapp: Add hooks to the MultiFileAppender implementation
• cmd/immuclient: fix panic in immuclient cli mode
• cmd/immuclient: update error comparisson
• embedded: col descriptor with attributes
• embedded/ahtree: minor changes towards code redabilitiy
• embedded/ahtree: minor refactoring improving readability
• embedded/ahtree: auto-truncate partially written commit log
• embedded/cache: Add Pop and Replace methods to LRUCache.
• embedded/sql: initial type specialization in place
• embedded/sql: dump catalog with a different database name
• embedded/sql: Remove linter warnings
• embedded/sql: explicit catalog reloading upon failed operations
• embedded/sql: parameters type inference working with aggregations
• embedded/sql: towards non-blocking sql initialization
• embedded/sql: remove public InferParameters operations from sql statements
• embedded/sql: several adjustments and completion in type inference functions
• embedded/sql: cancellable wait for catalog
• embedded/sql: expose InferParameters function in RowReader interface
• embedded/sql: type specialization
• embedded/sql: validate either named or unnamed parameters are used within the same stmt
• embedded/store: tx metatada serialization/deserialization
• embedded/store: minor code simplification
• embedded/store: validate replicated tx against current store
• embedded/store: minor refactoring improving readability
• embedded/store: edge-case validation with first tx
• embedded/store: auto-truncate partially written commit log
• embedded/tbtree: nodes and commit prefix renaming
• embedded/tbtree: auto-truncate partially written commit log
• embedded/tbtree: use setOffset for historical data overwriting
• embedded/tbtree: enable snapshot generation while compaction is in progress
• embedded/tbtree: expose current number of snapshots
• embedded/tbtree: full snapshot recovery
• embedded/tbtree: warn if an error is raised while discarding snapshots
• pkg/api: kept simple db creation api to guarantee backward compatibility
• pkg/api: kept same attribute id in TxEntry message
• pkg/api: fix typo inside a comment
• pkg/api: comment on deprecated and not yet supported operations
• pkg/api: remove information not required to cryptographically prove entries
• pkg/auth: list of supported operations in maintenance mode
• pkg/database: support the case where database tx is not the initial one due to migration
• pkg/database: sql operations after catalog is created
• pkg/database: single-store databases
• pkg/database: method to retrieve row cursor based on a sql query stament
• pkg/database: no wait for indexing during tx replication
• pkg/database: use fixed database name
• pkg/database: sql catalog reloading on replica
• pkg/database: migrate systemdb catalog to fixed database naming
• pkg/database: internal method renaming
• pkg/database: sql catalog per database. migration from shared catalog store when required
• pkg/database: replace fixed naming with current database
• pkg/database: replace fixing naming with current database
• pkg/database: expose catalog loading operation
• pkg/database: catalog reloading by replicas
• pkg/database: parameter inference for parsed statements
• pkg/database: re-construct sql engine once catalog is ready
• pkg/database: wait for sql engine initialization before closing
• pkg/database: add IsReplica method
• pkg/database: gracefully stop by cancelling sql initialization
• pkg/database: log when a database is sucessfully opened
• pkg/database: log warning about WIP feature when using replication capabilities
• pkg/errors: immuerrors use an internal map to determine code from the message
• pkg/errors: add more error codes and add Cod prefix to codes var names
• pkg/errors: fix user operations error codes with pgsql official ones, increase coverage
• pkg/errors: add comments
• pkg/pgsql: increase server coverage
• pkg/pgsql/server: protect parameters description message from int16 overflown
• pkg/pgsql/server: increase code coverage
• pkg/pgsql/server: handle positional parameters
• pkg/pgsql/server: add max parameters value size guard and move error package in a higher level to avoid cycle deps
• pkg/pgsql/server: add bind message negative value size guards
• pkg/pgsql/server: handle empty statements
• pkg/pgsql/server: hardened INTEGER parameters conversion
• pkg/pgsql/server: increase multi inserts tests number in simple and extended query
• pkg/pgsql/server: some polishments in the state machine
• pkg/pgsql/server: simplify query machine
• pkg/pgsql/server: add a guard to check max message size and handle default case in parsing format codes in bind messages
• pkg/pgsql/server/fmessages: uniform malformed bind messages
• pkg/server: systemdb renaming
• pkg/server: disable user mgmt operations in maintenance mode
• pkg/server: move userdata lock in the inner method getLoggedInUserDataFromUsername
• pkg/server: minor updates after rebasing
• pkg/server: remove duplicated property
• pkg/server: minor adjustments after rebasing from master branch
• pkg/server: fix typo in error message
• pkg/server: remove methods moved to user file
• pkg/stream: inject immu errors
• pkg/stream: fix namings on stream api objects
• pkg/stream: use io.Reader interface
• pkg/stream: use wrapped errors

Features

• immuclient running as auditor - replace "prometheus-port" and "prometheus-host" CLI flags with "audit-monitoring-host" and "audit-monitoring-port" (int) and start a single HTTP server which exposes all the needed endpoints (GET /metrics, /initz, /readyz, /livez and /version)
• add /healthz and /version endpoints for immudb and immuclient auditor
• add immudb error package
• appendable: Add remote s3 backend
• cmd/immuadmin: update database command
• cmd/immuadmin: upgrade database creation with settings
• cmd/immuadmin: add flag to create database as a replica
• cmd/immuclient: upgrade database creation with settings
• embedded/sql: support for named positional parameters
• embedded/sql: towards leveraging readers for type inference
• embedded/sql: catalog dumping
• embedded/sql: adding method to infer typed parameters from sql statements
• embedded/sql: support for unnamed parameters
• embedded/store: WIP replicatedCommit
• embedded/store: passive waiting for transaction commit
• embedded/store: tx export and commit replicated
• pkg/api: enhanced createDatabase endpoint to specify database replication
• pkg/api: new endpoint to update database settings
• pkg/api: endpoints for exporting and replicating txs
• pkg/client: replica creation and replication API
• pkg/client: implements update database settings operation
• pkg/client: deprecate CleanIndex operation
• pkg/database: suppport runtime replication settings changes
• pkg/database: implement passive wait for committed tx
• pkg/database: parameters type inference exposed in database package
• pkg/database: db as replica and replication operations
• pkg/error: add improved error handling
• pkg/pgsql/server: add extended query messages and inner logic
• pkg/server: initial handling of database replication settings
• pkg/server: replicas and replication endpoints
• pkg/server: implements update database settings endpoint
• pkg/server: leverage maintenance mode to recover systemdb and defaultdb databases
• pkg/server: stream of committed txs
• pkg/server: enable simultaneous replication of systemdb and defaultdb
  ...