RELEASE Scylla 0.19 Beta
Tzach Livyatan edited this page Mar 10, 2016
·
5 revisions
The Scylla team is pleased to announce the release of Scylla version 0.19
Scylla is an open source, Cassandra-compatible NoSQL database, with superior performance and consistent low latency. This version is an incremental improvement release on the beta series, focus on stability and bug fixes.
Noteworthy updates are:
-
iotune a new program which run once as part of Scylla installation, analyze Scylla data drive and extract optimal values for the number of io queues and max io requests. The results are written to /etc/scylla.d/io.conf for Scylla to use.
-
Enable rolling upgrade with formal specification of internal protocols. Starting from rel 0.19 and forward, all internal protocols, between nodes and to the disk, are defined by an IDL specification. This have two advantages:
- Make backward compatibility between Scylla version easier to define, implement and test. With the new specification, we can now guarantee internal communication will not be break unintentionally by upgrades, allowing users to safely execute rolling upgrade of Scylla releases.
- Make it easier to create 3rd party tools to listen and interact with these protocol
-
IO scheduler Scylla support for system with slow disks was improved with the introduction of the new I/O scheduler. The new scheduler balances between I/O requests, including commitlog, flush and compaction, to make sure all get their fair share. If I/O devices are too slow, it propagates the back pressure to the client, slowing down the request response rate to match the disk rate. In particular, this streamlines Scylla’s latency for slower disk cases, usually the case for SSDs in public cloud and HDDs.
-
scyllatop A new CLI tool provide key scylla metrics on the terminal. To also enable an external monitoring server, like collectd, graphite or similar, just update your local collectd server to act like a proxy.
-
Move to latest release of seastar and scylla-jmx Full list of scylla, scylla-jmx contributions below.
Get started with Scylla 0.19 here.
If you have any questions about the new release, please post to the scylladb-users mailing list.
Scylla Contributions
Amnon Heiman (12):
managed_bytes: don't return auto from non-member function
serializer: Add boost::variant, chrono::time_point and unknown variant
bytes_ostream: Allow place holder return a stream
Adding the serialization_visitors.hh file
serialization_visitors: Add skip template
idl-compiler: add writers and view to classes.
Adding the mutation idl
idl-compiler: Verify member type
serialization_visitors: Adding vector_position struct
idl-compiler: Add pos and rollback to serialize vector
idl-compiler: Add optional support
idl-compiler: change optional vector implementation
Asias He (21):
Revert "Revert "streaming: Send mutations on all shards""
locator: Fix get token from a range<token>
failure_detector: Fix false alarm of "Not marking nodes down due to local pause of"
messaging_service: Add GOSSIP_DIGEST_ACK
gossip: Make gossip message handling async
gossip: Log the error when fails to add local application state
gossip: Make add_local_application_state safe to call on any cpu
storage_service: Use get_gossip_status to get status
storage_service: Get rid of assert
gossip: Get rid of assert
streaming: Add get_progress_on_all_shards for plan_id
streaming: Log number of bytes sent and recevied when stream_plan completes
streaming: Remove unused progress() function
streaming: Fix stream_manager progress api
system_keyspace: Implement increment_and_get_generation
storage_service: Use increment_and_get_generation to get generation
storage_service: Do not take api lock for get_load_map
storage_service: Make replace-node error msg more friendly
streaming: Fix keep alive timer progress checking
storage_service: Fix run with api lock
gossip: Send syn message in parallel and do not wait for it
Avi Kivity (29):
managed_bytes: fix operator== for fragmented blobs
managed_bytes: don't copy data during hashing
managed_bytes: introduce with_linearized_managed_bytes()
key: relax some APIs to accept partition_key_view instead of const partition_key&
key: tighten partition_key::representation() to return a const managed_bytes&
Merge seastar upstream
Merge seastar upstream
build: build seastar's iotune
row_cache: run partiton-touching operations of row_cache::update in a linearization context
memtable: use managed_bytes linearization context when applying mutations
managed_bytes: get rid of explicit linearize/scatter
Merge seastar upstream
Merge seastar upstream
Update scylla-ami submodule
dist: Restart scylla on abnormal termination
Merge seastar upstream
dist: build only scylla and iotune binaries, not all the tests
Merge seastar upstream
Merge seastar upstream
commitlog: fix buffer size calculation
Merge seastar upstream
Update scylla-ami submodule
Merge seastar upstream
Merge seastar upstream
main: sanity check cpu support
Merge seastar upstream
build: add order-only dependency between building antlr .o and IDL headers
Merge seastar upstream
Merge seastar upstream
Calle Wilund (19):
IDL: Add idl definitions for replay_position and truncation_record
serializer_impl: Add convinience wrapper for one-obj deserialization
system_keyspace: Use IDL types + better verification of truncation record
db/serializer: Fix db::serializer<replay_position> format
system_keyspace: Always retain highest truncation time stamp
database::truncate/drop: Move time stamp generation to shard
utils::jointpoint: Helper type to generate a singular value for all shards
truncate: Require timestamp join-function to ensure equal values
database: Fix use and assumptions about pending compations
commitlog: Fix reserve counter overflow
auth: make scheduled tasks explicity cancellable
storage_service: Explicitly shutdown "auth" on system drain
cql_test_env: Shutdown auth on test stop
commitlog: Make segment->segment_manager relation shared pointer
managed_bytes: Change init of small object to silence error on gcc5
commitlog: Take segment_manager locks first in write/flush
commitlog: Clear reserve segments in orphan_all
commitlog: Ensure segment survives whole flush call
lists.cc: fix update insert of frozen list
Glauber Costa (4):
memtable reader: also include a priority class
database: use a reference instead of a pointer to store the priority classes
database: add a priority class to a few SSTable readers
mutation_query: attach the query priority read when reading mutations
Gleb Natapov (7):
Fix read_timeouts storage_proxy counter
Yield execution in mutation_result_merger
storage_proxy: fix cl achieved condition in digest resolver timeout handler
storage_proxy: store only one data reply in digest resolver.
storage_proxy: fix race between read cl completion and timeout in digest resolver
log: add space between log level and date in the outpu
fix EACH_QUORUM handling during bootstrapping
Nadav Har'El (4):
repair: stop ongoing repairs during shutdown
repair: use seastar::gate
repair: options for repairing only a subrange
Repair: don't use freeze() to calculate mutation checksums
Paweł Dziepak (42):
Revert "do not use boost::multiprecision::msb()"
storage_service: don't drain twice
serializer: fix duration deserializer
add set_size() overload for bytes_ostream()
idl-compiler: move writers and view to *.impl.hh
tests: add idl unit test
idl-compiler: add reindent() function
idl-compiler: allow auto-generated serializers in writers
idl-compiler: accept both bytes and bytes_view
idl-compiler: avoid copy of basic types
types: make serialize_mutation_form() static
column_mapping: add column_at()
frozen_mutation: switch to bytes_ostream
mutation_partition: switch serialization to IDL-based one
idl-compiler: add serialization callback interface
frozen_mutation: use IDL-based serialization
schema: make column_mapping serializer-friendly
canonical_mutation: switch to IDL-based serialization
commitlog: move implementation to *.cc file
data_output: add reserve()
commitlog: use IDL-based serialization for entries
column_mapping: drop old-style serializers
canonical_mutation: prepare for auto-generated serializers
batchlog_manager: use new canonical_mutation serializers
schema_mutations: prepare for auto-generated serializers
frozen_schema: use IDL-based serialization
canonical_mutation: drop old serializers
schema_mutations: drop old serializers
commitlog: add commitlog entry move constructor
tests/idl: add test for stdx::optional<> serialization
tests/commitlog: use unaligned_cast instead of reinterpret_cast
cql3: don't insert row marker if schema is_cql3_table()
mutation_partition_serializer: remove read() and read_as_view()
keys: remove old-style serializers
remove db/serializer.hh includes
db: remove old-style serializers
messaging_service: add SCHEMA_CHECK verb
storage_proxy: handle SCHEMA_CHECK verb
gms: add inet_address::to_sstring()
storage_service: implement describe_schema_versions
api: use proper describe_schema_versions implementation
lsa: set _active to nullptr in region destructor
Pekka Enberg (15):
transport: Remove read_unsigned_short() variant
cql3: Return const reference from ut_name::get_keyspace()
cql3: Make 'CREATE TYPE' error message human readable
cql3/functions: Fix is_pure() for native scalar functions
tools/scyllatop: Use 'erase' to clear the screen
tools/scyllatop: Sort metrics by name
transport: Add result metadata to PREPARED message
cql3: Fix quadratic behavior in update_statement::parsed_insert::prepare_internal()
Revert "database: Fix use and assumptions about pending compations"
service/storage_service: Use logger for CQL listening report
cql3: Fix duplicate column definition check
db/system_keyspace: Remove ifdef'd code
release: prepare for 0.19
Update scylla-ami submodule
types: Implement to_string for timestamps and dates
Raphael S. Carvalho (10):
sstables: improve log message when a sstable is sealed
sstables: keep track of compacting sstables in compacton manager itself
compaction manager: add support to wait for termination of cleanup
start using notation ks.cf everywhere
database: make column_family::rebuild_sstable_list safer
sstables: make compaction manager shutdown less verbose
sstables: use system clock's epoch for timestamp in compaction history
tests: sstable_test: submit compaction request through column family
Revert "Revert "database: Fix use and assumptions about pending compations""
sstables: fix lack of accuracy in disk usage report
Shlomi Livne (2):
Revert "dist: remove AMI entry from sysconfig, since there is no script refering it"
dist: align ami option with others (-a --> --ami)
Takuya ASADA (53):
dist: switch AMI base image to 'CentOS7-Base2', uses CentOS official kernel
dist: add posix_net_conf.sh on Ubuntu package
dist: support unstable repository on scylla_install_pkg
dist: generalize scylla_ntp_setup, drop '-a' option (means AMI) from it
dist: don't use -a option for scylla_bootparam_setup since it was removed
dist: remove AMI entry from sysconfig, since there is no script refering it
dist: move selinux setup code to scylla_selinux_setup
dist: add options to skip running setups on scylla_setup, support long options
dist: long options for scylla_coredump_setup
dist: long options for scylla_ntp_setup
dist: long options for scylla_raid_setup
dist: long options for scylla_sysconfig_setup
dist: don't ignore posix_net_conf.sh error
dist: long options for scylla_install_pkg
dist: remove AMI initialization code from scylla_setup, move to scylla-ami
dist: long options for build_ami.sh
dist: add --unstable on build_ami.sh
dist: long options for build_rpm.sh
dist: delete build_ami_local.sh, merge it to build_ami.sh
dist: interactive scylla_setup
dist: call scylla_install_ami directly from scylla.json
dist: On Ubuntu, log coredump before creating the core
dist: add iotune on .rpm/.deb packages
dist: use /usr/bin/python3.4 for idl-compiler.py on CentOS
dist: restart scylla on abnormal termination (Ubuntu)
main: notify service start completion ealier, to reduce systemd unit startup time
dist: introduce scylla-io-setup.service to setup io parameters on first startup
Revert "dist: align ami option with others (-a --> --ami)"
Revert "Revert "dist: remove AMI entry from sysconfig, since there is no script refering it""
Revert "dist: update SCYLLA_IO with params for AMI"
Revert "dist: introduce SCYLLA_IO"
dist: add comment about /etc/scylla.d/io.conf on sysconfig
scyllatop: output error message both on log file and stdout
scyllatop: add --logfile argument to specify path to log file
dist: Add collectd configuration support on .rpm/.deb
dist: add scyllatop on .rpm/.deb
dist: permission error fixed on scylla_io_setup
dist: add scyllaio-setup.service for %systemd_post and %systemd_preun on .rpm
dist: run scylla-io-setup.service before scylla-server.service on anycase
dist: install example io.conf by default
dist: don't run iotune on developer mode
dist: add user-defined prefix for AMI name
dist: add support scylla_io_setup for Ubuntu
dist: fix initctl start scylla-server failed on Ubuntu
Revert "dist: downgrade g++ to 4.9 on Ubuntu"
configure.py: add support --static-stdc++ for seastar binaries (iotune)
log: make log.cc able to compile with g++-4.9
dist: show message to use XFS for scylla data directory and also notify about developer mode, when iotune fails
dist: add /etc/scylla.d/io.conf on Ubuntu
Revert "Revert "Revert "dist: remove AMI entry from sysconfig, since there is no script refering it"""
Revert "Revert "dist: align ami option with others (-a --> --ami)""
dist: export sysconfig for scylla-io-setup.service
dist: export all entries on /etc/default/scylla-server on Ubuntu
Tomasz Grabiec (58):
abstract_replication_strategy: Fix generation of token ranges
cql3: Fix crash when parsing collection condition
storage_proxy: Simplify condition
cql3: Avoid using partition_key::from_bytes()
keys: Make constructor from bytes private
keys: Avoid unnecessary construction of temporary 'bytes' object
keys: Introduce partition_key::validate()
validation: Fix validation of empty partition key
query: Split send_timestamp_and_expiry into two separate options
Rename serialization_format to cql_serialization_format
cql_serialization_version: Abstract away collection format changes
cql_serialization_format: Make it CQL protocol version aware
query: Add cql_format property to partition_slice
types: Introduce data_value::serialize()
cql3: Fix handling of lists with static columns
query: Serialize collection cells using CQL format
cql3: select_statement: Set desired serialization format
compound: Throw exception when key is too large rather than abort
compound: Extract size_type alias
transport: server: Size check should allow max value
transport: server: 'short' should be unsigned
transport: server: Throw instead of abort on bounds check failures
transport: server: Fix typo
bytes_ostream: Avoid recursion when freeing chunks
db: Fix error handling in populate_keyspace()
cql: create_table_statement: Optimize duplicate column names detection
sstables: Do not write corrupted sstables when column names are too large
mutation_query: Don't include empty partitions
schema_builder: Fluentize property setters
tests: mutation_query: Add test for dropping partitions with expired tombstones
test.py: Print output on timeout as well
mutation_partition_view: Fix use-after-move on visitor instance
mutation_partition_visitor: Fix crash for large blobs
tests: Fix build of memory_footprint
idl-compiler: Put serializers inside template class specializations
Relax includes
Add missing includes
serializer: Introduce as_input_stream(bytes_view)
idl-compiler: Catch un-closed frame errors sooner
serializer: Fix wrong size_type being serialized into the placeholder
tests: memory_footprint: Report size in query results
query: result_view: Introduce do_with()
idl-compiler: Introduce add(*_view) on vector
query: Switch to IDL-generated views and writers
row_cache: Clear partitions with region locked
row_cache: Delete the entry even if invalidation failed
row_cache: Let the cleanup guard do invalidation of unmerged partitions
idl-compiler: Make sub-streams stored in views be properly bounded
query: Move implementaion of result_merger to .cc file
cql_serialization_format: Print version as integer instead of char
to_string: Make std::experimental::optional printable
query_pagers: Fix dereference of potentially disengaged _last_ckey optional
log: Prepend log lines with timestamp when printed to stdout
cql3: create_table_statement: Sort _defined_names by text
schema_tables: Log results of schema merge
serializer: Add missing include
bytes_ostream: Drop methods which serialize integers
log: Fix operator<<(std::ostream&, const std::exception_ptr&)
Vlad Zolotarov (3):
locator: token_metadata: add get_all_endpoints_count()
locator: added EverywhereReplicationStrategy
locator: everywhere_replication_strategy: change the class_registrator name to "EverywhereStrategy"
Yoav Kleinberger (5):
ScyllaTop: top-like tool to see live scylla metrics
bugfix: fixed broken --print-config option
tools/scyllatop: support glob patterns to specifiy metrics
tools/scyllatop: correct mistake in help text
tools/scyllatop: fix mistake in help message
Scylla JMX
Amnon Heiman (4):
MessagingService: Ignore exception on the dropped messages thread
StorageService: Sort the results of getTokenToEndpointMap
ProgressInfo: Add creation from json object and json array
SessionInfo: Add receiving_files and sending_files support
Nadav Har'El (1):
scylla-jmx: implement forceRepairRangeAsync
Pekka Enberg (1):
release: prepare for 0.19