0.8.15 Release

0.8.15 Release

• guid: 1ab4a560-c286-46d2-a058-1a8e7e208fe8
• author: Markus Raab
• pubDate: Tue, 16 Feb 2016 17:47:00 +0100

In case you do not yet know about it, here is an abstract about Elektra:

Elektra serves as a universal and secure framework to access configuration
parameters in a global, hierarchical key database.
Elektra provides a mature, consistent and easily comprehensible API.
Its modularity effectively avoids code duplication across applications
and tools regarding configuration tasks. Elektra abstracts from
cross-platform-related issues and allows applications to be aware of other
applications' configurations, leveraging easy application integration.

See http://libelektra.org

Overview

This is one of the largest release up to now. It includes many
user-visible improvements. Some highlights:

• Mounting is vastly improved:
  think about Debian's "dpkg" to "apt"-like functionality
• In previous releases you could already specify the structure
  of the configuration. Now you can also automatically enforce
  this property.
• We split the shared library libelektra into smaller parts.
  Now users can link against the parts of the library they need.
• As always, the ABI and API is fully forward-compatible.
• The release contains improvements in the
  bootstrapping process.
• We improved the ini, rename and crypto plugins.
• The tool kdb now supports bookmarks and profiles.
• The new tool kdb editor allows you to edit KDB configuration in
  your favorite text editor.
• We are glad of the new packages for Debian, Arch Linux and OpenWRT.

The same text as follows is also available
here as html
and
here on github

Global Mount

Sometimes you simply want some functionality for the whole key database.
For example, you want to enable logging or notification of configuration
changes. In previous versions, you had to change every mountpoint
individually. Even more problematic, every mountpoint created its
individual logs and notifications, without any way for someone to know
if an application has issued its last log/notification.

These problems are now solved by global plugins. The same plugins are
reused for this purpose. Also the mounting works nearly in the same way,
you only have to omit the configuration file and the mountpoint:

kdb global-mount syslog journald dbus

Voilà, from now on every configuration change gets logged to syslog
and journald. Additionally, every application gets notified via dbus.

If you want it more verbose for debugging, you can easily do so by:

kdb global-mount logchange counter

Which gives you detailed information to standard output which keys
were changed/edited/deleted. Additionally, Elektra counts how often
the counter plugin is invoked.

The underlying work for the global plugins, i.e. hooks in the core
libraries and the list plugin that allows us to use many plugins
without bloating the core was done by Thomas Waser.

It was already possible in earlier versions of Elektra to specify the
configuration of your program. Until now, this specification could be
mainly used to to generate code as described
here.
This release adds two more interesting options:

1. the spec plugin, and
2. the spec mount.

Spec Plugin

The most important global plugin that is now newly introduced with this
release (thanks to Thomas Waser) is the spec plugin. By default it
will be added for every global-mount. So for a new installation make
sure you executed at least once:

kdb global-mount

The spec plugin is a global plugin that copies metadata from the
spec-namespace to other namespaces. That means, it reads the
specification, and makes sure that the configuration conforms to it. The
actual validation is done by the many validation plugins already present.

Lets start by saying a key is a long and must have at least the
value 10:

kdb setmeta spec/example/longkey check/type long

Then we can create a key in a different namespace and see if the spec
plugin applies the meta-data correctly:

kdb set /example/longkey 25
kdb lsmeta /example/longkey

Should now at least print check/type.
By itself, this is useful for documentation of keys.
For example, the application developer says:

kdb setmeta spec/example/longkey description "Do not change"
kdb setmeta spec/example/longkey example 30

The user can retrieve this documentation by:

kdb getmeta /example/longkey description

But we want check/type to be not only a documentation, but also
enforced.

Spec Mount

Using kdb setmeta extensively and always looking out that all plugins
are mounted correctly is error-prone. So instead, one can directly mount
the plugins as specified. For the example above one simply needs:

kdb setmeta spec/example mountpoint example.ecf
kdb spec-mount /example

Now, when we try to modify /example/longkey it will be validated:

kdb set /example/longkey a
> Error (#52) [...] long [not] matched [...] a

Based on the present meta-data, the correct plugins (in this case type
because of the metadata check/type) will be loaded.

We can also create a whole specification file, first mount the
specification and then the mountpoint according the specification, e.g
when we have battery.ini in the folder $(dirname $(kdb file spec))
with following content:

[]
mountpoint = battery.ini
infos/plugins = ini

[level]
check/enum = 'critical', 'low', 'high', 'full'

Then we can use:

# mount the file itself:
kdb mount battery.ini spec/example/battery ni
# make sure all plugins are present (e.g. enum for check/enum):
kdb spec-mount /example/battery

Now lets verify if it worked:

kdb lsmeta /example/battery/level
# we see it has a check/enum
kdb getmeta /example/battery/level check/enum
# now we know allowed values
kdb set /example/battery/level low
# success, low is ok!
kdb set /example/battery/level wrong
# fails, not one of the allowed values!

The main idea of the spec-mount is: search a plugin for every
specification (meta-data) found in the spec-namespace.

General Mount Improvements

In earlier versions kdb mount failed when plugin dependencies were
not satisfied. Now dependencies will automatically be fulfilled, e.g.

kdb mount /etc/modules system/modules line

In earlier versions you would have got an error because of the missing
null plugin. Now it simply adds the needed plugins.

The plugins given in the command-line used to be real plugins. Now also
so called providers are accepted.

To make providers useful we need to actually know which plugin is the
best candidate. To get the information we added a infos/status clause
in the contract. In this clause the plugin developer adds many details
how well the plugin is tested, reviewed, documented, maintained and so
on. Based on this information, the best suited plugin will be chosen.
For example, you now can use:

kdb mount /etc/security/limits.conf system/limits augeas \
    lens=Limits.lns logging

And the best suitable logger will automatically be chosen.

The configuration variable /sw/kdb/current/plugins now allows
us to pass plugin configuration with the same syntax as the plugin
specification passed on the commandline. A subtle difference is that
thus the shell-splitting of arguments is missing, it is not possible to
include whitespaces in the plugin configuration that way.

Now it is also possible to include the same plugin multiple times and
also give them individual names. This feature is essential for
script-based plugins, e.g. you now might add:

kdb mount file.x /example/mountpoint \
    lua#resolver script=resolver.lua \
    lua#storage script=storage.lua

Furthermore, kdb mount now supports recommendations, which can be
enabled with --with-recommends. E.g. supplied to the mount command
using augeas above, comments will automatically transformed to meta-data
to avoid cluttering of the real configuration.

Library Split

Up to now, Elektra consisted only of a single shared library,
libelektra.so. Not all symbols in it were relevant to end users, for
example, some were only needed by plugins. Others were only proposed and
not yet part of the stable API. And finally, other symbols were not needed
in some situations, e.g. the plugins do not need the kdb interface.

Thus, we split libelektra.so, so that only coherent parts are in the
same library:

• libelektra-core.so only contains the KeySet data structure and
  module loading. Every binary using Elektra should link against it.
• libelektra-kdb.so contains the missing KDB symbols. Together with
  the core they contain everything declared in kdb.h.
  Michael Zehender plans to have multiple variants of libelektra-kdb.so
  that use different kinds of concurrency.
  Headerfile: <kdb.h>
• libelektra-ease.so adds functionality missing in core to make the
  life for C programmers easier.
  New headerfile: kdbease.h
• libelektra-proposal.so adds functionality proposed for core. It
  directly uses internal structures of core, thus they always need to
  have exactly the same version.
  Headerfile: kdbproposal.h

The source code is now better organized by the introduction of a libs
folder. The explanation of every library can be found in
/src/libs/.

The rationale of the library split is documented
here.
Shortly put, it was needed for further evolution and allowing us to grow
and enhance without getting a fat core.

Thanks to Manuel Mausz for fixing many bugs related to the library split
and also adapting all the bindings for it.

Benchmark

To show that the split does not make Elektra slower, Mihael Pranjić made
a small benchmark. The real time of
benchmarks/large
and revision 634ad92

1. kdb-static: Median :0.8800
2. kdb-full: Median :0.8700
3. kdb: Median :0.8700

So it seems that the split does not influence the time of running
elektrified processes.

Times were measured with time(1) and the median is calculated for 21
runs of benchmarks/large. This was done using
scripts/benchmark_libsplit.sh

Compatibility

As always, the ABI and API is fully forward-compatible, i.e. programs
compiled against an older 0.8 version of Elektra will continue to work
(ABI) and you will be able to recompile every program without errors
(API).

We added keyGetNamespace which allows us to handle all namespaces
in a switch and to iterate all namespaces. This is essential, when
a new namespace is added, thus then the compiler can warn you about
every part where the new namespace is not yet considered. We currently
do not plan to add a new namespace, but better be safe than sorry.

The internal function keyCompare now also detects any meta-data
change.

libtools was nearly rewritten. Even though it is mostly API compatible
(you should not use the low-level Backend anymore but instead use the
BackendBuilder), it is certainly not ABI compatible. If you have an
undefined symbol: _ZN3kdb5tools7Backend9addPluginESsNS_6KeySetE you need
to recompile your tool. Even the merging part has ABI incompatibility
(different size of _ZTVN3kdb5tools7merging14NewKeyStrategyE).
Unfortunately, we still cannot guarantee compatibility in libtools,
further changes are planned (e.g. implementing mounting of lazy plugins).

The python(2) and lua interfaces changed, an additional argument (the
plugin configuration) is passed to open.

The INI plugin was rewritten, so many options changed in incompatible ways.

The default format plugin (e.g. for import/export) is no longer hardcoded
to be dump. Instead KDB_DEFAULT_STORAGE will be used.
To get KDB_DEFAULT_STORAGE you can use the constants plugin:

kdb mount-info
kdb get system/info/constants/cmake/KDB_DEFAULT_STORAGE

Thanks to Manuel Mausz plugins do no longer export any method other than
elektraPluginSymbol. It now will fail if you directly linked against
plugins and did not correctly use their public interface. Please
use the module loading and access functions via the contract.

The CMake and Pkgconfig Files now only link against elektra-core and
elektra-kdb. If you used some symbols not present in kdb.h, your
application might not work anymore.

libelektra.so is still present for compatibility reasons. It should
not be used for new applications. Some unimportant parts, however,
moved to the "sugar" libraries. E.g. the symbol elektraKeyCutNamePart
is no longer part of libelektra.so.

Bootstrapping

When you use kdbOpen in Elektra as first step it reads mountpoint
configuration from /elektra. This step is the only hardcoded one,
and all other places of the KDB's tree can be customized as desired
via mounting.

The bootstrapping now changed, so that /elektra is not part of the
default configuration default.ecf (or as configured with KDB_DB_FILE),
but instead we use elektra.ecf (or as configured with KDB_DB_INIT).
This behaviour solves the problem that default.ecf was read twice
(and even differently, once for /elektra and once for /).

To be fully compatible with previous mountpoints, Elektra will still read
mountpoints from default.ecf as long as elektra.ecf is not present.

To migrate the mountpoints to the new method, simply use:

kdb upgrade-bootstrap

which basically exports system/elektra/mountpoints, then does kdb rm -r system/elektra/mountpoints so that default.ecf will be without an
mountpoint and thus the compatibility mode does not apply anymore. As
last step it will import again what it exported before.

Details are here

Plugins

We already highlighted the new spec plugin, but also other plugins
were improved at many places. Small other changes are:

• Conditionals now also support assign/condition syntax,
  thanks to Thomas Waser
• Lua and Python are not tagged experimental anymore.
  They now correctly add their configuration to the open-call.
• The plugin yajl (the json parser and generator) now also accepts the
  type string and yields better warnings on wrong types.
• Improved error message in the type plugin.

Larger changes were done in the following plugins:

INI

The INI plugin was rewritten and a huge effort was taken so that it
fully-roundtrips and additionally preserves all comments and ordering.
Currently, it is brand new. It is planned that it will replace dump
in the future.

Currently is has some minor problems when used as KDB_DEFAULT_STORAGE.
You can avoid most problems by mounting a different file into root:

kdb mount root.ini /

Read here about the details.

A huge thanks to Thomas Waser.

Rename

Thanks to Thomas Waser rename is now fixed for cascading keys.
Additionally, it does not change the sync status of the keys so
that notification plugins work properly afterwards.

Crypto

The crypto plugin is a facility for securing sensitive Keys by symmetric
encryption of the value. It acts as a filter plugin and it will only
operate on Keys, which have the meta-key „crypto/encrypt“ set.

The key derivation is still work-in-progress, so the plugin does not
work with kdb yet. A planned method for key derivation is to utilize
the gpg-agent.

For now the plugin requires the following Keys within the plugin
configuration in order to work properly:

1. /crypto/key - the cryptographic key (binary 256 bit long)
2. /crypto/iv - the initialization vector (binary 128 bit long)

Please note that this method of key input is for testing purposes only
and should never be used in a productive environment!

Thanks to Peter Nirschl, especially the patience for also supporting
different platforms where dependencies need to be handled differently.

KDB

A technical preview of a new tool was added: kdb editor allows you
to edit any part of Elektra's configuration with any editor and any
syntax. It uses 3-way merging and other stable technology, but it
currently does not provides a way to abort editing. So you only should
use it with care.

The tool kdb list now searches in the rpath for libraries and thus
will also find plugins not present at compile time (using glob).
Additionally, it sorts the plugins by infos/status score, which can
also be printed with -v. The last plugins printed are the ones ranked
highest.

When running as root, kdb will now use the system namespace when
writing configuration to cascading key names.

Long paths are cumbersome to enter in the CLI. Thus one can define
bookmarks now. Bookmarks are key-names that start with +. They are
only recognized by the kdb tool or tools that explicitly have support
for it. Applications should not depend on the presence of a bookmark. For
example, if you set the bookmark kdb:

kdb set user/sw/elektra/kdb/#0/current/bookmarks
kdb set user/sw/elektra/kdb/#0/current/bookmarks/kdb user/sw/elektra/kdb/#0/current

You are able to use:

kdb ls +kdb/bookmarks
kdb set +kdb/format ini

The kdb tool got much more robust when the initial configuration is broken,
no man page viewer is present or Elektra was installed wrongly.

The --help usage is unified and improved.

The new keyname naming conventions are now used for
configuration of the kdb-tool itself: /sw/elektra/kdb/#0/%/
and /sw/elektra/kdb/#0/current/ are additionally read. The option
-p/--profile is now supported for every command, it allows to fetch
configuration from /sw/elektra/kdb/#0/<profile>/ instead of current.
KDB is more robust when it cannot fetch its own configuration due to
KDB errors.

Coding Guidelines

Thanks to Kurt Micheli the code guidelines are
now properly documented.
Thanks to René Schwaiger we also provide a style file for clang-format.

Furthermore we unified and fixed the source:

• only use @ for doxygen commands
• always use elektra* functions for allocation
• added a file header for every file

C++11 migration

Since we now only use C++11, we applied clang-modernize which simplified
many loops and replaced many 0 to nullptr. Additionally, we added
override and default at many places.

We removed all places where we had ifdefs to use auto_ptr if no
modern smart pointer is available.

Because of these changes there is no easy way to compile Elektra without
C++11 support, except by avoiding the C++ parts all together.

The C++ KeySet now also supports a get to retrieve whole containers
at once. By specializing KeySetTypeWrapper you can support your own
containers. Currently only map<string, T> is supported (T is any type
supported by Key::get).

If you haven't removed it from your flags already, there is no use
anymore to set ENABLE_CXX11.

Documentation

The documentation was improved vastly. Most thanks to Kurt Micheli who
did a lot of editing and fixed many places throughout the documentation
Also thanks to Michael Zehender who added two paragraphs in the main
README.md.

Keynames of applications should be called /sw/org/app/#0/current,
where current is the default profile (non given). org and
app is supposed to not contain / and be completely lowercase.
Keynames are documented here.
See also here. The main
reason for long paths is the provided flexibility in the future
(e.g. to use profiles and have a compatible path for new major versions
of configuration). By using bookmarks, users should not be confronted
by it too often.

• many man pages were improved
• many typos were fixed, thanks to Pino Toscano!
• Fix documentation for kdb list, thanks to Christian Berrer
• Compilation variants are explained better,
  thanks to Peter Nirschl for pointing out what was missing
• document ronn as dependency, thanks to Michael Zehender
• fix broken links, thanks to Daniel Bugl

Thanks to Kurt Micheli, developers are now warned during compilation on
broken links in Markdown. The output format is the same as for gcc.
Additionally, the markdown documentation of plugins now get a proper
title in the pdf and html output of doxygen.

Qt-Gui 0.0.10

Raffael Pancheri again updated qt-gui with many nice improvements:

• update existing nodes in case the config was changed outside the gui
• safely update tree
• add update signal to metadata
• fix crash when closing the GUI
• move Actions in separate file to make main.qml less clustered
• plugins do not output messages at startup
• BackendBuilder is now used, which automatically takes cares of the correct ordering of plugins
• Qt 5.4 compatibility is still ensured
• C++11 is now used in Qt-Gui, too

Packaging and Build System

Elektra 0.8.14 now in Debian with qt-gui, man pages,
thanks to Pino Toscano
read more here

Thanks to Gustavo Alvarez for updating and splitting the packages on
Arch Linux!

Thanks to Harald Geyer,
Elektra is now packaged for OpenWRT. We fixed a number of
cross-compilation issues and now officially support building against musl
libc, with one minor limitation: RPATH works differently on musl so you
need to install all plugins directly in /usr/lib/ or set LD_LIBRARY_PATH.
Report any bugs in
Harald's OpenWRT packaging issue tracker.

• export errors/symbols are now called elektra-export-symbols and
  elektra-export-symbols and can be installed using INSTALL_BUILD_TOOLS
  (by default off). This is needed for cross-compilation.
  Thanks to Harald Geyer for reporting.
• some header files are renamed because they endlessly included themselves
  if the header files were found in wrong order.
  Thanks to Harald Geyer for reporting.
• fixed compilation when built on more than 20 cores with >= -j15,
  thanks to Gustavo Alvarez for reporting and Manuel Mausz for analyzing
• lua 5.1 now works too (except for iterators),
  thanks to Harald Geyer for reporting.
  thanks to Manuel Mausz for adding a new FindLua.cmake
• pdf builds do not fail due to half written files,
  reported by René Schwaiger and fixed by Kurt Micheli

Read about other packages here.

Fixes and Improvements

• 3 way merge now properly deals with binary data,
  thanks to Felix Berlakovich
• getenv: fix wrapping on powerpc, thanks to Pino Toscano
• markdownlinkconverter: fix char/int mismatch, thanks to Pino Toscano
• wresolver: use KDB_MAX_PATH_LENGTH instead of PATH_MAX,
  thanks to Pino Toscano
• Cleaning up #ifdefs that break statements, thanks to Romero Malaquias
• Daniel Bugl tested the INI plugin
• cmake list_filter was broken because of different behaviour in
  cmake_parse_arguments, thanks to Christian Berrer for reporting
• g++5.3 is now supported
• gtest does not link against pthread if not needed
• testcases that are built with BUILD_SHARED also successfully work
• kdb list works when libs are in same path as plugins,
  thanks to Harald Geyer for reporting
• fix Mac OS issues,
  thanks to Peter Nirschl, René Schwaiger and Mihael Pranjic
• fix resolver-baseflag docu, thanks to Harald Geyer for reporting
• do not create wrong directories called ( and ) in source,
  thanks to René Schwaiger
• fix cmake for systems where iconv is not part of libc,
  thanks to Michael Zehender and Peter Kümmel (for FindIconv.cmake)
• fix segfault in libgetenv if root keys are present
• lua: fix Key:tostring(), thanks to Manuel Mausz
• add list of supported bindings,
  thanks to Manuel Mausz

Get It!

You can download the release from
here
and now also here on github

• name: elektra-0.8.15.tar.gz
• size: 2338297
• md5sum: 33ec1e5982fb7fbd8893bf7b579b80f0
• sha1: 6b1fdd5aa5aaad6ba377b4bb5ef437e0c85319ff
• sha256: 6a406986cecb8d4a44485ced118ee803bc039b0824b72298e123b4dd47eb0b22
• sha512: 86a408dd546b33e3b437f92f415de7aee6a235189f9eab0762b3f44ab4c453ee369a53de10a9f5b0df1b446460b12c57c6b8b77c282648ec2a49f2328d9af13d

This release tarball now is also available
signed by me using gpg

already built API-Docu can be found here

Stay tuned!

Subscribe to the
RSS feed
to always get the release notifications.

For any questions and comments, please contact the
Mailing List
the issue tracker on github
or by mail elektra@markus-raab.org.

Permalink to this NEWS entry

For more information, see http://libelektra.org

Best regards,
Markus