Elektrad-v2 (written in Go)

[raphi011]

Basics

Check relevant points but please do not remove entries.
Do not describe the purpose of this PR in the PR description but:

• Short descriptions should be in the release notes (added as entry in
  doc/news/_preparation_next_release.md which contains _(my name)_)
  Please always add something to the the release notes.
• Longer descriptions should be in documentation or in design decisions.
• Describe details of how you changed the code in commit messages
  (first line should have module: short statement syntax)
• References to issues, e.g. close #X, should be in the commit messages.

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar none of these points below
need to be checked.

• I added unit tests
• I ran all tests locally and everything went fine
• affected documentation is fixed
• I added code comments, logging, and assertions (see Coding Guidelines)
• meta data is updated (e.g. README.md of plugins and METADATA.ini)

Review

Reviewers will usually check the following:

• Code is conforming to our Coding Guidelines
• APIs are conforming to our Design Guidelines
• Code is consistent to our Design Decisions
• Documentation is concise and good to read
• Examples are well chosen and understandable

Labels

• Add the "work in progress" label if you do not want the PR to be reviewed yet.
• Add the "ready to merge" label if the build server is happy and also you
  say that everything is ready to be merged.

[markus2330]

The documentation is missing which makes it difficult to review the PR. Please write a README.md describing what the binding offers and how to use it (in a tutorial style).

[markus2330]

Something went wrong here (huge changes unrelated to go), please rebase.

[raphi011]

Since the go bindings are now in their own repository I've removed them from this PR and added only the elektrad-v2 changes (hence the rename of the PR).
@markus2330

[markus2330]

If I understood you correctly you will also make changes in the API. I think it is then of little use to keep the old elektrad, as it will not be able to communicate with the updated webd. So you can simply remove the old elektrad and replace it with your implementation. (Unless you want some prereleases to be merged, then we better wait until the implementation is feature-complete.)

[raphi011]

Elektradv2 is more or less feature complete (read: same features as the Elektradv1).

Still missing:

• Tests
• Integration into Cmake (where I could use some help)
• Is the current elektrad deployed automatically somehow?
• Improve current README.md

Any other suggestions or help would be appreciated? @markus2330

[markus2330]

Yes, please finish the missing parts of this list. Documentation is very sparse.

Do you already noticed differences in the performance? Does the Web UI fully work now?

Is the current elektrad deployed automatically somehow?

Yes, see scripts/jenkins/Jenkinsfile buildWebUI and deployWebUI. It uses Docker for deployment.

[raphi011]

@markus2330 I would like to test the performance of elektrad with many keys but if i import https://github.com/ElektraInitiative/blobs/tree/master/configs/real-world my `get commands are failing with these messages

Sorry, 8 warnings were issued ;(
	Sorry, module kdb issued the warning C01200:
	Installation: Dlopen failed. Could not load module libelektra-simplespeclang.so. Reason: libelektra-simplespeclang.so: cannot open shared object file: No such file or directory
	Sorry, module kdb issued the warning C01200:
	Installation: Could not load plugin simplespeclang in process plugin
	Sorry, module kdb issued the warning C01200:
	Installation: Method 'elektraProcessPlugins' for get failed
	Sorry, module kdb issued the warning C01310:
	Internal: Could not reference back to plugin system/elektra/plugins/simplespeclang
	Sorry, module kdb issued the warning C01200:
	Installation: Dlopen failed. Could not load module libelektra-journald.so. Reason: libelektra-journald.so: cannot open shared object file: No such file or directory
	Sorry, module kdb issued the warning C01200:
	Installation: Could not load plugin journald in process plugin
	Sorry, module kdb issued the warning C01200:
	Installation: Method 'elektraProcessPlugins' for error failed
	Sorry, module kdb issued the warning C01310:
	Internal: Could not reference back to plugin system/elektra/plugins/logging

Output of cmake -GNinja -DPLUGINS=ALL -DTOOLS="kdb;web" ..

-- The C compiler identification is GNU 9.2.1
-- The CXX compiler identification is GNU 9.2.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test HAS_CXX_STD
-- Performing Test HAS_CXX_STD - Success
-- compiler/linker accepts version script? TRUE
-- compiler/linker supports symbol versioning? TRUE
-- GCC detected
-- Performing Test HAS_CFLAG_MAYBE_UNINITIALIZED
-- Performing Test HAS_CFLAG_MAYBE_UNINITIALIZED - Success
-- C flags are  -std=gnu99  -Wno-deprecated-declarations  -Wstrict-prototypes  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized -Wsign-compare -Wfloat-equal
-- CXX flags are  -std=c++11  -Wno-deprecated-declarations  -Wold-style-cast -Wstrict-null-sentinel -D_GLIBCXX_USE_NANOSLEEP -Wno-missing-field-initializers -Woverloaded-virtual  -Wsign-promo  -Wno-long-long -Wpedantic -Wno-variadic-macros -Wall -Wextra -Wno-overlength-strings -Wsign-compare -Wfloat-equal -Wformat -Wformat-security -Wshadow -Wcomments -Wtrigraphs -Wundef -Wuninitialized -Winit-self -Wmaybe-uninitialized
-- You are building Elektra 0.9.0
-- Detected unix-x86_64. Use make package to build packages (TBZ2).
Sorry, I cannot find ronn (https://github.com/rtomayko/ronn), thus generating the man pages will not work. I will install the man pages directly from the repository instead.
-- Looking for mkfifo
-- Looking for mkfifo - found
-- Looking for fork
-- Looking for fork - found
-- Looking for __GNU_LIBRARY__
-- Looking for __GNU_LIBRARY__ - found
-- Include Plugin augeas
-- Include Plugin base64
-- Include Plugin blockresolver
-- Include Plugin c
-- Looking for nftw
-- Looking for nftw - found
-- Include Plugin cache
-- Include Plugin camel
-- Include Plugin ccode
-- Include Plugin conditionals
-- Include Plugin constants
-- Include Plugin counter
-- Include Plugin cpptemplate
-- Exclude Plugin crypto_openssl because OpenSSL development files not found
-- Exclude Plugin crypto_gcrypt because libgcrypt development files not found
-- Could NOT find Botan (missing: Botan_LIBRARY Botan_INCLUDE_DIR) 
-- Exclude Plugin crypto_botan because botan development files not found
-- Include Plugin csvstorage
-- Exclude Plugin curlget because Curl-dev not found
-- Exclude Plugin curlget because libcurl >= 7.28.0 required
-- Exclude Plugin curlget because OpenSSL-dev not found
-- Include Plugin date
-- Exclude Plugin dbus because dbus package not found
-- Exclude Plugin dbusrecv because dbus package not found
-- Include Plugin desktop
-- Include Plugin directoryvalue
-- Include Plugin doc
-- Include Plugin dpkg
-- Include Plugin dump
-- Include Plugin error
-- Include Plugin fcrypt
-- Include Plugin file
-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Success
-- Include Plugin filecheck
-- Looking for setmntent
-- Looking for setmntent - found
-- Include Plugin fstab
-- Exclude Plugin gitresolver because Cannot find libgit2 >= 0.24.1
-- Exclude Plugin gitresolver because OpenSSL-dev not found
-- Include Plugin glob
-- gopts: using gopts_procfs.h
-- Include Plugin gopts
-- Exclude Plugin gpgme because GPGME development files not found
-- Exclude Plugin haskell because GHC not found, please refer to the readme in src/bindings/haskell/README.md
-- Include Plugin hexcode
-- Include Plugin hexnumber
-- Include Plugin hidden
-- Include Plugin hosts
-- Include Plugin iconv
-- Include Plugin ini
-- Include Plugin internalnotification
-- Include Plugin ipaddr
-- Include Plugin iterate
-- Exclude Plugin jni because jni not found
-- Exclude Plugin journald because systemd-journal not found
-- Include Plugin keytometa
-- Include Plugin line
-- Include Plugin lineendings
-- Include Plugin list
-- Include Plugin logchange
CMake Warning at cmake/Modules/FindLua.cmake:142 (message):
  Lua executable does not match lua library version
Call Stack (most recent call first):
  cmake/Modules/FindLua.cmake:247 (verify_lua_executable_version)
  src/plugins/lua/CMakeLists.txt:5 (find_package)


Lua library: .
Lua executable: 5.3
-- Search for swig2 instead
-- Exclude Plugin lua because Lua libs (>= liblua5.1-dev) not found
-- Include Plugin macaddr
-- Include Plugin mathcheck
-- Include Plugin mini
-- Include Plugin mmapstorage_crc
-- Include Plugin mmapstorage
-- Include Plugin mozprefs
-- Include Plugin multifile
-- Include Plugin network
-- Include Plugin ni
-- Include Plugin noresolver
-- Include Plugin null
-- Looking for fgetpwent
-- Looking for fgetpwent - found
-- Looking for getline
-- Looking for getline - found
-- Looking for putpwent
-- Looking for putpwent - found
-- Include Plugin passwd
-- Looking for euidaccess
-- Looking for euidaccess - found
-- Looking for access
-- Looking for access - found
-- Looking for geteuid
-- Looking for geteuid - found
-- Looking for getegid
-- Looking for getegid - found
-- Looking for seteuid
-- Looking for seteuid - found
-- Looking for setegid
-- Looking for setegid - found
-- Looking for getpwnam
-- Looking for getpwnam - found
-- Looking for getpwuid
-- Looking for getpwuid - found
-- Looking for getgrouplist
-- Looking for getgrouplist - found
-- Looking for getgrgid
-- Looking for getgrgid - found
-- Include Plugin path
-- Include Plugin process
-- Include Plugin profile
-- Search for swig2 instead
-- Exclude Plugin python because python 3 libs (libpython3-dev) not found
-- Search for swig2 instead
-- Exclude Plugin python2 because python 2 libs (libpython-dev) not found
-- Include Plugin quickdump
-- Include Plugin range
-- Include Plugin reference
-- Exclude Plugin regexdispatcher because GHC not found, please refer to the readme in src/bindings/haskell/README.md
-- Include Plugin rename
-- Include Plugin resolver_fm_hpu_b
-- Include Plugin resolver_fm_b_b
-- Include Plugin resolver_fm_pb_b
-- Include Plugin resolver_fm_hb_b
-- Include Plugin resolver_fm_hp_b
-- Include Plugin resolver_fm_ub_x
-- Include Plugin resolver_fm_xb_x
-- Include Plugin resolver_fm_xp_x
-- Include Plugin resolver_fm_xhp_x
-- Include Plugin resolver_fm_uhb_xb
-- Include Plugin rgbcolor
-- Search for swig2 instead
-- Exclude Plugin ruby because ruby not found
-- Include Plugin shell
-- Include Plugin simpleini
-- Include Plugin spec
-- Include Plugin specload
-- Include Plugin sync
-- Include Plugin syslog
-- Exclude Plugin tcl because boost not found (please install libboost-dev)
-- Include Plugin template
-- Include Plugin timeofday
-- Include Plugin tracer
-- Include Plugin type
-- Exclude Plugin typechecker because GHC not found, please refer to the readme in src/bindings/haskell/README.md
-- Include Plugin uname
-- Include Plugin unit
-- Include Plugin validation
-- Include Plugin wresolver
-- Exclude Plugin xerces because XercesC library libxerces-c-dev not found
-- Include Plugin xmltool
-- Include Plugin yajl
-- Exclude Plugin yambi because Bison 3 (bison) not found
-- Exclude Plugin yamlcpp because yaml-cpp (libyaml-cpp-dev >= 0.6) not found
-- Include Plugin yamlsmith
-- Exclude Plugin yanlr because ANTLR 4 executable (antlr4, antlr) not found
-- Exclude Plugin zeromqrecv because package libzmq (libzmq3-dev) not found
-- Exclude Plugin zeromqsend because package libzmq (libzmq3-dev) not found
-- Looking for fnmatch
-- Looking for fnmatch - found
-- Include Binding cpp
-- Exclude Binding jna because javac (java compiler) not found, which is only included in a jdk
-- Exclude Binding haskell because excluded by category EXPERIMENTAL
-- Exclude Binding rust because excluded by category EXPERIMENTAL
-- Search for swig2 instead
-- Exclude Binding swig because neither swig2/3 found. Please install swig3.0 and set -DSWIG_EXECUTABLE=
-- Exclude Binding gi_python because excluded by category DEPRECATED
-- Exclude Binding gi_lua because excluded by category DEPRECATED
-- Exclude Binding gsettings because excluded by category EXPERIMENTAL
-- Exclude Binding gi_python because excluded by category DEPRECATED
-- Exclude Binding gi_lua because excluded by category DEPRECATED
-- Exclude Binding intercept_fs because excluded by category EXPERIMENTAL
-- Include Binding intercept_env
-- Exclude Binding io_uv because excluded by category EXPERIMENTAL
-- Exclude Binding io_ev because excluded by category EXPERIMENTAL
-- Exclude Binding io_glib because excluded by category EXPERIMENTAL
-- Include Tool kdb
-- Include Tool web
-- Looking for clearenv
-- Looking for clearenv - found
-- Looking for setenv
-- Looking for setenv - found
-- Looking for futimens
-- Looking for futimens - found
-- Looking for hsearch_r
-- Looking for hsearch_r - found
-- Looking for futimes
-- Looking for futimes - found
-- Looking for glob
-- Looking for glob - found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for ctype.h
-- Looking for ctype.h - found
-- Looking for errno.h
-- Looking for errno.h - found
-- Looking for features.h
-- Looking for features.h - found
-- Looking for locale.h
-- Looking for locale.h - found
-- Looking for stdio.h
-- Looking for stdio.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for time.h
-- Looking for time.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of int
-- Check size of int - done
-- Check size of long
-- Check size of long - done
-- Check size of size_t
-- Check size of size_t - done
-- Check size of long long
-- Check size of long long - done
-- Check size of long double
-- Check size of long double - done
-- Check size of mode_t
-- Check size of mode_t - done
-- Check size of float
-- Check size of float - done
-- Check size of double
-- Check size of double - done
-- Check size of ((struct timeval*)0)->tv_usec
-- Check size of ((struct timeval*)0)->tv_usec - done
-- Check size of ((struct stat*)0)->st_size
-- Check size of ((struct stat*)0)->st_size - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Found PythonInterp: /usr/bin/python3 (found version "3.7.4") 
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Looking for mkdtemp
-- Looking for mkdtemp - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/raphi/code/libelektra/build

[sanssecours]

…my `get commands are failing with these messages…

I think the problem is, that we removed the simplespeclang plugin in the master branch.

[raphi011]

…my `get commands are failing with these messages…

I think the problem is, that we removed the simplespeclang plugin in the master branch.

Okay - are there any other key collections I can import?

[sanssecours]

If you want a relatively large key collection you can use generated_100000.yaml:

kdb import user yamlcpp < generated_100000.yaml
# `kdb import user yanlr < generated_100000.yaml` or 
# `kdb import user yambi < generated_100000.yaml` should also work

. The data in the file does not represent real configuration data though, unless you only want to store UUIDs in your database 😊.

[markus2330]

How did you import it?

https://github.com/ElektraInitiative/blobs/tree/master/configs/real-world also contains different test data.

There is no simplespeclang involved. Maybe you have wrong mountpoints? Try kdb stash
Or you have multiple Elektra installations? Try to reinstall.

[raphi011]

How did you import it?

https://github.com/ElektraInitiative/blobs/tree/master/configs/real-world also contains different test data.

There is no simplespeclang involved. Maybe you have wrong mountpoints? Try kdb stash
Or you have multiple Elektra installations? Try to reinstall.

I ran the ./import.sh command

[raphi011]

If you want a relatively large key collection you can use generated_100000.yaml:

kdb import user yamlcpp < generated_100000.yaml
# `kdb import user yanlr < generated_100000.yaml` or 
# `kdb import user yambi < generated_100000.yaml` should also work

. The data in the file does not represent real configuration data though, unless you only want to store UUIDs in your database blush.

So the first request of kdb get // GET localhost/kdb/ (that also opens kdb) takes >20 secs, every subsequent request not more than 5-6ms.

The client however is is unresponsive and crashes because of the many requests.

[markus2330]

Thank you for looking into it. Seems like optimizations in the client are essential.

[markus2330]

@raphi011 can you please create an issue about that imports that do not work? (If the error still exists, maybe it was only some problem in your local setup.)

[raphi011]

@raphi011 can you please create an issue about that imports that do not work? (If the error still exists, maybe it was only some problem in your local setup.)

Sure: #2990

[raphi011]

@markus2330 are there any arguments against removing the key arguments in the signatures of

kdb.Open() and kdb.Close()

and only return them wrapped as an error if necessary?

So instead of

kdb.Open(parentKey)
if parentKey contains error {
...
}

you can do this:

err := kdb.Open()

if err != nil {
...
}

Would feel more like go

[markus2330]

The C++ binding allows both (with and without parentKey as argument).

[raphi011]

The C++ binding allows both (with and without parentKey as argument).

Go does not allow overloading functions, and two methods would probably be confusing. Because the key is really only for error purposes I've already removed it - I hope that is ok?

[markus2330]

It is okay!

[raphi011]

Hi @markus2330, seems like the build is green - could you take another look at the PR?

[raphi011]

@raphi011 thank you! The deployment steps seems to work as you described and I get a nice webUI.

Unfortunately now it seems to crash soon after it is started. After trying to set a key I get:

FetchError: request to http://localhost:33333/kdb failed, reason: connect ECONNREFUSED 127.0.0.1:33333

I don't really know how to debug this. Can you reproduce the problem?

i can reproduce this error, will take a look!

[raphi011]

@raphi011 thank you! The deployment steps seems to work as you described and I get a nice webUI.

Unfortunately now it seems to crash soon after it is started. After trying to set a key I get:

FetchError: request to http://localhost:33333/kdb failed, reason: connect ECONNREFUSED 127.0.0.1:33333

I don't really know how to debug this. Can you reproduce the problem?

i fixed it, could you take a look?

[mpranj]

i fixed it, could you take a look?

Thank you! Now it does not crash instantly, but when i create a few keys under user/. Specifically, it crashes when trying to create one/multiple checkbox (boolean) key(s) via web UI. (same error message as before)

[raphi011]

Any idea why the build is failing?

[mpranj]

Any idea why the build is failing?

Yes, the python2 package was removed from homebrew. I will remove from the cirrus builds. Not sure what is going on on travis-ci.

[mpranj]

@raphi011 thanks for noticing the CI problems, they are fixed on master. Please rebase because a simple build re-trigger did not work (looks like travis does not merge the changes with master).

[mpranj]

Thank you so much for putting in the work! It looks good now, no crashes even after a while. Code looks good, only minor points regarding docu.

Fixed

[mpranj]

Thank you, everything looks great! Just one question regarding the builds.

[mpranj]

@raphi011 thank you for the clarifications. Is the PR ready to merge or do you have any more changes?

[raphi011]

Merge please ✌️

[mpranj]

@raphi011 great work! Thank you for rewriting and improving elektrad. 🎉