testshell_markdown_tutorial_crypto: fail #3283

mpranj · 2019-11-26T16:05:55Z

Steps to Reproduce the Problem

Build libelektra while having gpgme development files. (gpgme-devel on fedora)
I used

cmake -DBUILD_DOCUMENTATION=ON -DBINDINGS="ALL" -DBUILD_SHARED=ON -DBUILD_STATIC=ON -DBUILD_FULL=ON -DENABLE_COVERAGE=OFF -DENABLE_OPTIMIZATIONS=ON -DENABLE_DEBUG=ON -DENABLE_LOGGER=OFF -DBUILD_STATIC=ON -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" -DKDB_DB_SYSTEM="$SYSTEM_DIR" DCMAKE_INSTALL_PREFIX=./releaseInstallDir -DPLUGINS="ALL" -DTOOLS="ALL" ..

but not everything is relevant to the problem.

make run_all

Expected Result

Test #215: testshell_markdown_tutorial_crypto ........... Passed    2.58 sec

Actual Result

Seems that there is a problem with /tmp on tmpfs or similar? I get the problem on fedora and also on debian buster (in docker).

249/262 Test #215: testshell_markdown_tutorial_crypto ...........***Failed    2.58 sec
Input: /home/mpranj/workspace/libelektra/doc/tutorials/crypto.md
kdb mount test.ini user/tests ini
kdb set user/tests/password 1234
kdb file user/tests/password | xargs cat
kdb rm user/tests/password
kdb umount user/tests
kdb mount test.ini user/tests fcrypt "encrypt/key=$(kdb gen-gpg-testkey)" ini
kdb set user/tests/password 1234

ERROR - RET:
Return value “5” does not match “0”

kdb file user/tests/password | xargs cat

ERROR - RET:
Return value “123” does not match “0”

kdb rm user/tests/password

ERROR - RET:
Return value “11” does not match “0”

kdb umount user/tests
kdb mount test.ini user/tests fcrypt "sign/key=$(kdb gen-gpg-testkey)" ini
kdb set user/tests/password 1234

ERROR - RET:
Return value “5” does not match “0”

kdb file user/tests/password | xargs cat

ERROR - RET:
Return value “123” does not match “0”

kdb rm user/tests/password

ERROR - RET:
Return value “11” does not match “0”

kdb umount user/tests
kdb mount test.ini user/tests fcrypt "sign/key=$(kdb gen-gpg-testkey),encrypt/key=$(kdb gen-gpg-testkey)" ini
kdb set user/tests/password 1234

ERROR - RET:
Return value “5” does not match “0”

kdb file user/tests/password | xargs cat

ERROR - RET:
Return value “123” does not match “0”

kdb rm user/tests/password

ERROR - RET:
Return value “11” does not match “0”

kdb umount user/tests
kdb mount test.ini user/tests crypto_gcrypt "crypto/key=$(kdb gen-gpg-testkey)" base64 ini
kdb meta-set user/tests/password crypto/encrypt 1
kdb set user/tests/password 1234
kdb set user/tests/unencrypted "I am not encrypted"
kdb file user/tests/password | xargs cat
kdb meta-set user/tests/password crypto/encrypt 0
kdb file user/tests/password | xargs cat
kdb rm user/tests/unencrypted
kdb rm user/tests/password
kdb umount user/tests
shell_recorder /tmp/tmp.FLYIzi6Q4K RESULTS: 31 test(s) done 9 error(s).

—— Protocol ————————————————————————————————————————————————————
CMD: kdb mount test.ini user/tests ini
RET: 0

CMD: kdb set user/tests/password 1234
RET: 0
STDOUT: Create a new key user/tests/password with string "1234"

CMD: kdb file user/tests/password | xargs cat
RET: 0
STDOUT: password=1234

CMD: kdb rm user/tests/password
RET: 0

CMD: kdb umount user/tests
RET: 0

CMD: kdb mount test.ini user/tests fcrypt "encrypt/key=$(kdb gen-gpg-testkey)" ini
RET: 0

CMD: kdb set user/tests/password 1234
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: Sorry, module fcrypt issued the error C01100:
Resource: Renaming file /tmp/test.ini.2110573:1574783987.412609.tmpZJttxQ to /home/mpranj/.config/test.ini.2110573:1574783987.412609.tmp failed. Reason: Invalid cross-device link
ERROR: C01100

CMD: kdb file user/tests/password | xargs cat
RET: 123
=== FAILED return value does not match expected pattern 0
STDERR: cat: /home/mpranj/.config/test.ini: No such file or directory

CMD: kdb rm user/tests/password
RET: 11
=== FAILED return value does not match expected pattern 0
STDERR: Did not find the key

CMD: kdb umount user/tests
RET: 0

CMD: kdb mount test.ini user/tests fcrypt "sign/key=$(kdb gen-gpg-testkey)" ini
RET: 0

CMD: kdb set user/tests/password 1234
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: Sorry, module fcrypt issued the error C01100:
Resource: Renaming file /tmp/test.ini.2110783:1574783987.564306.tmpRJuvgG to /home/mpranj/.config/test.ini.2110783:1574783987.564306.tmp failed. Reason: Invalid cross-device link
ERROR: C01100

CMD: kdb file user/tests/password | xargs cat
RET: 123
=== FAILED return value does not match expected pattern 0
STDERR: cat: /home/mpranj/.config/test.ini: No such file or directory

CMD: kdb rm user/tests/password
RET: 11
=== FAILED return value does not match expected pattern 0
STDERR: Did not find the key

CMD: kdb umount user/tests
RET: 0

CMD: kdb mount test.ini user/tests fcrypt "sign/key=$(kdb gen-gpg-testkey),encrypt/key=$(kdb gen-gpg-testkey)" ini
RET: 0

CMD: kdb set user/tests/password 1234
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: Sorry, module fcrypt issued the error C01100:
Resource: Renaming file /tmp/test.ini.2111010:1574783987.747497.tmpti3bSR to /home/mpranj/.config/test.ini.2111010:1574783987.747497.tmp failed. Reason: Invalid cross-device link
ERROR: C01100

CMD: kdb file user/tests/password | xargs cat
RET: 123
=== FAILED return value does not match expected pattern 0
STDERR: cat: /home/mpranj/.config/test.ini: No such file or directory

CMD: kdb rm user/tests/password
RET: 11
=== FAILED return value does not match expected pattern 0
STDERR: Did not find the key

CMD: kdb umount user/tests
RET: 0

CMD: kdb mount test.ini user/tests crypto_gcrypt "crypto/key=$(kdb gen-gpg-testkey)" base64 ini
RET: 0

CMD: kdb meta-set user/tests/password crypto/encrypt 1
RET: 0

CMD: kdb set user/tests/password 1234
RET: 0
STDOUT: Set string to "1234"

CMD: kdb set user/tests/unencrypted "I am not encrypted"
RET: 0
STDOUT: Create a new key user/tests/unencrypted with string "I am not encrypted"

CMD: kdb file user/tests/password | xargs cat
RET: 0
STDOUT: unencrypted=I am not encrypted
#@META crypto/encrypt = 1
password=@BASE64IyFjcnlwdG8wMBEAAACCBjEzmVhqufXSsgK4VPRDUC9GyQxBhocVbgZwimonK+xHaRCSX/blNDSVdIoSRg0n

CMD: kdb meta-set user/tests/password crypto/encrypt 0
RET: 0

CMD: kdb file user/tests/password | xargs cat
RET: 0
STDOUT: unencrypted=I am not encrypted
#@META crypto/encrypt = 0
password=1234

CMD: kdb rm user/tests/unencrypted
RET: 0

CMD: kdb rm user/tests/password
RET: 0

CMD: kdb umount user/tests
RET: 0
————————————————————————————————————————————————————————————————

System Information

Elektra Version: master
Operating System: Fedora
Versions of other relevant software?

Further Log Files and Output

The text was updated successfully, but these errors were encountered:

mpranj · 2019-11-26T16:07:31Z

@markus2330 this succeeds on the buildserver. Is this a showstopper for you? It would be reflected in the logs of the release (kdb run_all -v > ~elektra/$VERSION/run_all 2>&1).

EDIT: I can try to do this part on a7/v2 as it maybe only fails in my scenario with fedora as host (and debian only in docker).

markus2330 · 2019-11-26T16:50:15Z

No, it is definitely not a showstopper. Most likely it is simply because some temporary file is still there from a previous run or #2957 is not completely fixed after all.

Hopefully, @petermax2 has time to fix it before 0.9.2 😉

@mpranj do you need help with release notes or something else?

petermax2 · 2019-11-26T18:49:47Z

I can not reproduce the issue when building 2bc994a from scratch using your cmake command from above.

    Start 188: testshell_markdown_tutorial_crypto 
222/235 Test #188: testshell_markdown_tutorial_crypto ...........   Passed    3.16 sec

Maybe your build directory is broken somehow. Could you please try to compile and test with a new (empty) build directory?

mpranj · 2019-11-26T18:52:31Z

@petermax2 thank you for checking it so quickly. I tried several times on my host system (fedora) and also on a debian buster docker container. I also cleaned the build directory many times.

It's always possible that it's just a problem at my end. I'll check it again later!

petermax2 · 2019-11-27T07:42:37Z

Hm strange! I think I used a Debian:stable container yesterday for building and testing. I can try again with Fedora in the evening. At the moment I don't have so much time for troubleshooting but I will have a quick look.

petermax2 · 2019-11-27T20:08:20Z

The issue is reproducable under Fedora.

The following tests FAILED:
 41 - testshell_markdown_base64 (Failed)
 56 - testshell_markdown_csvstorage (Failed)
 76 - testshell_markdown_iconv (Failed)
 78 - testshell_markdown_ini (Failed)
 93 - testshell_markdown_mini (Failed)
116 - testmod_resolver (Failed)
181 - testshell_markdown_tutorial_crypto (Failed)

Maybe this is not a crypto-tutorial specific issue. I have to investigate.

petermax2 · 2019-11-27T20:12:23Z

kdb gen-gpg-testkey can not be found. This issue is related to #3246 (KDB_EXEC_PATH).

EDIT: The analysis is only valid for testshell_markdown_tutorial_crypto. I did not check the other tests so far.

mpranj · 2019-11-27T20:15:30Z

Thank you for taking a look at this issue!

I haven't seen the other tests fail tbh. Are you sure this is the same problem? I did not even work with the installed kdb, I just ran ctest with make run_all.

In the long run we'll add some fedora docker images too (#3227), to catch something like this earlier.

petermax2 · 2019-11-27T20:22:34Z

Are you sure this is the same problem?

No, probably several different problems.

In the long run we'll add some fedora docker images too

Very good idea!

I edited my post from before to clarify what I meant with my analysis.

mpranj · 2020-04-08T11:10:18Z

The problem is not isolated to my machine. The test fails because the following rename() call fails:

libelektra/src/plugins/fcrypt/fcrypt.c

Line 231 in 263cbe6

if (rename (tmpFile, keyString (parentKey)) != 0)

The reason is that /tmp is a separate mountpoint on Fedora by default, which was not the case for Debian. Thus the file can not be renamed across different mountpoints.

I have verified that this is the problem by applying a (very dirty) patch. I would propose to keep it consistent with the resolver implementation and place the temp file in the same directory as the original file.

@petermax2 what do you think and would you have time to fix it with a proper patch?

petermax2 · 2020-04-08T18:20:34Z

The tmp directory used by fcrypt can be set via the plugin configuration.

fcrypt uses the configuration option fcrypt/tmpdir to generate paths for temporary files during encryption and decryption. If no such configuration option is provided, fcrypt will try to use the environment variable TMPDIR. If TMPDIR is not set in the environment, /tmp is used as default directory.

(Source: Plugin Documentation)

The most easiest way to mitigate on Fedora-like build jobs would be to use another directory. On the build server the local build directory can be used. What do you think?

@petermax2 what do you think and would you have time to fix it with a proper patch?

I can allocate some spare time over the weekends but what do you have in mind as a proper solution?

mpranj · 2020-04-08T18:32:04Z

I can allocate some spare time over the weekends but what do you have in mind as a proper solution?

No need to over-engineer anything. As I suggested I would put the temp file in the same directory as the target file. This is the way resolver does it when committing changes, so it would be somewhat consistent.

I am also fine with just configuring a different TMPDIR for the builds. I can implement that change myself.

We'll go with whatever you suggest here.

mpranj · 2020-04-08T18:49:34Z

From rename docu:

[EXDEV]
[CX] [Option Start] The links named by new and old are on different file systems and the implementation does not support links between file systems. [Option End]

Edit: fails with sudo with the same error:

213: CMD: kdb set user/tests/password 1234
213: RET: 5
213: === FAILED return value does not match expected pattern 0
213: STDERR: Sorry, module fcrypt issued the error C01100:
213: Resource: Renaming file /tmp/test.ini.571145:1586371945.946855.tmpHdGEMt to /root/.config/test.ini.571145:1586371945.946855.tmp failed. Reason: Invalid cross-device link
213: ERROR: C01100

petermax2 · 2020-04-08T19:07:53Z

A simplistic solution would be to replace the rename operation by a copy + remove.

mpranj · 2020-04-08T19:11:33Z

I will simply configure a different TMPDIR explicitly for the builds. If the issue reoccurs on target systems we can reopen and find a better solution.

petermax2 · 2020-04-08T19:17:02Z

This problem might occur more often, I think. We even recommend in the fcrypt documentation to remount /tmp to a RAM-disk. So we actually recommend to trigger this error straight away.

I will try to provide a proper fix.

I will simply configure a different TMPDIR explicitly for the builds. If the issue reoccurs on target systems we can reopen and find a better solution.

This is also fine as a work-around for the build servers.

markus2330 · 2020-04-09T12:48:26Z

Thank you so much for looking into it!

A simplistic solution would be to replace the rename operation by a copy + remove.

Not to replace the code but to do that if rename failed. (And to also shred the source file in that case.) Then people without RAM disc on /tmp have speed and people with RAM disc on /tmp have more security.

rename () accross file systems is not supported and causes failures. If rename () fails, fcrypt tries to perform a manual copy. See ElektraInitiative#3283 for full discussion.

cirrus: enable fcrypt on fedora again #3283

markus2330 assigned petermax2 Nov 26, 2019

mpranj modified the milestones: 0.9.1, 0.9.2 Nov 27, 2019

petermax2 added bug work in progress labels Apr 10, 2020

petermax2 mentioned this issue Apr 10, 2020

fcrypt: try manual copy if rename () fails #3390

Merged

16 tasks

mpranj closed this as completed in #3390 Apr 11, 2020

mpranj added a commit to mpranj/libelektra that referenced this issue Apr 11, 2020

cirrus: enable fcrypt on fedora again ElektraInitiative#3283

f33a1c1

mpranj added a commit that referenced this issue Apr 11, 2020

Merge pull request #3394 from mpranj/fedorafcrypt

2e74b66

cirrus: enable fcrypt on fedora again #3283

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testshell_markdown_tutorial_crypto: fail #3283

testshell_markdown_tutorial_crypto: fail #3283

mpranj commented Nov 26, 2019

mpranj commented Nov 26, 2019 •

edited

markus2330 commented Nov 26, 2019

petermax2 commented Nov 26, 2019

mpranj commented Nov 26, 2019

petermax2 commented Nov 27, 2019

petermax2 commented Nov 27, 2019

petermax2 commented Nov 27, 2019 •

edited

mpranj commented Nov 27, 2019

petermax2 commented Nov 27, 2019

mpranj commented Apr 8, 2020

petermax2 commented Apr 8, 2020

mpranj commented Apr 8, 2020

mpranj commented Apr 8, 2020 •

edited

petermax2 commented Apr 8, 2020

mpranj commented Apr 8, 2020

petermax2 commented Apr 8, 2020

markus2330 commented Apr 9, 2020

testshell_markdown_tutorial_crypto: fail #3283

testshell_markdown_tutorial_crypto: fail #3283

Comments

mpranj commented Nov 26, 2019

Steps to Reproduce the Problem

Expected Result

Actual Result

System Information

Further Log Files and Output

mpranj commented Nov 26, 2019 • edited

markus2330 commented Nov 26, 2019

petermax2 commented Nov 26, 2019

mpranj commented Nov 26, 2019

petermax2 commented Nov 27, 2019

petermax2 commented Nov 27, 2019

petermax2 commented Nov 27, 2019 • edited

mpranj commented Nov 27, 2019

petermax2 commented Nov 27, 2019

mpranj commented Apr 8, 2020

petermax2 commented Apr 8, 2020

mpranj commented Apr 8, 2020

mpranj commented Apr 8, 2020 • edited

petermax2 commented Apr 8, 2020

mpranj commented Apr 8, 2020

petermax2 commented Apr 8, 2020

markus2330 commented Apr 9, 2020

mpranj commented Nov 26, 2019 •

edited

petermax2 commented Nov 27, 2019 •

edited

mpranj commented Apr 8, 2020 •

edited