Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mmapstorage: store the OPMPHM #3136

Merged
merged 46 commits into from
Nov 6, 2019
Merged

Conversation

mpranj
Copy link
Member

@mpranj mpranj commented Oct 30, 2019

Store the OPMPHM and generally improve mmapstorage (some critical fixes and sanity checks).
Disable testmod_zeromqsend.
Increase Jenkins CI timeout #3157.

Basics

These points need to be fulfilled for every PR:

  • Short descriptions of your changes are in the release notes
    (added as entry in doc/news/_preparation_next_release.md which
    contains _(my name)_)
    Please always add something to the release notes.
  • Details of what you changed are in commit messages
    (first line should have module: short statement syntax)
  • References to issues, e.g. close #X, are in the commit messages.
  • The buildservers are happy.
  • The PR is rebased with current master.

If you have any troubles fulfilling these criteria, please write
about the trouble as comment in the PR. We will help you.
But we cannot accept PRs that do not fulfill the basics.

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar none of these points below
need to be checked.

  • I added unit tests for my code
  • I fully described what my PR does in the documentation
    (not in the PR description)
  • I fixed all affected documentation
  • I added code comments, logging, and assertions as appropriate (see Coding Guidelines)
  • I updated all meta data (e.g. README.md of plugins and METADATA.ini)
  • I mentioned every code not directly written by me in THIRD-PARTY-LICENSES

Review

Reviewers will usually check the following:

Labels

If you are already Elektra developer:

  • Add the "work in progress" label if you do not want the PR to be reviewed yet.
  • Add the "ready to merge" label if the basics are fulfilled and you also
    say that everything is ready to be merged.

@mpranj
Copy link
Member Author

mpranj commented Oct 31, 2019

Build failed because of docker:
Record trace of a Docker image used in FROM: remote file operation failed.

I don't know if this is a known problem but seems unrelated to the PR.

@markus2330
Copy link
Contributor

Thank you for creating this PR! Good to see activity from you 💖

@mpranj
Copy link
Member Author

mpranj commented Oct 31, 2019

@markus2330 I ran into #2130 once again. Since the addition of OPMPHM to mmap, I'd need the opmphmDel and opmphmPredictorDel functions to free the old resources before replacing them by mmap'ed ones.

The implementations are "entangled" either way, since mmapstorage needs knowledge of any structures it stores. The alternative to exporting the symbols is to copypaste the *Del functions into mmapstorage. On the positive side: it's really not too much code, and any change to the core structs (Key, KeySet, OPMPHM) inevitably implies adapting mmapstorage accordingly.

What approach to you like more: exporting vs copypasta?

@mpranj
Copy link
Member Author

mpranj commented Nov 1, 2019

Now the jenkins builds have finished successfully, but the status is not reported to github. 🤷‍♂

@sanssecours
Copy link
Member

Now the jenkins builds have finished successfully, but the status is not reported to github. 🤷‍♂

That happened in one of my PRs (a long time ago) too. I just restarted the build when it happened. For obvious reasons – Jenkins build fails all the time and takes forever – this is not an ideal solution. Anyway, I would consider this PR “ready to merge”.

@markus2330
Copy link
Contributor

In some cases it is unfortunately necessary to open a new PR. Jenkins build are now hopefully faster as we now got one more server.

@mpranj
Copy link
Member Author

mpranj commented Nov 1, 2019

jenkins build libelektra please

3 similar comments
@mpranj
Copy link
Member Author

mpranj commented Nov 2, 2019

jenkins build libelektra please

@mpranj
Copy link
Member Author

mpranj commented Nov 2, 2019

jenkins build libelektra please

@mpranj
Copy link
Member Author

mpranj commented Nov 2, 2019

jenkins build libelektra please

@mpranj
Copy link
Member Author

mpranj commented Nov 2, 2019

@markus2330 this should be done, but the tests are constantly failing because zeromq or gopts tests are timing out.

@markus2330
Copy link
Contributor

The zeromq tests can be disabled (as nobody is working on this). Can you please report the gopts issue, maybe @kodebach finds a solution.

@mpranj
Copy link
Member Author

mpranj commented Nov 2, 2019

jenkins build libelektra please

@markus2330
Copy link
Contributor

Please document in doc/todo/TESTING which tests were disabled (ideally also add commit SHA-1 how to revert the disabling of tests).

@mpranj
Copy link
Member Author

mpranj commented Nov 3, 2019

Seems like testmod_zeromqsend was disabled, but not for kdb run_all. I will disable it there now, instead of removing the tests completely. The removal was already documented, but only partially implemented.

@mpranj
Copy link
Member Author

mpranj commented Nov 5, 2019

@markus2330 done.

Copy link
Contributor

@markus2330 markus2330 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some documentation points.

@@ -13,6 +13,7 @@
#include <kdblogger.h>
#include <kdbmacros.h>
#include <kdbopmphm.h>
#include <kdbprivate.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the reason of this include as comment next to it.

@@ -10,6 +10,7 @@
#include <kdbhelper.h>
#include <kdblogger.h>
#include <kdbopmphmpredictor.h>
#include <kdbprivate.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the reason of this include as comment next to it.

@@ -99,7 +124,12 @@ typedef struct _mmapFooter MmapFooter;
/**
* Mmap information header
*
* shall contain only fixed-width types
* Changes to this struct can/will break compatibility with existing files.
* If a breaking changed needs to be introduced, change `ELEKTRA_MAGIC_MMAP_NUMBER`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please give details what a breaking change is:

  • reordering of struct?
  • adding to struct?
  • ...

@markus2330 markus2330 merged commit 82663a3 into ElektraInitiative:master Nov 6, 2019
@markus2330
Copy link
Contributor

Thank you, great work! I am looking forward to see the benchmarks!

@mpranj mpranj mentioned this pull request Nov 6, 2019
@mpranj mpranj deleted the cache-opmphm branch April 21, 2021 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants