remove internal meta-data

[markus2330]

@kodebach reported in #1164:

Other problems with kdb set are:

• internal/** metadata is not removed
• origvalue metadata is not removed (breaks all future kdb get, until it is removed)

fixing it in kdb is quite straight-forward. But isn't it better to fix it in the spec plugin, then not every tool needs to reimplement it?

See also #2700 (currently open because the global position for spec does not work).

[kodebach]

But isn't it better to fix it in the spec plugin

The spec plugin is not the problem in this case. origvalue shouldn't exist in the spec namespace, so the spec plugin won't copy it. internal/** is already ignored by spec (although it shouldn't need to do this IMO). In my case it was the internal/ini/** metadata (created by the ini plugin) that got stored.

A plugin running after spec could also still add internal/** or the origvalue metadata.

Note: I actually have no idea, where the origvalue metadata comes from. The type plugin calls keySetString, which in turn always removes origvalue. The only possible explanation I have right now, is that the type metadata was removed between kdbGet and kdbSet (if so, not intentionally).

fixing it in kdb is quite straight-forward.

Fixing it in kdb is actually impossible. The tool cannot influence, which plugins are called. Therefore it cannot prevent plugins from adding such metadata. AFAIK kdb does everything correctly.

IMO there has to be a way to mark metakeys (and maybe even normal keys) for automatic removal. All the marked keys should then be automatically removed, before the storage plugin is called.

[markus2330]

The spec plugin is not the problem in this case. [...] A plugin running after spec could also still add internal/** or the origvalue metadata.

We once discussed that spec should be the vacuum cleaner to fix everything that is wrong with metadata.

A plugin running after spec could also still add internal/** or the origvalue metadata.

The idea was that spec will be the very last plugin.

Fixing it in kdb is actually impossible.

Yes, if the metadata is added during kdbSet. I thought we talk about metadata that should vanish between kdbGet and before kdbSet.

IMO there has to be a way to mark metakeys (and maybe even normal keys) for automatic removal. All the marked keys should then be automatically removed, before the storage plugin is called.

There is no need for automatic removal because the semantics as given in doc/METADATA.ini already tell us what should be removed and what is needed for storage plugins. The spec plugin would need a list of metadata to be removed. This is not super-pretty but the alternatives need code-generation which is too much for 1.0 now.

[kodebach]

We once discussed that spec should be the vacuum cleaner to fix everything that is wrong with metadata.
The idea was that spec will be the very last plugin.

Yes, I know (#2700 (comment)), but it is not possible right now. I also think spec is not the right plugin for that.

If we don't want to do the cleanup in the core (or the new backend plugin), it should be a separate cleanup plugin. spec is already complicated enough.

I thought we talk about metadata that should vanish between kdbGet and before kdbSet.

I don't know, where the internal/** metadata should vanish. That depends on the metadata. origvalue should definitely not vanish between kdbGet and kdbSet (unless the key was changed) that would defeat its whole point.

There is no need for automatic removal because the semantics as given in doc/METADATA.ini already tell us what should be removed and what is needed for storage plugins. The spec plugin would need a list of metadata to be removed. This is not super-pretty but the alternatives need code-generation which is too much for 1.0 now.

It might be true that doc/METADATA.ini tells us what should be removed and what shouldn't (although not in an explicit machine-readable way), but marking (meta-)keys that should be removed would be a lot easier.

[markus2330]

If we don't want to do the cleanup in the core (or the new backend plugin), it should be a separate cleanup plugin. spec is already complicated enough.

Fair point. We can also make a new cleanup plugin. I thought that having it in spec is okay because the cleanup code in spec is not complicated.

origvalue should definitely not vanish between kdbGet and kdbSet (unless the key was changed) that would defeat its whole point.

Nothing should vanish between kdbGet and kdbSet, see explanation in #2700. The crucial point here is to implement the hook correctly. Afaik it was never correct and because of that it never worked.

but marking (meta-)keys that should be removed would be a lot easier.

For the removal code: certainly. Overall: I do not know. There are many plugins that set metadata to be removed, they all would need to be changed to mark their metadata.

Furthermore, this would open the door for many different markers. E.g. for storage plugins we might want to remove different meta-data as for exporting configuration.

[stale]

I mark this issue stale as it did not have any activity for one year. I'll close it in two weeks if no further activity occurs. If you want it to be alive again, ping the issue by writing a message here or create a new issue with the remainder of this issue.
Thank you for your contributions 💖

[stale]

I closed this issue now because it has been inactive for more than one year. If I closed it by mistake, please do not hesitate to reopen it or create a new issue with the remainder of this issue.
Thank you for your contributions 💖