Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: include configuration from URL #1403

Open
nichtich opened this issue Feb 15, 2023 · 6 comments · May be fixed by #1412
Open

Feature request: include configuration from URL #1403

nichtich opened this issue Feb 15, 2023 · 6 comments · May be fixed by #1412

Comments

@nichtich
Copy link
Contributor

Description of the enhancement

Support reading content of config.ttl from a SPARQL endpoint. This would require to add two new properties in config.ttl, such as skosmos:configEndpoint and skosmos:configGraph to config.ttl so additional triples are taken from referenced SPARQL endpoint

Configuration is loaded in model/GlobalConfig.php and cached for performance reasons. I think the end of initializeConfig may be a good place to add loading of additional configuration via SPARQL. To avoid doing a SPARQL request each time, the result should be cached as well. I'm not sure on cache invalidation. Maybe only reload on request with a special query parameter e.g. ?config=reload.

Who are the users that would benefit from the enhancement and how?

User who prefer to manage their configuration in Fuseki database.

What new functionalities would the enhancement make possible?

In particular this would allow to manage vocabularies in one place (Fuseki).

Why is the enhancement important?

Adition of vocabularies requires to modify both config.ttl and import into Fuseki. This feature woul allow to manage it in one place and make sure the configuration is valid RDF (no invalid Turtle syntax).

@osma
Copy link
Member

osma commented Feb 17, 2023

This is very similar to PR #1252 (loading configuration from a URL), which unfortunately was never finished. I think if we had support for just loading the configuration from a simple URL, it would be possible to use that facility to point to a graph in a Fuseki database (e.g. via the HTTP Graph Store protocol), right?

Caching and cache invalidation are tricky questions here, as they are in Computer Science generally.

@nichtich
Copy link
Contributor Author

I assume that both loading from and URL loading from a Triple store are too costly to do it without caching: even when the response is cached it must be parsed. By now the configuration is only reloaded when modification time of config.ttl changed. How abbout additional forced reload via URL parameter? Or provide an easy way to flush the whole APC cache?

I'd keep config.ttl as main source of configuration and optionally point to included sources (URL or graph stored in Fuseki) from there. This would also allow to separate categories, vocabularies, and main configuration.

@osma
Copy link
Member

osma commented Feb 17, 2023

The APC cache contains many things unrelated to configuration, in particular cached responses from external Linked Data sources. I don't think flushing the whole cache makes sense here, just the configuration part.

Forcing reloads with a URL parameter doesn't seem like a particularly RESTful way of doing things, and it could also be problematic if there is a reverse proxy such as nginx or Varnish in front of the Apache server running PHP and Skosmos. In addition, it could create conditions for a DoS attack, unless protected by some kind of access control.

If the main configuration is still in config.ttl as you suggest, would it be enough to use the same mechanism as currently, i.e. touch config.ttl (updating the timestamp of the file) would force a reload of all configuration, also the parts loaded from an external URL / endpoint? Of course this cannot be done from the web side directly. But it would be fairly easy for a sysadmin to set up a separate CGI or PHP script, outside Skosmos code, that simply performs touch config.ttl when accessed via URL.

@nichtich
Copy link
Contributor Author

You're right, I like the idea to trigger config reloading via touch config.ttl!

My current workaround is to collect configuration from multiple sources (core, categories and vocabularies) into config.ttl but I'd prefer to use a fixed config.ttl with core configuration and manage vocabularies (vocabulary data and their configuration) in Fuseki.

@nichtich
Copy link
Contributor Author

nichtich commented Feb 21, 2023

it would be possible to use that facility to point to a graph in a Fuseki database (e.g. via the HTTP Graph Store protocol), right?

Yes, I tried out with Fuseki: a plain URL such as http://localhost:3030/skosmos/get?graph=default is enough, so basically this issue covers #1252. In contrast to the implementation there, I'd propose to extend configuration with a property skosmos:includeConfig to load configuration from an URL or from another file. Recursive inclusion is not needed and as discussed, changes in included configuration are not detected automatically but config.ttl must be touched to trigger reloading.

By the way this would also help to use same main configuration for multiple instances (dev, prod, test...) and include differing settings such as skosmos:baseHref and skosmos:logBrowserConsole via skosmos:includeConfig "config-local.ttl" from a local file.

@nichtich nichtich changed the title Feature request: get configuration via SPARQL endpoint Feature request: include configuration from URL Feb 21, 2023
nichtich added a commit to nichtich/Skosmos that referenced this issue Feb 22, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
@nichtich
Copy link
Contributor Author

Implementation here. No test coverage so far but "works on my machine". I'll do a polished PR if the functionality is agreed upon.

nichtich added a commit to nichtich/Skosmos that referenced this issue Feb 23, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
nichtich added a commit to nichtich/Skosmos that referenced this issue Feb 23, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
@nichtich nichtich linked a pull request Feb 23, 2023 that will close this issue
4 tasks
nichtich added a commit to nichtich/Skosmos that referenced this issue Feb 23, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
nichtich added a commit to nichtich/Skosmos that referenced this issue Mar 1, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
nichtich added a commit to nichtich/Skosmos that referenced this issue Mar 7, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
nichtich added a commit to nichtich/Skosmos that referenced this issue Mar 28, 2023
This adds repeatable configuration property `skosmos:includeConfig` to
include configuration from file or URL. The configuration is cached as
usual, based on modification time of `config.ttl` only.

See NatLibFi#1403 for discussion.
@MikkoAleksanteri MikkoAleksanteri added this to Selected for analysis in NatLibFi Skosmos Backlog via automation Apr 6, 2023
@MikkoAleksanteri MikkoAleksanteri added the size-small max 2 hours label Apr 6, 2023
@MikkoAleksanteri MikkoAleksanteri moved this from Selected for analysis to Analysis and size estimate done in NatLibFi Skosmos Backlog Apr 6, 2023
@MikkoAleksanteri MikkoAleksanteri moved this from Analysis and size estimate done to NatLibFi priority Medium in NatLibFi Skosmos Backlog Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
NatLibFi Skosmos Backlog
NatLibFi priority Medium
Development

Successfully merging a pull request may close this issue.

3 participants