Manually (or automatically ?) update site config from Wallabag

[j0k3r]

We are going to use https://github.com/j0k3r/graby instead of FullTextRss in the v2. Graby will integrate site config from FiveFilters but allow to use our own.

What could be a great idea is to give ability to the user to update site config directly from Wallabag. Something like a page in the config area where you can hit an "update site config now".

• Manually update site config from a git repository
• Remember last update (last commit ?) to be able to display "new site config available"

[tcitworld]

Yup, I had the idea since a long time ago, thanks to graby this should be easy. ;)

[ghost]

to go further, it coud be great to add personnal site config directly from the config page and maybe another way to submit the config file to the community.

[nicosomb]

I had a look to http://siteconfig.fivefilters.org/
and here are the used libs:

• https://github.com/andrewchilds/jQuery.DomOutline
• https://code.google.com/p/css2xpath/

We can develop our own tool (because siteconfig.fivefilters.org is not opensource).

[ghost]

Excellent. i hope you can do that soon.

[tcitworld]

Maybe we can also directly ask @fivefilters if they plan to release the tool as open-source.

[fivefilters]

Hey guys, I'll try to release the code - was always intended to be out to help projects like this one. Been a bit slow doing that. To be honest, most of it is the JS stuff making use of the libraries Nicolas linked to, which I think is already viewable. But I'll release the PHP bits too.

[fivefilters]

Here it is: https://github.com/fivefilters/ftr-site-config-build

[j0k3r]

@fivefilters thanks !

[tcitworld]

Regarding the auto-update part, it seems there's no real library that does what we want here.
Either they are simple calls to the git command which has to be installed, either (in the case of gitlib) they don't work with remotes, so you can't do a git pull or merge, therefore the whole interest of git is gone.

[nicosomb]

cURL or wget can do what we want, maybe?

[j0k3r]

I'll prefer cURL to avoid calling a system command.. but could be more difficult

[ghost]

in owncloud admin, there is a Cron menu with Ajax, webcron and cron.
I don't know if it could be usefull.

[j0k3r]

I like the idea from @Strubbl about make update-site-config

[tcitworld]

How would this work ? composer update only the site-config repo ?

[j0k3r]

Yeah, maybe:

SYMFONY_ENV=PROD composer update j0k3r/graby-site-config --no-dev --prefer-dist

But I don't how could this deal on small instance (like with not that much memory). I don't know if composer will needs to build the whole deps tree or just update the repo.

[shtrom]

Thinking about distribution packages, it may not be possible or even desirable to overwrite the composer vendor.

An alternative solution could be to let the admin specify an alternate location for graby-like files in the configuration file, which would take priority over the vendored version.

The admin could then manage/update that location however they want. The obvious solution would be another git checkout, and perhaps a cron.

[techexo]

Isn't it possible to modify graby in order to have something like unix configuration files, with a subfolder containing locally-defined site configuration files, which will not be overwritten when pulling from graby-site-config?

As I see it, it would be a vendor/j0k3r/graby-site-config/local directory, and graby would be modified to merge configuration files with local > vendor > global files.

[j0k3r]

@techexo for custom siteconfig, I think it's better to keep them in the upper folder, like in data/siteconfig

[shtrom]

I guess we could do something like this (just from reading the code and jumping around the repos):

• (wallabag/wallabag) Wallabag gets an optional override directory for site-configs from the parameters.yml; data makes a lot of sense from a distro-packaging perspective.
• (wallabag/wallabag) If present, Wallabag sets this override on its instance of Graby\SiteConfig\ConfigBuilder, either in code, or perhaps simply through the services.yml (I'm not too familiar with this, but it looks like it would work)
• (j0k3r/graby) When the override is given, the ConfigBuilder then sets the path override on GrabySiteConfig\SiteConfig\Files
• (j0k3r/graby-site-config)...which results in getFiles to look in that override first-or-instead (see Bonus points, below)

I think this could work nicely, as the override would not need to be a clone of graby-site-config, as the PHP logic would still be in the vendors, and could be a straight ftr-site-config instead.

Bonus points for:

• supporting an array of overrides
• ... which would require supporting looking into all the directories with File (first, above), rather than just one (instead, above)
• it looks like it would be time to get an instance of Files, rather than using static methods

[j0k3r]

@shtrom It doesn't need to be that complicated :)
f43.me already use an extra folder when you test a siteconfig from the UI (and it's just configuration): https://github.com/j0k3r/f43.me/blob/master/app/config/services.yml#L70

[shtrom]

😬 I did say the services.yml could help q: But it all does sound good.

[shtrom]

Ok, so I cobbeld the following together; spoiler alert: it doesn't seem to work just yet.

I added this to app/config/services.yml (once again, random guesses, really):

    wallabag_core.graby:
        class: Graby\Graby
        arguments:
            -
                error_message: '%wallabag_core.fetching_error_message%'
                error_message_title: '%wallabag_core.fetching_error_message_title%'
                extractor:
                    config_builder:
                        site_config:
                                - "%kernel.root_dir%/data/site_config"
            - "@wallabag_core.guzzle.http_client"
            - "@wallabag_core.graby.config_builder"
        calls:
                - [ setLogger, [ "@logger" ] ]
        tags:
                - { name: monolog.logger, channel: graby }

This is essentially

wallabag/src/Wallabag/CoreBundle/Resources/config/services.yml

Lines 39 to 50 in e586d65

	wallabag_core.graby:
	class: Graby\Graby
	arguments:
	-
	error_message: '%wallabag_core.fetching_error_message%'
	error_message_title: '%wallabag_core.fetching_error_message_title%'
	- "@wallabag_core.guzzle.http_client"
	- "@wallabag_core.graby.config_builder"
	calls:
	- [ setLogger, [ "@logger" ] ]
	tags:
	- { name: monolog.logger, channel: graby }

, with the added configuration for the ConfigBuilder pointing to data/site_config, where I checked out my shtrom/ftr-site-config.

I then tried to fetch a page from a site that I recently added to the repo. First removing the file altogether, the fetch failed, then re-adding the file... the fetch failed again 😞

[ngosang]

Is there any update about this?
Wallabag release cycle has slowdown making this issue more noticeable.