Fancy generic templating stuff for nested data #620

bergus · 2018-03-17T20:53:00Z

This takes various ideas from #529, #572 and #565 into action.

I've got a data file (json, or better yaml as that has comments) in my project whose contents I want to use in rendering various files - both html and json. Call it "shared (meta)data" if you want.

My first idea was a classic Compiler - parse the JSON file using Aeson, store in a snapshot, spit back out (without whitespace). Access the parsed data from the snapshot elsewhere (in the html templates).
Problem: snapshotted data must be a Binary instance. Hakyll can do this already with data, but the BinaryYaml (a newtype wrapper over Value) is not exported.

My second idea was using Hakyll's builtin metadata parsing. Have a xyz.metadata file, use getMetadata "xyz" in the Compiler monad. Hakyll would do the parsing and caching for me, right?
Problem: xyz does not exist as a resource, so no metadata can be read from it (instead it always comes back as an empty object by default).

I guess this could have been solved with an empty xyz file, but instead I went ahead and ~~fixed~~ updated the Provider to consider it as existing when there's a xyz.metadata file.

Next step: write a custom context that allows me to access and iterate deep (nested) Values - generically, without parsing the JSON in a dedicated structure and supplying the relevant list fields.
Converting Aeson Values into the respective strings was cumbersome though, and Hakyll already does have toString which I didn't want to replicate. Also this functionality seemed so useful that it might be interesting to add natively, possibly also for Metadata like #572.
I've got a proof of concept working in the attached commits (starting at "WIP", the rest is from #462), but this is much less a pull request than a request for comments as of now. What do you all think? Specifically

do we need to support keys that contains dots? If yes, how - and does it need to be backwards-comptible?
what fields should be available inside a $for(…)$ loop that iterates an array, or an object? Should the outer context be sustained, i.e. if $var$ is available outside of the loop should it be accessible from inside the loop as well?

Closes jaspervdj#461

Force templates to be consumed completely, propagate error appropriately

...makes better error messages :-) Notice the breaking change in 'applyElem', where $if(...)$ conditions in templates now will throw errors if their field 'fail'ed (instead of just being 'empty')!

...when a more important error prevails. Also not throwing from 'if' conditions any more, only logging those errors to the debug screen

* boolFields used outside of 'if'-conditions now get a "stack trace" using a new 'NoField' they don't have to rely on 'error' any more * templates applied to their own file get proper description (did use incompatible paths/identifiers before) * renamed 'compilerFail' to more descriptive name

Closes jaspervdj#507 (actually was fixed by 9ec43a6 already, this just adds the test)

See jaspervdj#462 (comment) and below for detailed explanation Also abstracted out `testCompilerError` in the test suite, and added a `compilerTry` that is much easier to use (and specifically, to branch on) than `compilerCatch`

bergus · 2018-03-17T21:03:20Z

@jaspervdj What do you think about 73cc078? This seemed like the right idea, but my understanding of Provider is too shallow yet. Specifically, treating identifiers as resources just because a .metadata file exists for them might have some weird side effects - no file at that path exists, but resourceExists says otherwise. Also it does get matched by wildcard patterns that do not contain a file extension, and trying to compile such a non-existing file will fail.
Is my commit an appropriate hack, and the errors it could cause are warranted?
Or does this need a more holistic design approach - I also noticed there's still a TODO in the code about modification dates of metadata (in general, not only external files). Should metadata and contents (bodies) be versioned separately, as they are already cached separately?

jaspervdj · 2018-03-18T14:24:28Z

I like what this PR tries to accomplish -- it's something I've also wanted for a long time.

do we need to support keys that contains dots? If yes, how - and does it need to be backwards-comptible?

No, I don't think we need to support this. In the worst case we can add escaping `"foo.bar" later if it turns out to be truly necessary.

what fields should be available inside a $for(…)$ loop that iterates an array, or an object? Should the outer context be sustained, i.e. if $var$ is available outside of the loop should it be accessible from inside the loop as well?

I think it makes sense to do lexical scoping -- so the outer context should be sustained, but the inner context can overwrite it if there's a conflict.

Going to have a look at the code now.

jaspervdj · 2018-03-18T14:25:01Z

lib/Hakyll/Core/Compiler.hs

+-- | Turn on caching for a compilation value to avoid recomputing it
+-- on subsequent Hakyll runs.
+-- The storage key consists of the underlying identifier of the compiled
+-- ressource and the given name.


Typo: resource

jaspervdj · 2018-03-18T15:03:09Z

lib/Hakyll/Core/Provider/Internal.hs

-  where
-    mdRsc = fromFilePath $ flip addExtension "metadata" $ toFilePath identifier
+getResourceInfo :: FilePath -> Identifier -> IO (Identifier, ResourceInfo)
+getResourceInfo directory identifier = do


Do we need to support asking for resource info about .metadata files?

I guess I should rename this function to getFileInfo. It is called on every (non-ignored) file in the directory, and then it decides itself whether that file is metadata of another resource or a resource itself.

jaspervdj · 2018-03-18T15:04:24Z

lib/Hakyll/Web/Feed.hs

-rssTemplate :: String
-rssTemplate = T.unpack $
+rssTemplate :: Item String
+rssTemplate = Item "templates/rss.xml" $ T.unpack $


Maybe we should call these something like Item "_hakyll/templates/rss.xml" so they can't conflict with actual user-provided items?

Ah, this change is actually part of #462 on which this PR is based on :-)
I think the Item here is solely used for generating template error messages, it never faces the user anywhere. Or am I missing something that Compiler internals would do to an Item?

jaspervdj · 2018-03-18T15:25:05Z

lib/Hakyll/Web/Template/Context.hs

+-- | Creates a variadic function field.
+--
+-- The function will be called with the dynamically evaluated string arguments
+-- from the template as well as the page that is currently rendered.


Thanks for improving the docs!

jaspervdj · 2018-03-18T15:30:52Z

lib/Hakyll/Web/Template/Internal.hs

    , template
+    , unTemplate
+    , getOrigin


Do we need to export unTemplate? In either case I'd call these templateElements and templateOrigin.

I think we need to export them from Template.Internal, yes. About renaming them: unTemplate had been used and exported before as part of the non-smart Template constructor.

jaspervdj · 2018-03-18T15:32:35Z

lib/Hakyll/Web/Template/Context.hs

+    (k:ks) | k == key -> lookupNestedValue ks (Item i val)
+    _                 -> failBranch $ "Tried field " ++ key -- and functionField get
+  where
+    get = let (h:rest) = key in "get" ++ toUpper h : rest


I don't really understand what this does 🤔

Are we encoding getFoo as special keys?

This allows one to create a dataField "foo" (Object metadata) or something, then use $foo.bar.baz$ in templates or $getFoo("bar", "baz")$ - this allows lookup with dynamic names (generated by another field) and also supports names with dots (or any other special syntax).

jaspervdj · 2018-03-18T15:56:37Z

I think the cleanest thing would probably to consider a .metadata file and the resource as "one thing".

bergus · 2018-03-18T17:41:32Z

I think the cleanest thing would probably to consider a .metadata file and the resource as "one thing".

Yes, that's what I've been trying to do with M.fromListWith combine <$> mapM (getResourceInfo directory) list and the combine function.
The problem is that I don't know which code in Hakyll does depend on the assumption that every "thing" (Identifier) has a resource.

bergus · 2018-03-18T17:44:49Z

I think it makes sense to do lexical scoping

Yes, I figured that out when trying it - I need to have some control over the contexts in the loops.

That means my current approach with Context Value, Context (Int, Value) and Context (Text, Value) won't work, I need to pass the value to the creation of the context itself not extract it from the Item.

bergus · 2018-03-20T12:20:16Z

lib/Hakyll/Web/Template/Context.hs

@@ -92,7 +94,7 @@ data ContextField
    = EmptyField
    | StringField String
    | forall a. ListField (Context a) [Item a]
-
+    | forall a. LexicalListField (forall b. Context b -> a -> Context b) [a]


Lexical scoping was not possible with the current ListField constructor, needed something more advanced :-) Also requires RankNTypes, not sure whether that's a bad thing.
Maybe it's also possible to further generalise this, but I don't see a way to do it without breaking compatibility for everyone who directly uses ListField. Maybe these should be moved to a Context.Internal module anyway.

bergus · 2018-03-20T12:25:56Z

lib/Hakyll/Web/Template/Context.hs

 module Hakyll.Web.Template.Context
    ( ContextField (..)
    , Context (..)
+    , context
+    , functionContext


I wonder whether we should export the general functions under new names. I don't really like context and functionContext, as they construct contexts for specific keys. Should be keyContext at best.
Do you think it would do any harm to just widen the type of field and functionField?

I think it's fine to widen the type if it doesn't break existing code. I'm a bit afraid it could break existing code though, since people usually use {-# LANGUAGE OverloadedStrings #-}.

jaspervdj · 2018-03-23T13:23:13Z

To get back to your earlier question, there is no assumption that every Identifier has an underlying resource. In fact, it is quite common to produce additional pages based on just existing pages, or just Haskell code, so there is no resource matching the identifier name on the disk.

but allow resources without a "body", consisting only of their metadata file (Even if that sounds a bit crazy)

nicola-gigante · 2019-06-10T15:04:44Z

Is there any news on this functionality?

bergus added 12 commits August 4, 2016 16:08

Better error messages for context fields

2b87d13

Closes jaspervdj#461

Better error messages for template applications

8ac7949

Better error messages for template parsing

9ec43a6

Force templates to be consumed completely, propagate error appropriately

docs, docs, docs! Full coverage for Template and Context

7031661

Less debug messages from templates

f98a220

...makes better error messages :-) Notice the breaking change in 'applyElem', where $if(...)$ conditions in templates now will throw errors if their field 'fail'ed (instead of just being 'empty')!

Better error messages from Alternative fails

0ec007e

...when a more important error prevails. Also not throwing from 'if' conditions any more, only logging those errors to the debug screen

Merge branch 'master' into error-messages

dd68b2e

test case for mismatched template syntax

f546381

Closes jaspervdj#507 (actually was fixed by 9ec43a6 already, this just adds the test)

Better names, some minor simplifications

1e04f93

Added useful exports and improved documenttion

2f6ef3a

jaspervdj reviewed Mar 18, 2018

View reviewed changes

bergus force-pushed the fancy-context branch from b9958a9 to 744c65f Compare March 20, 2018 12:09

bergus commented Mar 20, 2018

View reviewed changes

bergus added 8 commits March 27, 2018 16:39

Merge remote-tracking branch 'origin/master'

c098216

better error messages for file system operations

e523fb7

more precise error messages for template application

f6af7d4

fix documentation syntax

e263ccf

WIP

394cb08

Do not consider .metadata files to be separate resources

78207ae

but allow resources without a "body", consisting only of their metadata file (Even if that sounds a bit crazy)

WIP

87602fa

lexical scoping for list fields

f49d972

more dry code

6663735

bergus force-pushed the fancy-context branch from 744c65f to 6663735 Compare April 9, 2018 19:19

This was referenced Aug 25, 2019

Better error messages #722

Merged

Made inner loops have access to outer loop values #726

Open

Generic record as field value #727

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fancy generic templating stuff for nested data #620

Fancy generic templating stuff for nested data #620

bergus commented Mar 17, 2018

bergus commented Mar 17, 2018 •

edited

jaspervdj commented Mar 18, 2018

jaspervdj Mar 18, 2018

jaspervdj Mar 18, 2018

bergus Mar 18, 2018

jaspervdj Mar 18, 2018

bergus Mar 18, 2018

jaspervdj Mar 18, 2018

jaspervdj Mar 18, 2018

bergus Mar 18, 2018

jaspervdj Mar 18, 2018

bergus Mar 18, 2018

jaspervdj commented Mar 18, 2018

bergus commented Mar 18, 2018

bergus commented Mar 18, 2018

bergus Mar 20, 2018

bergus Mar 20, 2018

jaspervdj Mar 23, 2018

jaspervdj commented Mar 23, 2018

nicola-gigante commented Jun 10, 2019

Fancy generic templating stuff for nested data #620

Are you sure you want to change the base?

Fancy generic templating stuff for nested data #620

Conversation

bergus commented Mar 17, 2018

bergus commented Mar 17, 2018 • edited

jaspervdj commented Mar 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaspervdj commented Mar 18, 2018

bergus commented Mar 18, 2018

bergus commented Mar 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaspervdj commented Mar 23, 2018

nicola-gigante commented Jun 10, 2019

bergus commented Mar 17, 2018 •

edited