Rendering from data (not individual files)

[u8sand]

I had a use-case where I created a static webpage to display information available in a database. In particular it required me to create my own render function/loop:

def render(template, dump, **kwargs):
  ''' Render custom page with jinja template engine '''
  print('Rendering %s...' % (dump))
  site.get_template(template).stream(**dict(config.funcs, **kwargs)).dump(os.path.join(config.build, dump))

for post in posts:
  render('_post.html', 'post/%s.html' % (post['id']), **post.items())

Would there be a way to register these pages with staticjinja such that they would be regenerated if the underlying _post.html page was changed. Perhaps even a "changed" hook could be exposed so that the underlying data could report changes and get it regenerated.

I'm specifically interested if there is already a way to do this, else I may fork and contribute a solution if you're interested.

[PatrickMassot]

I'm not sure I understand what you mean. Could you provide a bit more details?

Anyway, I'm pretty sure it will be easier if you use my stalled PR #65 whose goals were both to allow the reloader to rebuild only the needed pages and to be much easier to subclass for specific needs.

[u8sand]

Glancing at your pull request, it looks like I might be able to achieve what I'm looking for by subclassing the Builder, SourceManager, and Reloader, will try that sometime soon.

Clarifying my request somewhat, fundamentally staticjinja operates on a directory of files, it processes each file one by one according to some rule allowing modification of how each specific file is rendered/its context. I'd like to pass an arbitrary list of elements with the following attributes: source_template, context, output_file, any number of elements may call the same source_template, e.g.

<!-- templates/_count.html -->
<p>{{ num }}</p>

for i in range(10):
  site.add_render(source_template='_count.html', output_file='%d.html' % (i), context={'num': i})

These add_render rules wouldn't be built until site.render() is executed (they'd get rendered with everything else and rebuilt automatically).

Right now it seems like I would need to have files 0-9 and set a render rule which seems silly. My underlying data shouldn't need to be represented by files. If I changed _count.html, all objects dependent on that source_template would be rebuilt (e.g. [0-9].html). Similarly, I should be able to trigger an update manually: site.rebuild('5.html').

[Fredericco72]

@u8sand Did you ever find a solution for this? I have a similar requirement

[u8sand]

@Fredericco72 Basically I did not come up with any good solution but the project I was using this for is long done. In retrospect, staticjinja is simple and good for something specific -- if your use-case is not that, you're likely better off just using jinja2 directly.

If your use case is along the lines of using a database to generate content and rendering cascading changes #65 was on the right track. If I was going to use something like this again I would likely just use makefile, potentially generating one with python or doing it all in Makefile.

Just a sketch of what it might look like:

# jinja2-cli from https://github.com/mattrobenolt/jinja2-cli
TEMPLATE_COMPILER = jinja2

TEMPLATES=$(shell find templates -type f -exec sh -c 'realpath --relative-to=templates {}' \;)
DIST=$(foreach template, $(TEMPLATES), build/$(template))

# rule to build a template
build/%: templates/%
  mkdir -p $(shell dirname $@) && $(TEMPLATE_COMPILER) $^ context.json > $@

# trigger all template builds
build: $(DIST)

# auto-remake support https://stackoverflow.com/questions/7539563/is-there-a-smarter-alternative-to-watch-make
WATCHMAKE=build
watch:
    while true; do \
        make $(WATCHMAKE); \
        inotifywait -qre close_write .; \
    done


# additional constraints -- i.e. load data from db
records.json:
  mysql 'select * from records' > $@

[NickCrews]

Finally looking through these ancient issues trying to close them out...

At this point I believe that you could accomplish this, by using a custom rendering function. See an example at e6b7ef6. You could use a regex to match against _count.html, and in the custom renderer for that template you could write to disk as many output files based on that template as you wanted, passing in different contexts in the template.stream() call.

I agree however that I think this use case is a bit too complicated for staticjinja to really excel at, just use jinja2 directly.

[NickCrews]

This was brought back up as a feature request in #124.

I do see the usefulness of this feature personally. And, it seems as though there is a bit of demand, since two people in the last year out of the ~100 current dependents (according to GitHub, probably a slight underestimate?) have asked for it. So, I'm considering how to support this. However, I also think it's very valuable how simple staticjinja is right now, so if it's going to happen, this feature had better be pretty streamlined.

At this point, the design requirements I'm thinking of are:

1. Defaults are sane, so without any extra config, things work as they are, mapping 1:1 source to dest.
2. Existing "simple" uses of the python API and the CLI should not be broken by the change. It would be OK if the change is backwards incompatible with more complex uses (e.g. ones that use custom render functions or context generators).
3. No changes to the CLI, this is too complicated for that.
4. The API allows for multiple levels of complexity. For example in the current state of custom render functions, you have to make the actual call to template.stream() in order to write the rendered template to disk. It would be awesome if instead you could just supply the output file name, and staticjinja was the one actually calling template.stream(). However, if you were doing something more complex, then you could use the existing custom render functions to control the entire rendering process.
5. This change should be compatible/consistent with Handle static files outside the templates folder #58
6. [Will probably think of more later]

I'll reopen this Issue as I experiment with APIs, and comment here with any PRs I propose, and would love some feedback.

[killjoy1221]

I managed to hack together a solution. Here's a build.py.

#!/usr/bin/env python3
import os
import re

from staticjinja.staticjinja import Site, _ensure_dir


class Site(Site):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.extras = {}

    @property
    def templates(self):
        yield from super().templates
        for name, template_name in self.extras.items():
            template = self.get_template(template_name)
            # the template name is what is used for the file name
            template.name = name
            yield template

    def add_extra(self, name, template_name, context):
        if name in self.extras:
            raise ValueError(name + " is already registered")

        self.extras[name] = template_name

        # partial copy of Site.render_template
        def render_extra_page(site, template, **context):
            filepath = os.path.join(site.outpath, name)
            _ensure_dir(filepath)
            template.stream(**context).dump(filepath, self.encoding)

        file_pattern = "^{}$".format(re.escape(name))
        self.rules.insert(0, (file_pattern, render_extra_page))
        self.contexts.insert(0, (file_pattern, context))


def main():
    site = Site.make_site()
    # add extra pages individually
    site.add_extra("hello.html", "_layout.j2", {"message": "Hello!"})
    site.add_extra("child/index.html", "_layout.j2", {"message": "This is a child index page."})
    site.render()



if __name__ == "__main__":
    main()

[s-ol]

+1 for this feature request.

I'll give @killjoy1221's implementation a try for now, but I also wanted to suggest an API option:

Rendering multiple result pages from the same template is tied directly to the data source (you need multiple datasets to do it in the first place). Therefore the natural place for configuring it in the existing API is the contexts list, which is where the user specifies to staticjinja what the data is (or how to get it). The contexts setup already is a good interface for configuring how different templates should be handled and where their data comes from. It's only lacking:

• the option to specify an output path different from the one derived from the template
• the option to return multiple datasets

Both of these can be solved with a small change to the contexts API. It could work like this:

• as usual, every input template is matched against contexts
• if the matched context is a dictionary, rendering proceeds as usual
• if the matched context is a list:
  • each entry should be a (outputname, contextdict) tuple
  • each list entry is rendered separately from the same template
• if the matched context is a function, it is called as usual, and the same logic is applied to its result

This change should be fully backwards compatibile and keeps the declarative API intact.