[Question] Allow more input formats for Franklin.jl?

[Klafyvel]

Hi there!
I am the developper of Norg.jl, a parser for the Norg markup language. The Norg syntax originates from neorg, an org-mode inspired neovim plugin.
Norg.jl aims to eventually cover the whole specification and provide an easy way to deal with .norg files in pure Julia. The main goal is to provide ASTs for developpers to play with, but there are also some code generation capabilities. I would love to be able to use it to write my personnal blog, which uses Franklin.jl.
The hacky way to do that would be to export the ASTs to markdown and then feed them to Franklin.jl. But with the upcoming 1.9 release of Julia, we will soon have the ability to have weak dependencies within packages. This way, one can think of adding support for norg in Franklin.jl at a reasonable cost.
That being said, I understand that it would require some work on the Franklin.jl side, that I am willing contribute to.

What are you thoughts on such a feature?

[tlienart]

Hello @Klafyvel,

I think you might want to experiment with Xranklin.jl; basically Franklin's next version (if I find the time to finish up the last few things...). (Also Franklin is in a frozen state until I merge Xranklin into it so working on Franklin for anything else than bug fixes is not a good idea)

If I understand properly, what you'd like is to be able to have for instance a folder my-norg-posts with a bunch of .norg files in it, that get processed upon changes by Norg and generate dedicated pages in __site. If that's correct then this is doable by imitating the Literate pathway (you could do this in Franklin as well but it should be cleaner in Xranklin which explicitly assumes that a user may want to do this e.g. to support Weave, Pluto, CommonMark or whatever, see https://github.com/tlienart/Xranklin.jl/blob/1394b6eb72ae58801e195633e25275cc4aa366d6/src/convert/markdown/lxfuns/literate.jl)

That function lx_norg which would be simlar to lx_literate would be define in your utils.jl file which would do the using Norg.

This would still require having one, or several .md files which act as "callers" for each of the .norg file, i.e. dummy files with \norg{path/to/foo.norg}. But actually you could do this with a single .md file which could call another utils function which would just call the processing as if there was one .md file per .norg file except without creating them (basically just calling html(r"\norg{path/to/norgfile.norg}) in a loop over norg files.

If this kind of sounds like what you might be after, I'd suggest experimenting with a dummy Xranklin repo (happy to help you there if you need a hand), try to write a lx_norg (also happy to help) and see if that works with you?

[Klafyvel]

Thank you for that answer!

I wasn't aware of Xranklin.jl, I'll give it a look. With the process you describe, would it be possible to change the metadata/page variable definition from Norg (there is a specific metadata bloc in the spec)? For example, from lx_norg?

[tlienart]

I wasn't aware of Xranklin.jl, I'll give it a look. With the process you describe, would it be possible to change the metadata/page variable definition from Norg (there is a specific metadata bloc in the spec)? For example, from lx_norg?

well in lx_norg you can do whatever you want; basically you read a string (the norg file), process it yourself (via Norg), and there are hooks that you can use to access Xranklin's local or global page variables via setgvar!(:symbol, value) and setlvar!(:symbol, value) (global are available on all pages, local only on the current page).

[Klafyvel]

Excellent, I'll try to play with that soon. Thank you for your help!

[Klafyvel]

Still hacky, but here is an experimental Norg.jl-in-Xranklin.jl website! I guess at some point -when the parser is more advanced- I'll write a proper glue-package that smooths things a little bit, most likely with a custom Franklin.jl code generator.

[tlienart]

Looks pretty good. The lx_norg seems fairly ok to me (there are things that are more of the order of how you want the information between the two packages to flow but the structure seems sound to me, glad it's working!)

If there are things you would have liked to have from Xranklin, let me know, this can be useful to others who may want to do similar things in the future

[wklimowicz]

Thanks for all your work on this package! To tag on here, it would be great if Xranklin would have an "official" way to use other markdown formats (either the md or html output). I've managed to get Quarto to work with Franklin, with quarto generating html files, which populate a post listing with some utils.jl infrastructure. (Quarto also has a static site generator, but much less control compared to Franklin).

The only hacky thing I need is dummy ".md" files to trigger the RSS feed, because I can't pass the RSS variables through the Quarto qmd file itself. If I could redefine the RSS feed function in utils to look for rss_description in the YAML of qmd documents, everything else would work perfectly.

[tlienart]

@wklimowicz , there is already a path in Xranklin to hook another markdown to html converter (e.g. CommonMark.jl or anything else even external programs), maybe this is best discussed on the Xranklin repo though.
In terms of RSS, I'm not sure what you need but that's one of the remaining things that have to be done in Xranklin (i.e. there's no RSS support yet).

[Klafyvel]

I know next to nothing about how Franklin handles files, but what do you think of using Julia type system to handle various file formats? You could have something like a process(::T, filename, context) where {T<:MIME} method that external libraries could overload to extend (X|F)ranklin. For example, @wklimowicz would have to redefine Franklin.process(::MIME"text/markdown", filename, context), I would define Franklin.process(::MIME"text/norg", filename, context) and so on.

[tlienart]

Hmm that's an interesting idea. I'd rather carry out this discussion in the Xranklin repo for now as this won't happen in the current state of Franklin. There are a few things to bear in mind but I think it could happen.

In the global loop over files (this file: https://github.com/tlienart/Xranklin.jl/blob/main/src/build/build_loop.jl) there's essentially three cases:

• a "core" file (e.g. layout) has been changed, the whole site needs to be rebuilt
• a "dependency" file (e.g. literate file) has been changed, any page that calls it needs to be rebuilt
• a "simple" file (e.g. page.md) has been changed, that page is processed

For that last one, it calls process_file(...) which has a case keyword which is based on the file extension (special treatment of .md and .html otherwise basically just copy the file somewhere appropriate

Then it calls the deeper logic to do the actual file processing:

https://github.com/tlienart/Xranklin.jl/blob/45e9d395e6d9c1c8090c6afde9f4d5d2ada59e95/src/process/process.jl#L61-L85

A straightforward modification would be to have this else be actually something like

elseif Utils.special_filetype(fpath)
    Utils.special_process(fpath)
else
    # same as before
end

With Utils.special_filetype(x) = false by default but can be overwritten by the user in utils.jl, and same for Utils.special_process. The special_process could be whatever the user wants and use FileTypes.jl or whatever to consider the mime type of the file (I'm reluctant to add that as a dependency though because not all files have a mime type so that this would be more constraining than just passing the filepath and letting the user do whatever they want with it)

A few things not specified in the outline above:

• https://github.com/tlienart/Xranklin.jl/blob/45e9d395e6d9c1c8090c6afde9f4d5d2ada59e95/src/process/process.jl#L52 would need to also be adjustable in Utils to know where to place the output file, though that could also be figured out by special_process
• a user who wants full dependency triggers such as if you update a .norg file A.norg and some other file B.norg depends on it so it should also be updated; that stuff is done for .md file and could be imitated in the special_process but is complex, probably best to assume that this wouldn't be done (or if it has to be done, then it's simpler to call the .xxx file from a .md file so that this logic is delegated to the parent .md file

But let's say that a user just wants to have a special path for a few of their files which should be independently processed and the output injected in layout files without code execution or cross-page dependencies, then this should work.