recursive navigation

[thealjey]

1. makes navigation recursive, which means that it can go down to an arbitrary depth
2. makes the notion of grand parents/children obsolete. If a page is a child of a page, that itself is a child of another page, that automatically makes this page a grandchild.
3. has_children and grand_parent options are removed as unnecessary
4. breadcrumbs are no longer dependent on main navigation setting certain variables
5. nav_exclude excludes the page from any depth in the menu hierarchy
6. adds categories

[thealjey]

I just noticed something strange, btw.
I can approve the pull request myself, lol.
You guys might want to look into permissions configuration. ;)

[pmarsceill]

Thanks for this -- going to do some testing on it this week.

[pdmosses]

Although the current 3-level navigation is adequate for my current projects, recursive navigation seems to be simpler to specify, as well as more general. If this PR gets included in the next release, I'll be happy to drop #188.

3. has_children and grand_parent options are removed as unnecessary

Presumably existing sites that use those options will still work OK?

I've created a small project to test and demonstrate the effect of PR #188: https://pdmosses.github.io/test-nav/ . It has 3 levels of navigation, and uses a new config option grandchildren_branch, which displays grandchildren only when their parents are selected. When this option is not specified, selecting a grandparent displays the links for all its children and grandchildren, which gives backwards compatibility with 0.2.6.

Assuming that recursive navigation automatically gives the same effect as grandchildren_branch: true, could it include a config option (say reveal_grandchildren: true) to produce exactly the same behaviour as 0.2.6? Alternatively, the display of grandchildren links as in 0.2.6 could be regarded as a bug, and not supported in future releases.

[thealjey]

@pdmosses

This pull request maintains full backwards compatibility, it just simplifies things and fixes a few bugs.

1. has_children and grand_parent options do nothing
   sites using those options will be completely unaffected, as just setting parent correctly is sufficient now
2. only direct(!) children of an "active" menu item are displayed

Do you want grandchildren to be also rendered with reveal_grandchildren set?
How deep do you want to go?
Do you want to render the whole tree all the way down to the leaves, as soon as the top level item is selected?
This will likely necessitate some changes to the css.
What is the use-case for this, if you don't mind me asking?

It can certainly be done, but I feel that it is a feature that probably needs further discussion and approval of @pmarsceill.
Maybe even implemented with a different PR.

[plancomps]

@thealjey,

2. only direct(!) children of an "active" menu item are displayed

Right -- but that is slightly different from the current behaviour.

PR #188 includes restriction of the displayed navigation to direct children, as suggested in #173. For backwards compatibility, however, this is activated by a config option (grandchildren_branch). If backwards compatibility isn't essential in the next release, #188 can be simplified to display only direct children without a config option.

I don't personally have a use-case for displaying grandchildren. In fact the restriction of the display to direct children is essential for my main project.

[thealjey]

I think the current behavior must have been a bug then, since it is not documented (and I followed the documentation).
I don't personally have more than two levels of navigation, that's why I have not spotted that.

[thealjey]

@pmarsceill
One problem with this is that you can start getting "Nesting too deep included" errors.
To fix that "_includes/active.html" would need to be replaced with a custom Jekyll plugin.
Unfortunately I'm not familiar enough with ruby to do that myself.

[thealjey]

@pmarsceill
disregard my last comment
I have figured out how to do it
however, that did not solve the issue I'm having
I must have a circular reference somewhere in the actual pages that api-documenter generates
it is not a problem with the theme

[pmarsceill]

Since this is a "breaking" change -- I'm going plan for it to be included in the 0.3.0 release and minimize conflicts for the next release 0.2.7.

[pdmosses]

The changes in this PR address also issue #173. I've tested them in https://pdmosses.github.io/test-nav/ and I'm happy with the resulting behaviour. I look forward to exploiting deeper navigation on other sites.

[pdmosses]

@thealjey, I've experimented with using the files from _includes and _layouts in this PR on a moderate-size site: https://plancomps.github.io/CBS-beta. The site has 145 pages; almost all of them are included in the current 3-level navigation.

Before adding the changed files, the local build time for the site was 25 to 30 seconds. After adding them (without altering the navigation hierarchy or the YAML data) it is more than 500 seconds! Moreover, after making a trivial change in one file (/index.md) rebuilding took almost 560 seconds (despite using the --incremental option, which surprised me).

The navigation in the local built site looks fine. However, I'd prefer the build time to be lower, so I haven't pushed the changes to GitHub.

Looking at the liquid code in _includes, it seems my build involves many recursive calls of active.html, which all filter site.html_pages. The code looks neat, but I suspect the build time is something like quadratic in the size of the site. I'm wondering whether it's possible to get it close to linear somehow?

BTW, I tried removing the nested sorting at the start of nav.html, but that didn't help at all.

[thealjey]

@pmarsceill bringing grand_parent back would completely defeat the purpose, though. Wouldn't it?
I still think that an extra id field is the key here.
It can default to some other value, like a title.
On the other hand, I do not believe that backwards compatibility is a concern at all, since if you were relying on has_children and grand_parent for menu structure, that already doesn't work anymore.
And if you're already willing to change your front matter configuration to have recursive menus, why not add an extra required id field to all of them at the same time?

[pdmosses]

bringing grand_parent back would completely defeat the purpose, though. Wouldn't it?

The grand_parent and has_children fields would still be optional. The grand_parent field is only needed for disambiguation of non-unique parent titles; the has_children field is ignored.

And personally, I don't want to have to update any fields on a large site unless it's essential.

[thealjey]

@pdmosses yes, but if the id itself was optional and defaulted to the value of title,
then you would be able to disambiguate, just the same, by setting ids onto the parents with non-unique titles

[pdmosses]

@thealjey I don't see the disadvantage of continuing to use the grand_parent field for disambiguation, provided that it is optional.

I agree that an optional id field also supports disambiguation, but by ignoring the grand_parent field, that would break existing sites.

[pdmosses]

@thealjey I have added a branch https://github.com/pdmosses/just-the-docs/tree/rec-nav-0.3.0 to my fork of JtD. I created it based on the 0.3.0 master, then replaced the navigation by your code, with very few changes.

The navigation is currently based entirely on parent: has_children fields are ignored, and I haven't yet tried exploiting grand_parent to disambiguate ambiguous parent references (so there are some duplicated nav links in the Testing section). Please take a look, and let me know if any changes are needed.

I have not submitted a PR based on my rec-nav-0.3.0 branch, since it is essentially just an update of the present PR to work with 0.3.0 - I haven't added any new code at all. If you have time, could you copy the changed files to your PR, so that we can proceed to review it?

The current implementation of recursive navigation creates the navigation hierarchy object by grouping the site pages by parent. Perhaps it could then split each group of pages having the same parent field but differing grand_parent fields? That would give the same effect as specifying parent: A>C or parent: B>C, while ensuring backwards compatibility.

I'll now proceed with trying to implement that idea.

[pdmosses]

Perhaps it could then split each group of pages having the same parent field but differing grand_parent fields?

It turned out to be simpler to filter the groups when needed, rather than splitting the groups in hierarchy in advance.

The updated code now gives the correct navigation for the pages in Testing: the displayed children of the C child of A don't include the children of the C child of B, and vice versa.

I'll now try using my rec-nav-0.3.0 on a larger site, to check the build efficiency.

[mikeecb]

By using grand_parent.title to disambiguate pages, where essentially saying that the (grand_parent.title, parent.title) should be unique across all pages, which is certainly not always the case.

Where I'm using just the docs, we're trying to standardize pages and subpages for easy reading (e.g. all docs for microservices have the same format and same child page names).

[pdmosses]

@mikeecb my aim with using grand_parent was to make @thealjey's recursive navigation feature available without requiring any changes to existing sites using 0.3.0.

Where I'm using just the docs, we're trying to standardize pages and subpages for easy reading (e.g. all docs for microservices have the same format and same child page names).

I would also like to support that kind of navigation (which isn't available in 0.3.0).

To disambiguate navigation for pages having identical titles in different sections of a website, one could include a distinguishing prefix in all the titles; but that is clearly not a good solution.

Something equivalent to such a prefix could be included in the front matter of the pages, and not appear in the titles. My initial idea was to specify as much of the ancestor path as needed for disambiguation. An optional id or a path field (defaulting to the title) were suggested as alternatives to that, but I think they could be tricky to maintain when sites change.

However, what all the pages for a particular section of a website have in common is the node at the top of that section (which need not be at the top of the entire site, of course). Those section top nodes naturally have different titles. So for disambiguation, it would be adequate to specify the title of the section top node directly in all the pages (except the top) using a new field:

ancestor: Title

(Changing a section title would involve updating the same string in the ancestor fields of all its pages, but that should be easy enough.)

Would that be an adequate solution for your sites?

[Potherca]

@pdmosses I've made an alternative implementation to recurse pages without the need to add anything to front-matter.
(It uses URLs).

You can see the implementation details here: https://github.com/Potherca/extend-the-docs/tree/HEAD_includes/nav_recurse

Cleanup is still needed and I haven't looked at speed optimizations yet, but I thought it might be of interest to you...

[pdmosses]

@Potherca, to base the navigation entirely on the URL path hierarchy surely has some advantages – especially in connection with this recursive navigation feature. And I guess that in many sites using Just the Docs, the URL path hierarchy does correspond closely to the navigation structure. However, the names in URLs are sometimes quite cryptic abbreviations, which makes them unattractive for general use in navigation, I think.

BTW, you have a navigation expander on your extend-the-docs link, but it doesn't seem to do anything when clicked.

[Potherca]

the names in URLs are sometimes quite cryptic abbreviations, which makes them unattractive for general use in navigation,

I completely agree, that's why I only use the URL/path for folders that have no readme (so no way to get a title) or pages that do not have a title (which are not supported by Just the Docs).¹ I'm still thinking about the best way to deal with those scenarios. ²

If I manage to get my changes stable and performant, it might be worth seeing if they can be ported back into Just the Docs.

The main reason for "going solo" is that I am not familiar enough with the community and I don't want to frustrate any existing efforts.

BTW, you have a navigation expander on your extend-the-docs link, but it doesn't seem to do anything when clicked.

Good catch! Looks like I have a bug to fix. 😹

Footnotes

1. See _includes/nav_recurse/page_list.lqd and _includes/nav_recurse/node_information.lqd

   Click for more details

   {% assign site_page_title = site_page.title %}
   {% if site_page_title == nil %}
     {% assign site_page_title = site_page_url | remove_first: "/" | replace: "/", " " | trim %}
   {% endif %}

2. I'll need something similar for listing posts in the menu

   Click for more details

   I want to avoid the menu looking like this:

   2020                    ⌄
       2020 07             ⌄
           2020 07 11      ⌄
               First post
   

[pdmosses]

@mikeecb @Potherca I've now added a branch with a new implementation of recursive navigation:

https://github.com/pdmosses/just-the-docs/tree/rec-nav

The previous branch (reg-nav-0.3.0) was a merge of @thealjey's implementation and JTD-0.3.0, with the addition of grand_parent fields for backwards compatibility (and the removal of navigation groups). The differences in the new branch are:

1. support for replicating the same-looking navigation structure in different sections;
2. addition of a front-matter section field that is independent of title, and can be an arbitrary string;
3. nav_order can now use a mixture of numbers and strings;;
4. additional tests and documentation;
5. improved efficiency.

Could you take a look at it, and let me know whether it addresses at least some of your concerns regarding my previous proposals?

Regarding efficiency, the build time appears to scale somewhat better than the previous version. Testing locally with a real site of 150 pages (mostly included in the navigation, 4 levels) using Jekyll-3.8.7 with profiling on a MacBook Air (1.6 GHz Dual-Core Intel Core i5) I get the following approximate times for clean builds of subsets of pages:

Pages	Time (CPU seconds)	Previously	Improvement
35	10	11	9%
78	55	74	25%
120	120	162	26%
150	170	264	35%

With Jekyll-4 the build times should be significantly faster. But before proceeding, it would be good to check scaling to larger sites.

BTW, it appears that the (nice!) navigation expander feature introduced by @SgtSilvio requires all pages to include the entire navigation hierarchy, which can be quite big on larger sites; in previous releases, only the part of it visible on the page was needed. I've added an option to hide the expanders and reduce the amount of code for navigation.

[pdmosses]

With Jekyll 4.1.1, the builds are a bit faster for both versions (the total times below are the average for two clean builds):

Pages	Time (CPU seconds)	Previously	Improvement
37	7	6	-16%
80	42	61	31%
122	97	132	27%
152	152	245	38%

The profiling reports the number of times each file is included and the time that takes, indicating where significant optimisation might be possible. For instance, when building 152 pages in the previous branch (rec-nav-0.3.0), active.html is included 95K times, taking 94 seconds; in the new branch (rec-nav), nav/children.html is included 35K times, taking 18 seconds. In both versions, the file that outputs the navigation links is included 3648 times (24 times per page), generating about 12MB.

It appears that (for both versions) the build time complexity is roughly quadratic in the number of pages included in the navigation. The navigation hierarchy of the profiled website isn't exceptionally deep or wide: the number of pages is consistent with an average depth 3 and average width 5.

Should we be concerned about such build times? Are users with somewhat larger sites likely to get timeout errors when building on GitHub Pages?

[pdmosses]

BTW, setting nav_expanders: false when using the rec-nav branch significantly reduces the amount of generated navigation links and the time taken. Building 152 pages:

• The file that outputs the navigation links is included only 481 times (was 3648 times), generating about 1MB (was 12MB).

• Jekyll-4.1.1 takes about 40 CPU-seconds (done in 18 seconds), instead of 105 CPU-seconds (done in 30 seconds).

I've added an overview of the implementation in https://github.com/pdmosses/just-the-docs/blob/rec-nav/docs/test/about.md

(Updated) Site Render Stats reported by Jekyll:

Filename	Count	Bytes	Time
_layouts/default.html	152	4094.77K	10.502
_includes/nav/page.html	1487	0.00K	8.850
_includes/nav/links.html	481	1115.48K	7.455
_includes/nav/init.html	152	0.00K	5.819
_includes/nav/children.html	6208	0.00K	4.561
_includes/nav/sorted.html	1583	0.00K	1.179
assets/js/zzzz-search-data.json	1	603.88K	1.007
_includes/vendor/anchor_headings.html	152	2510.16K	0.497
_layouts/vendor/compress.html	152	3876.31K	0.422
_includes/head.html	152	317.60K	0.416
_layouts/table_wrappers.html	152	4100.34K	0.041
_includes/nav/toc.html	152	19.19K	0.023
_includes/nav/crumbs.html	136	67.63K	0.013
_includes/title.html	152	1.93K	0.008

The reported CPU times vary somewhat between (clean) builds.

The improvements over the previous version of the above table are due to adding heuristics that can locate pages quicker in the navigation hierarchy.

[ghost]

Any status on this 🙌

[pdmosses]

@kdvtrifork apologies for the delay. Please take a look at PR #462, which can also be used for further discussion of points raised in various comments above.

[kevinslin]

I ended up creating a 11ty version of a recursive just-the-docs template. you can see what it looks like here

this template my bigger project, dendron, an open source note note taking tool built on top of vscode and markdown with static publishing capabilities.

posting here in case it helps anyone with recursive navigation. feedback also appreciated :)

[pdmosses]

@kevinslin thanks for the post. You wrote:

posting here in case it helps anyone with recursive navigation.

This PR has been superseded by #462, for various reasons. Are there significant differences between the features you have implemented on your site and those implemented by #462?

feedback also appreciated :)

Feedback on #462 is welcome too!

[kevinslin]

Yeah, I would say the main difference is that Dendron went the ID route. Each note has a unique guid at the top (frontmatter) and then written with the guid when compiled. See https://dendron.so/notes/5fcb8564-7209-4a80-9bb8-025bc8eb489b.html#permanent-ids

…

On Sat, Jan 16, 2021 at 2:23 PM Peter Mosses ***@***.***> wrote: @kevinslin <https://github.com/kevinslin> thanks for the post. You wrote: posting here in case it helps anyone with recursive navigation. This PR has been superseded by #462 <#462>, for various reasons <https://github.com/pdmosses/just-the-docs/blob/rec-nav-2/_tests/navigation/recursion/about.md>. Are there significant differences between the features you have implemented on your site and those implemented by #462 <#462>? feedback also appreciated :) Feedback on #462 <#462> is welcome too! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#192 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADD52PJBOM47N4K6YJQ2L3S2IGWHANCNFSM4IMOV4GA> .

[pdmosses]

Yeah, I would say the main difference is that Dendron went the ID route.
Each note has a unique guid at the top (frontmatter) and then written with
the guid when compiled. See
https://dendron.so/notes/5fcb8564-7209-4a80-9bb8-025bc8eb489b.html#permanent-ids

Right, that's indeed a major difference. For parent title disambiguation, #462 allows (globally-unique) section IDs in front matter, but they do not affect the URLs of pages. Jekyll's permalink feature can be used to make URLs independent of file names, when needed.

[ragalgut]

Hello,

I am interested in not having a limitation of levels in the navigation tree. Where can I find the modified content with a more current version of just-the-docs?

Thank