Feature: needimport supports caching and configured needs.json sources

[arwedus]

We've used needextract to show a few needs out of many, many needs imported from some other tool (via needs.json and needimport) in the right place in the documentation.

What bugs me is that the links never go to the place where the needextract is rendered, but always to the location where the original need is defined. This is a problem for the actual use case needextract was made for: If I want to generate a customer documentation (in HTML!) and want to show only the relevant parts of needs to the customer, I would do something like this:

Doc1: Needimport Page
=====================

.. just importing my architecture model or whatever...

.. needimport:: //build/model/architecture_needs.json
   :hide:

And in the actual docu:

Doc2: Customer docu
===================

Component Foo documentation
---------------------------

.. needextract::
   :filter: id == "COMP_FOO"
   :layout: complete

.. needextend:: COMP_FOO
   :+implements: REQ_SW_FOO_123495823

The Foo component does X and Y. It is implemented in ...

What I get with this approach is an error message / extension error at the end of the build process:

Extension error (sphinx_needs.needs):
Handler <function process_creator.<locals>.process_caller at 0x7f1eef8d5160> for event 'doctree-resolved' threw an exception (exception: Need COMP_FOO has no content node.)

Alternatives considered

An alternative would be to "needimport" the same needs.json many times. But I fear that this would not be so performant for my use case (7000 needs). Maybe, if needimport would cache all need and not re-read the same json file twice, this might not be a big build time problem, but I also don't want to have people provide the same needs.json path in 1000 individual documents.

Also, one needs to be super careful with the filters to avoid importing the same needs twice, while at the same time not forgetting to import those needs which are linked but have no dedicated documentation page.

Expectations

In order to not break the current behavior, maybe add an option to set whether a needextract "overrules" the location of the original link.

Also, add a user friendly error message instead of the extension error above.

Needextract "overruling" locations of original definition may bring it's own bunch of non-determinism problems. Unfortunately, attempts to implement it have also not been successful so far (see #689 ).

So let's rephrase the expectations more generally:

I want a directive to include selected imported needs exactly where I need them, with links going to this page. I want to define the path to the data source (needs.json) only once. I want the data source to be read only once.

Solution idea

I have a suggestion to improve the yaaa-needimport:

• Define needimport sources in conf.py centrally, just like needs_external_needs, and reference them by key in the needimport directive.
• Support a cached needimport for centrally configured needimport sources. This way, one can use the needimport directive many times for the same source with different filters, but the needs are only imported from file once.

[danwos]

We have done already some analysis and tests in the past but Sphinx is a beast and has always overwritten or not accepted our changed links. So further work is needed here.

I guess this ticket is a duplicate of #689.
So I would like to close this one

[arwedus]

@danwos: Thanks for your quick answer. I already assumed that it would not be easy to do. Also, needextract can appear multiple times for the same need... in this case, it's probably hard to do proper error handling.

I've therefore adapted the issue to consider a new solution approach, improving the featur set of needimport.