Uploading wrong attachments for drawio

[wangnangg]

Hi,

First, thanks for this great tool! I finally don't have to deal with confluence editor anymore.

I have installed both drawio and confluencebuilder extensions:

extensions = [
    'sphinx_rtd_theme',
    'sphinxcontrib.drawio',
    'sphinxcontrib.plantuml',
    'breathe',
    'sphinxcontrib.confluencebuilder'
]

# drawio settings
drawio_no_sandbox = True
drawio_builder_export_format = {'confluence': 'svg'}

I tried to include drawio figures using this directive:

.. drawio-image:: diagram/xxx.drawio

The problem is the figure won't show up in confluence. I checked the source, it is referring to a xxx.svg file, which is expected. However, I noticed confluencebuilder is trying to upload the raw drawio file xxx.drawio instead of the generated svg file.

[jdknight]

Thanks for the report -- there looks to be something up.

Using the drawio sample set, a run appears to render drawio content as expected. When trying to match the configuration exactly, the same results. Although, when inspecting the Confluence instance being published to, both the .drawio and .svg extensions appear to be uploaded as attachment. When inspecting the output/implementation, it looks like the following occurs:

1. confluencebuilder performs its image processing check to see what images are to be uploaded. It finds (for example) an image candidate entry and registers it into the asset manager:

   <image candidates="{'*': 'diagram/box.drawio'}" classes="['drawio']" uri="diagram/box.drawio"/>
   

2. At some point, the drawio extension will update the image with new candidate values. When confluencebuilder performs its second pass of finding any new images generated from other events, it then detects the SVG one.

   <image candidates="{'*': '.\\output\\py\\sample-drawio\\.doctrees\\drawio\\c10f10d8c2d1aeef4d59a7ca6354e021557e3d8b\\box.svg'}" classes="['drawio']" uri=".\output\py\sample-drawio\.doctrees\drawio\c10f10d8c2d1aeef4d59a7ca6354e021557e3d8b\box.svg"/>
   

3. confluencebuilder will then publish both files during the attachment uploading stage.

Uploading both files is less than ideal. We should maybe update confluencebuilder to detect when image content is "replaced" and remove it from asset tracking.

That being said, it does not explain why the figures are not working for the reported use case.

Some questions about your case:

• For the comment "it is referring to a xxx.svg file", I just want to confirm that you see DrawIO generate an svg file in the project's output directory (I assume this is the case).

• When monitoring the output of a publish event, do you see attempts to publish both .drawio and .svg?

  publishing assets... [ 33%] box.drawio
  publishing assets... [ 67%] box.svg
  publishing assets... [100%] objects.inv
  

  Or just the .drawio file?

  publishing assets... [ 50%] box.drawio
  publishing assets... [100%] objects.inv
  

• In the case where you may see the SVG being uploaded as well, do you see it uploaded on the page's attachment?

• Do you see any different results when using png instead of svg?

• Can you indicate which version of sphinxcontrib-drawio and the version of DrawIO installed?

[wangnangg]

The svg is generated locally: .doctrees/drawio/3801198468179d90799067f4827c526171438f1f/kmd_shim.svg

I don't see svg being uploaded, only drawio.

[100%] Generating Sphinx documentation
Running Sphinx v7.1.2
/home/nanwa/.local/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py:24: RemovedInSphinx80Warning: The alias 'sphinx.util.SkipProgressMessage' is deprecated, use 'sphinx.util.display.SkipProgressMessage' instead. Check CHANGES for Sphinx API modifications.
  from sphinx.util import SkipProgressMessage, progress_message
/home/nanwa/.local/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py:24: RemovedInSphinx80Warning: The alias 'sphinx.util.progress_message' is deprecated, use 'sphinx.http_date.epoch_to_rfc1123' instead. Check CHANGES for Sphinx API modifications.
  from sphinx.util import SkipProgressMessage, progress_message
/home/nanwa/.local/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py:26: RemovedInSphinx80Warning: The alias 'sphinx.util.progress_message' is deprecated, use 'sphinx.http_date.epoch_to_rfc1123' instead. Check CHANGES for Sphinx API modifications.
  from sphinx.util import progress_message
making output directory... done
running in non-headless mode, not starting Xvfb
building [mo]: targets for 0 po files that are out of date
writing output... 
building [confluence]: targets for 2 source files that are out of date
updating environment: [new config] 2 added, 0 changed, 0 removed
reading sources... [100%] kmd_shim
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
copying assets... done
(drawio) 'diagram/kmd_shim.drawio' -> 'drawio/3801198468179d90799067f4827c526171438f1f/kmd_shim.svg'
writing output... [100%] kmd_shim
[dryrun] updating existing page Docs (2795954634)    
[dryrun] updating existing page Design (2795954636)                    

building intersphinx... done
[dryrun] adding new attachment kmd_shim.drawio                                  
[dryrun] adding new attachment objects.inv                                      

Publish point: ...
build succeeded.
[100%] Built target docs-design-confluence

The resulting confluence page only has the drawio as attachment.

I tried jpg as well, no difference.

sphinxcontrib-drawio: sphinxcontrib-drawio==0.0.17

drawio --version
Checking for beta autoupdate feature for deb/rpm distributions
Found package-type: deb
24.2.5

Thanks!

[wangnangg]

One thing I do that might be different from others is this:
I have a source directory and a build directory. I don't want to generate any temporary files in my source folder. So my conf.py is copied to build folder at build time. Then I run sphinx-build from the builder folder and provide the location of the conf.py through the -c option. Here's my cmake snippet:

add_custom_target(
    docs-design-confluence ALL
    COMMAND
        ${SPHINX_EXECUTABLE} -j auto -b confluence -c ${CMAKE_CURRENT_BINARY_DIR}
        ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}/design -vvv
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
    COMMENT "Generating Sphinx documentation"
)

I suspect the builder failed to find the svg asset because of this. I don't know if I am using this feature correctly.

[wangnangg]

I may be completely off but I noticed that we are always looking for an asset in the srcdir? If an assert is generated (like drawio), then it will be in the build dir, right? When looking for an assert, can we check both src dir and build dir?

    if normalized_path.is_absolute():
        abs_path = normalized_path

        # some generated nodes will prefix the path of an asset with `/`,
        # with the intent of that path being the root from the
        # configured source directory instead of an absolute path on the
        # system -- check the environment's source directory first before
        # checking the full system's path
        if normalized_path.name[0] == os.sep:
            new_path = Path(env.srcdir, normalized_path.name[1:])

            if new_path.is_file():
                abs_path = new_path
    else:
        abs_path = Path(env.srcdir, normalized_path)

        # extensions may dump a generated asset in the output directory; if
        # the absolute mapping to the source directory does not find the
        # asset, attempt to bind the path based on the output directory
        if not abs_path.is_file():
            abs_path = out_dir / normalized_path

    # if no asset can be found, ensure a `None` path is returned
    if not abs_path.is_file():
        abs_path = None

[jdknight]

You might be on the right path. I assume confluencebuilder cannot find the path of the SVG when using find_env_abspath which could prevent it from being registered internally for uploading. It looks like a user can only know that this is happening if verbose logging is enabled.

Do you by chance see any [confluence] prefixed log that print failed to find path:? This would help confirm that the path is not being found internally.

If an assert is generated (like drawio), then it will be in the build dir, right?

Unfortunately, it is not always consistent. It has been observed various extensions to unique things when referencing certain files -- a mix of absolute paths, relative to a source and relative to a build directory (on a side note, no idea why drawio is generating content in .doctrees). The confluencebuilder implementation tries to handle all these cases -- and there may be some little thing that is not being handled gracefully for this use case (assuming the issue is in this function).

I do no think you are doing anything wrong from a quick inspection of the CMake snippet you have provided.

[wangnangg]

Hmm.. after switching to png/jpg, the image will be uploaded to confluence. It seems to be specific to SVG.

[jdknight]

If you do not mind trying this out, but do other SVG images (non-DrawIO generated) work as expected?