Prefetch of WADO ，outcomeMessage：Fetched 0 objects by WADO Exporter WADO-JSON

[bird-bro]

Hi, there,
I try to use Prefetch of WADO ，
https://github.com/dcm4che/dcm4chee-arc-light/wiki/Pre-fetch-of-WADO-objects
https://github.com/dcm4che/dcm4chee-arc-light/wiki/Prefetch-WADO-Metadata
But it can't run normally,
Server Log:Failed to find Web Application for WADO request invocation : Unknown WebApplication，
Archive UI：Monitoring--Export，Status：COMPLETED ，outcomeMessage： Fetched 0 objects by WADO Exporter WADO-JSON
I'm not sure if there is a configuration error.
Archive Version 5.23.3

Thanks in advance.

Attach error log and Archive configuration。

Using Archive UI：
Web Applications：

Attribute Set：

Storage Descriptor：

Exporter Descriptor：

Export Rule：

[vrindanayak]

Which archive version are you running? The Wado Exporter was redesigned as part of this issue. The wiki provides configurations for both pre/post 5.23.2 archive version. There was a minor error w.r.t archive version in the wiki which I've corrected. Please refer the wiki again and see the difference for the following

In addition to the above, your web application configuration is incorrect. You may reuse the existing DCM4CHEE web application (which already contains the correct Web Service Path i.e. /dcm4chee-arc/aets/DCM4CHEE/rs and Web Service Classes i.e. WADO_RS); you would just need to additionally configure a http connection to be referenced in the web application with HTTP as Protocol on dcm4chee-arc device and reference this in the DCM4CHEE web application's Network Connection Reference.

Also, you don't need an Attribute Set configuration for pre-fetching metadata using Wado Exporter.

[bird-bro]

Hi, vrindanayak
I really appreciate your help！
After modifying the configuration according to your prompts, it can run successfully on the archive version 5.23.3。

[bird-bro]

Hi, vrindanayak
Sorry, it was foun during the test that the metadata obtained was not generated by prefetching.
the json file has been produce and correctly stored in the specified location.

debug：if (location.getObjectType() == Location.ObjectType.METADATA) == false.

Attach Archive configuration

Storage Descriptor:

Exporter Descriptor:

[vrindanayak]

outcomeMessage：Fetched 1 objects by WADO Exporter WADO-JSON

This indicates that metadata was successfully prefetched into the configured storage. Check the files at /opt/f1/storage/wado/{now,date,yyyy/MM/dd}/{StudyIUID/{SeriesIUID-hash}.json

[bird-bro]

Yes，I did find that json file generated successfully in the specified location . My question is : when I retrive the metadata with wado request from client browser, I am not sure waht I get is that prefetched json file or the one generated on the fly.
I have tried to debug the source code to see what happened, but unfortunately did not find the code lines.Simply speaking , how can I verify what I get from client browser is read from that prefetched json ?
Any help will be appreciated.

[vrindanayak]

Not fully sure what exactly you are trying to achieve.

• If you are planning to build a cache server between client and archive, you don't need to configure a storage for pre-fetching objects. Instead, your cache server (with a separate device configuration and web application) can be configured in the exporter to receive WADO requests which then your cache server forwards back to archive to get the objects pre-fetched and store it in your cache which can be used later by the client for faster retrieval. See Prefetch of WADO objects - Use Cases.

• Prefetching to a storage was provided as a use case for applications where copies of images / metadata are to be used for some processing by some third party applications.

• With archive version 5.24.0, prefetching QIDO responses shall be possible as well. - Prefetch QIDO Responses wiki

• As previously mentioned, independent of prefetching features, you may configure storing instance metadata when studies are stored to archive. On subsequent wado requests to archive either using browser or prefetch wado exporters or any other client, the retrieve service shall read the metadata from the instance metadata json files (instead of reading from DICOM files) and send back in the response. In this case, the following

  if (location.getObjectType() == Location.ObjectType.METADATA) {

  would return true for location record of an instance in database which has object_type as METADATA.
  Note : Configuring instance metadata storage creates two location records for a single instance - one with object_type as DICOM and second with object_type as METADATA

[bird-bro]

Hi, vrindanayak
Here is our case:
We have a dicom viewer. Before loading all of the images in one study, we have to build the study-series-instance tree-like data structure in the viewer. This data structure’s building relies on the study-level metadata retrieved by WADO from dcm4chee. To improve the user experience, it is necessary to get that metadata as fast as possible.
In wiki, this seems what we are looking for:

1. Pre-fetch the image and/or metadata on some cache server for faster access. In such use cases it is required that the user >configure a separate device and configure HTTP Protocol specific connections on it which are then used for referencing in Web >Application(s) to be configured on this new created device used as a cache device.

But we don’t know how to configure it. Any detailed instructions will be helpful.

[vrindanayak]

You may build a simple webhook callback application which invokes archive's QIDO / WADO service(s) to get the attributes information. To do this :

• ensure your application can be reached at http[s]://<your-webhook-host>:<webhook-port> and is capable of invoking one (or more) of the following archive's services
  • For QIDO Prefetch - Recommended - Response time of QIDO services is way faster than WADO metadata services when you consider getting metadata for very large studies :
    • Search Instances of Series of Study - Recommended
    • Search Instances of Study - Not recommended (as export gets triggered for the whole study - each time any new series / instance is received by archive)
    • Search Series of Study - Not recommended (as export gets triggered for the whole study - each time any new series / instance is received by archive)
  • For WADO Prefetch - Optional
    • Retrieve Series Metadata - Lesser preference compared to Search Instances of Series of Study
    • Retrieve Study Metadata - Not recommended (as export gets triggered for the whole study - each time any new series / instance is received by archive)
• Next, configure a separate device (eg. cache) in archive configuration with a network connection with <your-webhook-host> and <webhook-port>. Configure HTTP as Protocol for this network connection.
• In this cache device, configure a Web Application (Devices -> dcm4chee-arc -> Extensions -> Device Extension -> Child Objects -> Web Application).
  • Configure Web Service Path as /dcm4chee-arc/aets/<ARCHIVE_AET>/rs (default <ARCHIVE_AET> = DCM4CHEE)
  • Select the HTTP network connection in Web Application Network Connections
  • Configure QIDO_RS (for QIDO Prefetch) / WADO_RS (for WADO Prefetch) as Web Service Class.
• Follow Prefetch QIDO responses for export rule and exporter configurations (ignore storage configurations), specifying wado:your-web-app-name in the URI field in exporter.

  dn: dcmExporterID=QIDO-JSON-InstancesOfSeries,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
  dcmExporterID: QIDO-JSON-InstancesOfSeries
  dcmQueueName: Export2
  objectClass: dcmExporter
  dicomAETitle: DCM4CHEE
  dcmURI: wado:DCM4CHEE
  dcmProperty: QidoService=/studies/[0]/series/[1]/instances
  dcmProperty: Accept=application/json
  dicomDescription: Export to QIDO-JSON-InstancesOfSeries
  
  dn: cn=Forward to QIDO-JSON-InstancesOfSeries,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
  dcmExporterID: QIDO-JSON-InstancesOfSeries
  dcmDuration: PT1M
  dcmEntity: Series
  dcmProperty: SendingApplicationEntityTitle=QIDO_JSON
  objectClass: dcmExportRule
  cn: Forward to QIDO-JSON-InstancesOfSeries
  

• Configure Qido ETag as true which ensures that archive returns Last-Modified and ETag in responses

Use Case Example : Prefetch Search Instances of Series of Study

• On receive of studies, the export rule triggers prefetching of QIDO / WADO by invoking http[s]://<your-webhook-host>:<webhook-port>/dcm4chee-arc/aets/<ARCHIVE_AET>/rs/studies/[0]/series/[1]/instances
• When your webhook / cache application receives this request, it shall invoke the same request back to archive replacing <your-webhook-host>:<webhook-port> with <archive-host>:<archive-port>. You may send one or more of the following request headers :
  • If-Modified-Since
  • If-Unmodified-Since
  • If-Match
  • If-None-Match
• Your webhook / cache application receives Search Instances of Series of Study response and caches it for faster access by clients.
• Ensure to respond back to initial request sent from archive to webhook / cache application.

[luxunxiansheng]

Hi, vrindanayak
I read the thread carefully. I wonder if there is simpler and easy way to achive the fast access. Here is a quick thoughts:

1. At study or series level, create a metadata.json file in default.
2. Update this json file when a new instance is stored in the archive.
3. For study level metadata's WADO request. return this metadata.json file directly.

I am not sure if this is feasible.

Cheers
L.B.

[luxunxiansheng]

1. I will use QIDO for its good performance
2. I will query the archive rather than the cache
3. How to get the result same as a WADO request "/studies/{study}/series/{series}/metadata"

I had tried to QIDO with /studies/{study}/series/{series}/instances , but no detailed meta info of instance contained in the json ( but WADO does have)

Sorry for bothering again and again.

[vrindanayak]

QIDO follows the Search Transaction whereas WADO follows the Retrieve Transaction. To get more information in QIDO, you would have to extend the default Attribute Filters but doing this (i.e. storing all attributes of instance on receive of studies) is not very helpful as it increases the DB blobs size.

• What is the average / maximum size of studies (no. of instances) in your env?
• For most clients / viewers, QIDO response should be sufficient for initial loading of data. If they really need some attributes for initial processing which are not available in corresponding QIDO responses, you may extend Attribute Filters with individual attributes accordingly. Else post initial load, the clients / viewers can retrieve metadata as required or you may think of cache servers between clients and archive (with prefetch explained above) for faster retrieval.

[luxunxiansheng]

Thanks a lot. We will try the arribute filter.
Our avaerage : 200 and max.: 600 or so . but most of the attributes are not useful.

BTW:
I wolud like come back to the pervious topic about prefetch WADO . From the discussion, it is abvious that a metadata.json file on study or serise level has been generated in the file system. What not let a wado request
"/studies/{study}/series/{series}/metadata" read that json file ? It is only I/O once and should be faster than to loop each single instance json file .

[vrindanayak]

Again, as mentioned previously - using retrieve metadata WADO service to store json file on a storage is an optional use case where the metadata files are passively lying for any third party applications which may need it for post processing. This use case was not intended for WADO services invoked by client / viewer to archive, wherein archive needs to read it from this single metadata file.
For faster retrievals by client systems / viewers, it is better to simply build / implement a web cache server (ATS) between archive and viewer to reduce the response times considerably by prefetching the metadata (or any other data like images / thumbnails etc) in advance

[luxunxiansheng]

Thank you very much ! It is clear now.