S3 as storage: Working example #3146

proteeti13 · 2021-04-20T10:02:42Z

proteeti13
Apr 20, 2021

Hello! I am trying to integrate S3 bucket as storage in my dcm4chee. I am now using the version 5.23.2 of dcm4chee.

I did the following steps:

Installed dcm4chee from here. Got it running at http://localhost:8080/dcm4chee-arc/ui2.
Created an s3 bucket, made it public
In the dcm4chee ui, did these from here

Went to Menu --> Configuration --> dcm4chee-arc --> Extensions --> Edit Extensions --> Edit Extensions of Archive Device --> Deleted [X] fs1 from the Storage Descriptor list --> Save
Went to Menu --> Configuration --> dcm4chee-arc --> Extensions --> Edit Extensions --> Edit Extensions of Archive Device --> Add new Storage Descriptor
Added the following attributes:
Storage ID: proteeti1
Storage URI : jclouds:s3:
Digest Algorithm: MD5
In storage Property added the following from here
identity=
credential=
container=
containerExists=true
jclouds.trust-all-certs=true
jclouds.relax-hostname=true
jclouds.s3.virtual-host-buckets=true
jclouds.strip-expect-header=true

Associate the new Storage with the Archive Application Entity:
Configuration --> dcm4chee-arc --> Child Objects -> Network AEs -> DCM4CHEE -> Network AE Extension --> Archive Network AE -> Attributes -> Object Storage ID
Send CT images to the Archive with docker run --rm --network=dcm4chee_default dcm4che/dcm4che-tools storescu -cDCM4CHEE@arc:11112 /opt/dcm4che/etc/testdata/dicom

Result:

E"ERROR: Received C-STORE-RSP with Status 0110H for /opt/dcm4che/etc/testdata/dicom/MR15.dcm"
(0000,0100) US [32769] CommandField
(0000,0120) US [15] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

09:42:39,163 INFO  - STORESCU->DCM4CHEE(1) >> 16:C-STORE-RSP[pcid=5, status=110H, errorComment=java.io.IOException: Upload failed
  tsuid=1.2.840.10008.1.2 - Implicit VR Little Endian]
09:42:39,164 DEBUG - STORESCU->DCM4CHEE(1) >> 16:C-STORE-RSP Command:
(0000,0100) US [32769] CommandField
(0000,0120) US [16] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

E"ERROR: Received C-STORE-RSP with Status 0110H for /opt/dcm4che/etc/testdata/dicom/MR03.dcm"
(0000,0100) US [32769] CommandField
(0000,0120) US [16] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [272] Status
(0000,0902) LO [java.io.IOException: Upload failed] ErrorComment

09:42:39,166 INFO  - STORESCU->DCM4CHEE(1) << A-RELEASE-RQ
09:42:39,166 DEBUG - STORESCU->DCM4CHEE(1): enter state: Sta7 - Awaiting A-RELEASE-RP PDU
09:42:39,167 INFO  - STORESCU->DCM4CHEE(1) >> A-RELEASE-RP
09:42:39,167 INFO  - STORESCU->DCM4CHEE(1): close Socket[addr=arc/172.19.0.4,port=11112,localport=52241]
09:42:39,169 DEBUG - STORESCU->DCM4CHEE(1): enter state: Sta1 - Idle
Sent 0 objects (=0MB) in 6.235s (=0MB/s)

Answered by vrindanayak

Apr 22, 2021

Here's how I've configured for the first time and tested out exporting objects to Amazon S3

Configuration

Storage Descriptor

dn: dcmStorageID=amazon-s3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
objectClass: dcmStorage
dcmStorageID: amazon-s3
dcmURI: jclouds:aws-s3:https://<bucket-name>.s3.eu-central-1.amazonaws.com
dcmDigestAlgorithm: MD5
dcmProperty: container=<bucket-name>
dcmProperty: jclouds.s3.virtual-host-buckets=true
dcmProperty: credential=<secret-key>
dcmProperty: identity=<access-key>
dcmProperty: jclouds.strip-expect-header=true
dcmProperty: containerExists=true
dcmProperty: jclouds.relax-hostname=true
dcmProperty: jclouds.trust-all-certs=…

View full answer

vrindanayak · 2021-04-20T13:30:50Z

vrindanayak
Apr 20, 2021
Maintainer

What do you see in the archive server.log? Can you post your storage descriptor configuration here?

1 reply

proteeti13 Apr 21, 2021
Author

Storage Descriptor config;
Storage ID: proteeti1
Storage URI : jclouds:s3:< s3 bucket-endpoint-url >
Digest Algorithm: MD5
container=< s3-bucket-bucket-name >
identity= < access-key >
credential= < secret-key >
containerExists=true
jclouds.trust-all-certs=true
jclouds.relax-hostname=true
jclouds.s3.virtual-host-buckets=true
jclouds.strip-expect-header=true

How can I see the archive server.log? Can you provide me some help/
Should I put the s3 bucket endppint url or the vpc enpoint in the "Storage URI"

vrindanayak · 2021-04-21T08:57:23Z

vrindanayak
Apr 21, 2021
Maintainer

Archive server log should be in /var/local/dcm4chee-arc/wildfly/log. Does your start with http[s]?

4 replies

proteeti13 Apr 21, 2021
Author

My s3 bucket url starts with http.
Here is the server log
server log

proteeti13 Apr 21, 2021
Author

Also I am configuring this on https://localhost:8443/dcm4chee-arc/ui2/#/

vrindanayak Apr 21, 2021
Maintainer

Caused by: java.io.IOException: Upload failed
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear//org.dcm4chee.arc.storage.cloud.CloudStorage.afterOutputStreamClosed(CloudStorage.java:179)
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear//org.dcm4chee.arc.storage.AbstractStorage$1.close(AbstractStorage.java:175)
	at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:188)
	at org.dcm4che.core@5.23.2//org.dcm4che3.io.DicomOutputStream.close(DicomOutputStream.java:244)
	at org.dcm4che.imageio@5.23.2//org.dcm4che3.imageio.codec.Transcoder.close(Transcoder.java:422)
	at deployment.dcm4chee-arc-ear-5.23.2-psql.ear.dcm4chee-arc-store-5.23.2.jar//org.dcm4chee.arc.store.impl.StoreServiceImpl.writeToStorage(StoreServiceImpl.java:238)
	... 28 more
Caused by: org.jclouds.blobstore.KeyNotFoundException: /2021/04/20/366D49C5/60AB0084/625FEBA5 not found in container proteeti-test-for-dcm4chee.proteeti-test-for-dcm4chee.s3-website-ap-southeast-1.amazonaws.com: <html>
<head><title>404 Not Found</title></head>
<body>
<h1>404 Not Found</h1>
<ul>
<li>Code: NoSuchBucket</li>
<li>Message: The specified bucket does not exist</li>
<li>BucketName: proteeti-test-for-dcm4chee.proteeti-test-for-dcm4chee</li>
<li>RequestId: 7V8P45NBZ71NQM01</li>
<li>HostId: 8KO+LnkxMxLz3/u4wBuv4s9K7mNaDACmRMuFeGVtpIHfkaLcA82NQQFrLFXbBpNwl4QUZjllEM4=</li>
</ul>
<hr/>
</body>
</html>

proteeti13 Apr 22, 2021
Author

Are there some specific policies that I need to attach to the bucket? I have made the bucket public already. Also, should I use the endpoint url or the vpc endpoint?

vrindanayak · 2021-04-22T15:54:26Z

vrindanayak
Apr 22, 2021
Maintainer

Here's how I've configured for the first time and tested out exporting objects to Amazon S3

Configuration

Storage Descriptor

dn: dcmStorageID=amazon-s3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
objectClass: dcmStorage
dcmStorageID: amazon-s3
dcmURI: jclouds:aws-s3:https://<bucket-name>.s3.eu-central-1.amazonaws.com
dcmDigestAlgorithm: MD5
dcmProperty: container=<bucket-name>
dcmProperty: jclouds.s3.virtual-host-buckets=true
dcmProperty: credential=<secret-key>
dcmProperty: identity=<access-key>
dcmProperty: jclouds.strip-expect-header=true
dcmProperty: containerExists=true
dcmProperty: jclouds.relax-hostname=true
dcmProperty: jclouds.trust-all-certs=true

Exporter configuration

dn: dcmExporterID=StoreToS3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOMConfiguration,dc=dcm4che,dc=org
objectClass: dcmExporter
dcmExporterID: StoreToS3
dcmQueueName: Export6
dcmURI: storage:amazon-s3
dicomAETitle: DCM4CHEE

Settings in AWS :

While creating the bucket, I did the following

unchecked Block Public Access settings for this bucket : By default, this setting was checked (ON)
enabled Bucket Versioning : By default, this setting was Disable
other parameters unchanged

Next, I created the Access Key and used it in the storage descriptor configuration above.

Verification

Export study using Queued Exporter to S3 Storage

Verify in Monitoring - Export

Verify in Amazon AWS

Tested also by enabling the Block public access (bucket settings) setting to block all public access to this bucket and objects within it, had no issues on exporting objects to S3

Exporting existing study from archive to Amazon S3

Using Amazon S3 as Object Storage for studies received by archive

Configured amazon-s3 as Object Storage ID for DCM4CHEE archive AE

Stored study to archive. Verify in Archive UI and AWS S3

I'll add these configs / verification settings in a wiki document

5 replies

proteeti13 Apr 26, 2021
Author

I followed the following steps you mentioned. I exported and a success message was showed. When I got to the monitoring section to verify, it shows the status is "Completed".

Then I checked the S3 bucket, and couldn't find any study. So I expanded the study details to match the messages with your screenshots. I noticed that, in the "outcome message", your one's show "completed:1", while mine shows "completed:0". I am attaching a screenshot here.

I've followed all the config details as you mentioned. But got confused on the first lines of the Storage configuration:

dn: dcmStorageID=amazon-s3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
objectClass: dcmStorage

And in the Exporter Configuration, you have mentioned this:

dn: dcmExporterID=StoreToS3,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOMConfiguration,dc=dcm4che,dc=org
objectClass: dcmExporter

What are you denoting by these? I couldn't find these properties while configuring. But was able to found and configure the rest of the mentioned. By any chance are you exporting the config from terminal?

Also, I couldn't understand the section "Using Amazon S3 as Object Storage for studies received by archive". What is the difference with this from using the "Export Study with the Queued exporter" option?

vrindanayak Apr 26, 2021
Maintainer

I had tried two test cases to test storing objects to Amazon S3.

Using Amazon S3 as ONLINE storage for studies received by archive. By default, it is always fs1 referencing the filesystem where the archive is installed/deployed. For testing, I changed this from fs1 to amazon-s3. So this is denoted by Storage configuration above.
I kept ONLINE storage of studies received by archive as default i.e. fs1 referencing the filesystem where the archive is installed/deployed. There are normally use cases where an user wants to store archived studies to a NEARLINE storage. To do this with minimal configuration, I configured an Exporter and used archive UI to export studies to Amazon S3. This is denoted by Exporter configuration above.

Please check if your bucket settings if it can be accessed by the application - as in the server logs you had attached previously, it mentioned No such bucket found

proteeti13 Apr 26, 2021
Author

I have checked the bucket settings and it is I've unchecked Block Public Access settings for this bucket, as you have mentioned. I've also checked my server log. It is showing this:

Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 1 spans
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:85)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:113)
	... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Cannot flush closed transport
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:148)
	at org.apache.thrift@0.13.0//org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
	at org.apache.thrift@0.13.0//org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:83)
	... 4 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
	at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
	at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
	at io.jaegertracing.jaeger@1.5.0//io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:146)
	... 9 more

Also, previously I was uploading new studies via dcmsnd command and it worked fine. Now it is throwing this error:

./dcmsnd DCM4CHEE@127.0.0.1:11112 ../../dcm

Scanning files to send
.
Scanned 1 files in 0.052s (=52ms/file)
log4j:WARN No appenders could be found for logger (org.dcm4che2.net.NetworkConnection).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to DCM4CHEE@127.0.0.1:11112 in 0.042s
ERROR: Received RSP with Status 0110H for ../../dcm/0012.dcm, cuid=1.2.840.10008.5.1.4.1.1.12.1, tsuid=1.2.840.10008.1.2.4.50
(0000,0100) US #2 [32769] Command Field
(0000,0120) US #2 [1] Message ID Being Responded To
(0000,0800) US #2 [257] Command Data Set Type
(0000,0900) US #2 [272] Status
(0000,0902) LO #64 [java.io.IOException: No space left on configured storage systems] Error Comment

F
Sent 0 objects (=0.0KB) in 0.057s (=0.0KB/s)
Released connection to DCM4CHEE@127.0.0.1:1111

Also, does it mean doing ONLY the Storage Configuration that you've mentioned is enough to export files in the S3 as well as using it as the Online storage? Configuring the Object Storage ID for DCM4CHEE archive AE and the Exporter is for using the Nearline storage?

gunterze Apr 26, 2021
Maintainer

Caused by: java.net.PortUnreachableException: ICMP Port Unreachable

is an unrelated warning caused by WFLY-14625. s.o. #3105

vrindanayak Apr 26, 2021
Maintainer

Also, previously I was uploading new studies via dcmsnd command and it worked fine. Now it is throwing this error:

./dcmsnd DCM4CHEE@127.0.0.1:11112 ../../dcm

Scanning files to send
.
Scanned 1 files in 0.052s (=52ms/file)
log4j:WARN No appenders could be found for logger (org.dcm4che2.net.NetworkConnection).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to DCM4CHEE@127.0.0.1:11112 in 0.042s
ERROR: Received RSP with Status 0110H for ../../dcm/0012.dcm, cuid=1.2.840.10008.5.1.4.1.1.12.1, tsuid=1.2.840.10008.1.2.4.50
(0000,0100) US #2 [32769] Command Field
(0000,0120) US #2 [1] Message ID Being Responded To
(0000,0800) US #2 [257] Command Data Set Type
(0000,0900) US #2 [272] Status
(0000,0902) LO #64 [java.io.IOException: No space left on configured storage systems] Error Comment

F
Sent 0 objects (=0.0KB) in 0.057s (=0.0KB/s)
Released connection to DCM4CHEE@127.0.0.1:1111

Also, does it mean doing ONLY the Storage Configuration that you've mentioned is enough to export files in the S3 as well as using it as the Online storage? Configuring the Object Storage ID for DCM4CHEE archive AE and the Exporter is for using the Nearline storage?

If you just want to use Amazon S3 as your ONLINE storage for studies received by archive, then Storage Configuration denoted by dcmStorageID=amazon-s3 should be sufficient. Additionally, ensure on DCM4CHEE Archive AE Extension level to change the Object Storage ID from fs1 to amazon-s3. You don't need an Exporter for this, as I mentioned those were just the two test cases I tried to test Amazon S3 as storage system.

Regarding No space left on configured storage systems, which storage have you configured as ONLINE. What is the value of
Object Storage ID on DCM4CHEE Archive AE Extension level? You may choose to configure multiple storages - See https://groups.google.com/g/dcm4che/c/L0zQVesgMIU/m/kuNFMonFCQAJ

In dcm4chee 5.x, you may configure multiple Object Storage ID(s) of Archive Network AE and specify a Storage Threshold for each referred Storage. If you want to fill Storage Systems sequential one by one, keep Object Storage Count = 1. The Storage which Storage ID comes first in literal order will be filled first. So you may choose a naming pattern like fs001, fs002, fs003, ....

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 as storage: Working example #3146

{{title}}

Replies: 3 comments 10 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

S3 as storage: Working example #3146

proteeti13 Apr 20, 2021

I did the following steps:

Result:

Configuration

Storage Descriptor

Replies: 3 comments · 10 replies

vrindanayak Apr 20, 2021 Maintainer

proteeti13 Apr 21, 2021 Author

vrindanayak Apr 21, 2021 Maintainer

proteeti13 Apr 21, 2021 Author

proteeti13 Apr 21, 2021 Author

vrindanayak Apr 21, 2021 Maintainer

proteeti13 Apr 22, 2021 Author

vrindanayak Apr 22, 2021 Maintainer

Configuration

Storage Descriptor

Exporter configuration

Settings in AWS :

Verification

Exporting existing study from archive to Amazon S3

Using Amazon S3 as Object Storage for studies received by archive

proteeti13 Apr 26, 2021 Author

vrindanayak Apr 26, 2021 Maintainer

proteeti13 Apr 26, 2021 Author

gunterze Apr 26, 2021 Maintainer

vrindanayak Apr 26, 2021 Maintainer

proteeti13
Apr 20, 2021

Replies: 3 comments 10 replies

vrindanayak
Apr 20, 2021
Maintainer

proteeti13 Apr 21, 2021
Author

vrindanayak
Apr 21, 2021
Maintainer

proteeti13 Apr 21, 2021
Author

proteeti13 Apr 21, 2021
Author

vrindanayak Apr 21, 2021
Maintainer

proteeti13 Apr 22, 2021
Author

vrindanayak
Apr 22, 2021
Maintainer

proteeti13 Apr 26, 2021
Author

vrindanayak Apr 26, 2021
Maintainer

proteeti13 Apr 26, 2021
Author

gunterze Apr 26, 2021
Maintainer

vrindanayak Apr 26, 2021
Maintainer