Backup of Nearline Storage System

Overview

• Nearline storage is configured to have backup of objects on online cache storage system. The objects shall get deleted from the online cache storage system, once objects have been exported to nearline storage system.
• Now assume that the nearline storage (from a different provider, eg. S3) system is also getting full day by day, which calls for backup of objects on this nearline storage system to an external archive, so that the online cache storage system can continue to backup the objects without any disruptions.

Note : The use case here is very similar to Replacement of Nearline Storage System. The only difference is that, instead of replacing the old nearline storage system altogether, the objects are sent to external archive (which acts as long term archive) from where they can be retrieved later.

Configuration

• Assume you already have an online cache storage and nearline storage configuration as shown below.

• Note here that online cache storage is acting as retrieve cache for nearline storage system (dcmRetrieveCacheStorageID: cache1).

• Nearline storage acts as export storage for online cache storage (dcmExportStorageID: nearline1).

• Also, cache1 is referred as Object Storage ID for the DCM4CHEE Application Entity.

    version: 1
    dn: dcmStorageID=cache,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dcmInstanceAvailability: ONLINE
    dcmProperty: pathFormat={now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
    dcmProperty: checkMountFile=NO_MOUNT
    objectClass: dcmStorage
    dcmStorageID: cache
    dcmStorageDuration: CACHE
    dcmExportStorageID: nearline
    dcmURI: ${jboss.server.data.url}/cache/
    
    dn: dcmStorageID=nearline,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dcmInstanceAvailability: NEARLINE
    dcmProperty: pathFormat={now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
    dcmProperty: checkMountFile=NO_MOUNT
    objectClass: dcmStorage
    dcmStorageID: nearline
    dcmStorageDuration: PERMANENT
    dcmRetrieveCacheStorageID: cache
    dcmURI: ${jboss.server.data.url}/nearline/
  
    dn: dicomAETitle=DCM4CHEE,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    objectClass: dcmArchiveNetworkAE
    objectClass: dcmNetworkAE
    objectClass: dicomNetworkAE
    dicomAETitle: DCM4CHEE
    dicomAssociationAcceptor: TRUE
    dicomAssociationInitiator: TRUE
    dicomNetworkConnectionReference: cn=dicom,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dicomNetworkConnectionReference: cn=dicom-tls,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dcmAcceptedUserRole: user
    dcmAcceptedUserRole: admin
    dcmMetadataStorageID: metadata
    dcmObjectStorageID: cache
    dcmQueryRetrieveViewID: hideRejected
    dcmUpdateLocationStatusOnRetrieve: FALSE
    dicomDescription: Hide instances rejected for Quality Reasons
  

Backing up of objects present on old nearline storage system is explained below :

• To do this, add a Remote Application Entity for the external archive.

• Add a separate retrieve cache storage system. Ensure it has a deleter threshold set to it and also it is cache storage.

• Newly created Retrieve cache storage is acting as retrieve cache for the nearline storage system (dcmRetrieveCacheStorageID: rc).

• Nearline storage system shall start acting as cache for the external archive (dcmStorageDuration: CACHE).

• External Archive AET shall be configured as External Retrieve AET for the nearline storage (dcmExternalRetrieveAET: TEST).

• Configure also an exporter (with storage commitment) to export objects from nearline storage to external archive, and once exported shall send storage commitment request to external archive. This exporter shall also delete objects from rc (dcmDeleteStudyFromStorageID: rc). Ensure that external archive configuration has your AE (which acts as Storage Commitment SCU), since it is needed for storage commitment to work (in this case it is dicomAETitle: DCM4CHEE).

    version: 1        
    dn: dcmStorageID=rc,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dcmInstanceAvailability: ONLINE
    dcmProperty: pathFormat={now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
    dcmProperty: checkMountFile=NO_MOUNT
    objectClass: dcmStorage
    dcmStorageID: rc
    dcmStorageDuration: CACHE
    dcmDeleterThreshold: 500MB
    dcmURI: ${jboss.server.data.url}/rc/        
    
    dn: dcmStorageID=nearline,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    dcmInstanceAvailability: NEARLINE
    dcmProperty: pathFormat={now,date,yyyy/MM/dd}/{0020000D,hash}/{0020000E,hash}/{00080018,hash}
    dcmProperty: checkMountFile=NO_MOUNT
    objectClass: dcmStorage
    dcmStorageID: nearline
    dcmStorageDuration: CACHE
    dcmRetrieveCacheStorageID: rc
    dcmExternalRetrieveAET: TEST
    dcmURI: ${jboss.server.data.url}/nearline/
    
    dn: dcmExporterID=ExportFromNearlineToExternalArchive,dicomDeviceName=dcm4chee-arc,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    objectClass: dcmExporter
    dcmExporterID: ExportFromNearlineToExternalArchive
    dcmQueueName: Export1
    dcmURI: dicom:TEST
    dcmStgCmtSCP: TEST
    dcmDeleteStudyFromStorageID: rc
    dicomAETitle: DCM4CHEE
    dicomDescription: Export from nearline to external archive
  

External Archive Configuration :

    version: 1
    
    dn: dicomDeviceName=test-ng,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    objectClass: dcmDevice
    objectClass: dicomDevice
    dicomDeviceName: test-ng
    dicomInstalled: TRUE
    
    dn: cn=dicom,dicomDeviceName=test-ng,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    objectClass: dcmNetworkConnection
    objectClass: dicomNetworkConnection
    dicomHostname: test-ng
    cn: dicom
    dicomPort: 11112
    
    dn: dicomAETitle=TEST,dicomDeviceName=test-ng,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org
    objectClass: dcmNetworkAE
    objectClass: dicomNetworkAE
    dicomAETitle: TEST
    dicomAssociationAcceptor: TRUE
    dicomAssociationInitiator: TRUE
    dicomNetworkConnectionReference: cn=dicom,dicomDeviceName=test-ng,cn=Devices,cn=DICOM Configuration,dc=dcm4che,dc=org

• Export the objects from nearline1 to external archive using RESTful service Export Matching Studies and passing the query parameter storageID=nearline1

• One may also do a C-Move from external archive to rc storage system to retrieve the objects when needed. To do this, configure a new AE Title(s) of the Archive and select only rc as Object Storage ID in its Archive Network AE Extension. Use this newly configured AE as destination in the C-Move. Ensure that external archive configuration has your newly configured AE which is acting as the destination.