Add PACS Connector extension

[rbumm]

Please consider including the extension in the extension index.
It does only work in 4.13, does not in 4.11
Thank you !

[rbumm]

The demo CT dataset predefined in the extension disappeared over the weekend from the public server.
I will implement a different solution.

[lassoan]

Probably you need to start the test with uploading your test data set (if not available on the server already).

[rbumm]

I would like to show more information -> some  DICOM tags (PatientName, PatientID, StudyDate ..) in the python console after performing just the query. 

Are these included in tempDb ? Tried to analyze tempDB with 

```

tempDb = ctk.ctkDICOMDatabase()
tempDb.openDatabase(':memory:')
dicomQuery.query(tempDb)
   
import sqlite3

conn = sqlite3.connect(':memory:')
c = conn.cursor()
cmd = "SELECT * FROM Patients;"
c.execute(cmd)
rows = c.fetchall()
 

for row in rows:
    print(row)

#commit the changes to db
conn.commit()
#close the connection
conn.close()
```

This only gives error messages .

[lassoan]

Each :memory: data base is a different instance, so you won't be able to connect to in-memory database via low-level sqlite3 library calls. If you create a database file and you just want to see what's in it then there is no need to mess with Python scripting, you can use https://sqlitebrowser.org/ to browse its content.

You can access all data in the database using the ctkDICOMDatabase object. See some examples here. In the Query/Retrieve database we don't store references to files, we just store the database tables, so not all examples will work.

[pieper]

I have not used this in a long time, but the way I recall it working is that the query populates a dicom dataset with the tags that you want to match on and returns all the matching entries. So the server may only return you the values that are specifically in the filter list, or maybe it returns some other tags too. I don't recall if you can specify other tags through the ctkDICOMQuery interface or only the ones exposed in the ctkDICOMQueryWidget. The API could be extended to handle that case if needed.

[lassoan]

List of queried fields are defined here:

https://github.com/commontk/CTK/blob/2a7851a405bda83639c6bc43dee7aa1e0a4928d7/Libs/DICOM/Core/ctkDICOMQuery.cpp#L306-L317

https://github.com/commontk/CTK/blob/2a7851a405bda83639c6bc43dee7aa1e0a4928d7/Libs/DICOM/Core/ctkDICOMQuery.cpp#L439-L445

It could be nice to have an API to edit this list, maybe by adding the tags in the tagsToPrecache list to the query. The only complication I see is that in this tag list we don't know what query level they should be retrieved.

[rbumm]

Thanks to you both again, I got this working by simply storing the query database locally in a temp dir and then using sqlite3. 
The query returns all necessary tags in several tables.

So cool to have splite3 available in python and Slicer.

[rbumm]

Updated the code and tested with GEPACS. 

Now, this becomes really helpful.

[lassoan]

Using low-level sqlite3 API breaks the encapsulation: exposes private implementation details that are subject to change at any time. It just means that your code can break anytime when we make internal changes.

Was there anything that you could not accomplish with the CTK database class interface?

[rbumm]

I was not able to display the "Patients", "Studies" and "Series" tables in tempDb as well as not able to display all available column headers (tags). It is very interesting to experiment with the public database and check the inputs with all available tags.

 seriesForStudies in the ctkDICOMDatabase class returned empty sometimes, although there were series records in the tempDB table. 

And I was just happy to have the good old SQL available.

If there are better ways to do this - any ideas welcome

[lassoan]

There are CTK table widgets that can display the contents of database tables, so it should not be necessary to create and manually populate generic table widgets.

seriesForStudies in the ctkDICOMDatabase class returned empty sometimes, although there were series records in the tempDB table.

Things may be more complex than you assume, due to differences between raw and displayed fields, but of course simple things like accessing list of patients, studies, etc. Should not be a problem. Can you share a code snippet that did not work as expected?

[rbumm]

Sure: (one commit down) 

not seriesForStudyList

(see below) became true although there were series entries in the  series table and the patient has study containing series 

    patientList = tempDb.patients()
    if not patientList: 
        logging.info("No results.")
    else: 
        studyList = tempDb.studiesForPatient(patientList[0])
        if not studyList: 
            logging.info("Patient detected, but no studies for this patient available.")
        else:             
            seriesForStudyList = tempDb.seriesForStudy(studyList[0])
            if not seriesForStudyList: 
                logging.info("Patient and study detected, but no series for this patient and study available.")
            else:             
                for study in studyList:
                    slicer.app.processEvents()
                    for series in seriesForStudyList:
                        slicer.app.processEvents()
                        if queryFlag==0:
                            if dicomQuery.preferCGET:
                                logging.info(f" ... getting patientID:{study} STUDY:>{study}< SERIES:>{series}<")
                                success = dicomRetrieve.getSeries(str(study),str(series))
                                logging.info(f"  - {'success' if success else 'failed'}")
                            else: 
                                logging.info(f" ... moving STUDY:>{study}< SERIES:>{series}<")
                                success = dicomRetrieve.moveSeries(str(study),str(series))
                                logging.info(f"  - {'success' if success else 'failed'}")
                        else:
                             logging.info(f" ... detected STUDY:>{study}< SERIES:>{series}<")                             

[lassoan]

Can you also add code that you used to create tempDb?

[rbumm]

Yes:

 def process(self, queryFlag, checkNumberPatients, patientID, accessionNumber,modalities,seriesDescription,studyDate,\
               callingAETitle, calledAETitle,storageAETitle, calledHost,calledPort,preferCGET):
    """
    Run the processing algorithm.
    Can be used without GUI widget.
    """

    import subprocess
    from datetime import datetime


    import time
    startTime = time.time()
    logging.info('Processing started ...')

    if not patientID:
      raise ValueError("You need to specify a patientID.")

    if not callingAETitle:
      raise ValueError("callingAETitle missing.")

    if not calledAETitle:
      raise ValueError("calledAETitle missing.")

    if not storageAETitle:
      raise ValueError("storageAETitle missing.")

    if not calledHost:
      raise ValueError("calledHost missing.")

    if not calledPort:
      raise ValueError("calledPort missing.")
   
    # check status of DICOM listener, must be running to retrieve images
    if hasattr(slicer, 'dicomListener') and slicer.dicomListener.process is not None:
      newState = slicer.dicomListener.process.state()
    else:
      newState = 0
      
      .....
      
 # Query
    dicomQuery = ctk.ctkDICOMQuery()
    dicomQuery.callingAETitle = callingAETitle
    dicomQuery.calledAETitle = calledAETitle
    dicomQuery.host = calledHost
    dicomQuery.port = int(calledPort)
    dicomQuery.preferCGET = bool(preferCGET)
    
    if len(studyDate)>0: 
        dicomQuery.filters = {'ID':patientID, 'AccessionNumber':accessionNumber, 'Modalities':modalities,'Series':seriesDescription,'StartDate':studyDate,'EndDate':studyDate}    
    else:
        dicomQuery.filters = {'ID':patientID, 'AccessionNumber':accessionNumber, 'Modalities':modalities,'Series':seriesDescription}    
    
    # database for storing query results
    
    databaseFilePath = slicer.app.temporaryPath +'/query.sql'
    import os
    if os.path.exists(databaseFilePath):
        os.remove(databaseFilePath)

    tempDb = ctk.ctkDICOMDatabase()
    tempDb.openDatabase(databaseFilePath)
    dicomQuery.query(tempDb)
    
    .........
    
    # Retrieve
    dicomRetrieve = ctk.ctkDICOMRetrieve()
    dicomRetrieve.setDatabase(slicer.dicomDatabase)
    dicomRetrieve.keepAssociationOpen = True
    dicomRetrieve.connect("progress(QString)", print)
    dicomRetrieve.setCallingAETitle(dicomQuery.callingAETitle)
    dicomRetrieve.setCalledAETitle(dicomQuery.calledAETitle)
    dicomRetrieve.setHost(dicomQuery.host)
    dicomRetrieve.setPort(dicomQuery.port)
    dicomRetrieve.setMoveDestinationAETitle(dicomQuery.callingAETitle)
    

Query parameters were:
Public database as defined in the extension
PatientID = "*"
modality = "CT"

[lassoan]

Sorry for keep asking, but I would need a complete script that I can copy-paste into Slicer's Python console. Please include everything - server name, etc. and no ...-s.

[rbumm]

No problem, miraculously it works now on the public server, I will try in the hospital remotely

queryFlag = 0 # this is not query only

# Query
dicomQuery = ctk.ctkDICOMQuery()
dicomQuery.callingAETitle = "SLICER"
dicomQuery.calledAETitle = "ANYE"
dicomQuery.host = "dicomserver.co.uk"
dicomQuery.port = 11112
dicomQuery.preferCGET = True
dicomQuery.filters = {'ID':'*', 'AccessionNumber':'', 'Modalities':'CT','Series':'','StartDate':'','EndDate':''}    
    

tempDb = ctk.ctkDICOMDatabase()
tempDb.openDatabase('')
dicomQuery.query(tempDb)

# Retrieve
dicomRetrieve = ctk.ctkDICOMRetrieve()
dicomRetrieve.setDatabase(slicer.dicomDatabase)
dicomRetrieve.keepAssociationOpen = True
dicomRetrieve.connect("progress(QString)", print)
dicomRetrieve.setCallingAETitle(dicomQuery.callingAETitle)
dicomRetrieve.setCalledAETitle(dicomQuery.calledAETitle)
dicomRetrieve.setHost(dicomQuery.host)
dicomRetrieve.setPort(dicomQuery.port)
dicomRetrieve.setMoveDestinationAETitle(dicomQuery.callingAETitle)
    
    
patientList = tempDb.patients()
if not patientList: 
    logging.info("No results.")
else: 
    studyList = tempDb.studiesForPatient(patientList[0])
    if not studyList: 
        logging.info("Patient detected, but no studies for this patient available.")
    else:             
        seriesForStudyList = tempDb.seriesForStudy(studyList[0])
        if not seriesForStudyList: 
            logging.info("Patient and study detected, but no series for this patient and study available.")
        else:             
            for study in studyList:
                slicer.app.processEvents()
                for series in seriesForStudyList:
                    slicer.app.processEvents()
                    if queryFlag==0:
                        if dicomQuery.preferCGET:
                            logging.info(f" ... getting patientID:{study} STUDY:>{study}< SERIES:>{series}<")
                            success = dicomRetrieve.getSeries(str(study),str(series))
                            logging.info(f"  - {'success' if success else 'failed'}")
                        else: 
                            logging.info(f" ... moving STUDY:>{study}< SERIES:>{series}<")
                            success = dicomRetrieve.moveSeries(str(study),str(series))
                            logging.info(f"  - {'success' if success else 'failed'}")
                    else:
                            logging.info(f" ... detected STUDY:>{study}< SERIES:>{series}<")

[rbumm]

On the GEPACS I get the error:

Patient and study detected, but no series for this patient and study available.

although I queried a case with two studies and several series. 

Can´t test the public server from the hospital because the external network access is blocked and I get a "no results" error from my extension.

[lassoan]

Have you only received a single item in studyList? Are the StudyInstanceUID values in the studyList correct?

[rbumm]

When I only put the correct patientID, I receive all the seven studies the patient has. 

When I additionally put "Lung" into series description, I just see that I still receive the seven studies, but (in SQL) I see that I only - correctly -  get the two series that have a lung CT. 

That is possibly why I get this error, Will try to eliminate this and report back.

[rbumm]

I got it working like this

    patientList = tempDb.patients()
    if not patientList: 
        logging.info("No results.")
        return
    for patient in patientList: 
        studyList = tempDb.studiesForPatient(patient)
        if not studyList: 
            continue
        else:
            for study in studyList: 
                seriesForStudyList = tempDb.seriesForStudy(study)
                if not seriesForStudyList: 
                    continue
                else:             
                    slicer.app.processEvents()
                    for series in seriesForStudyList:
                        slicer.app.processEvents()
                        if queryFlag==0:
                            if dicomQuery.preferCGET:
                                logging.info(f" ... getting patientID:{study} STUDY:>{study}< SERIES:>{series}<")
                                success = dicomRetrieve.getSeries(str(study),str(series))
                                logging.info(f"  - {'success' if success else 'failed'}")
                            else: 
                                logging.info(f" ... moving STUDY:>{study}< SERIES:>{series}<")
                                success = dicomRetrieve.moveSeries(str(study),str(series))
                                logging.info(f"  - {'success' if success else 'failed'}")
                        else:
                             logging.info(f" ... detected STUDY:>{study}< SERIES:>{series}<")

Will update github later .... Thank you for pointing that out !

[lassoan]

This looks good. Those hardcoded [0]-s did not look good.

Also note that dicomRetrieve stores the received series in the main database, not in tempDb and not all IDs are globally unique identifiers, so for example a patient ID that you get from tempDb are not transferable to the patientID in the main database.

[rbumm]

Ok interesting, thank you

[rbumm]

Was aware of the fact that I write to the main database - Why not transferable ? What would happen If I tried the transfer ?

[lassoan]

For example, PatientID is an auto-counter, which assigns simple integers to patients as IDs. These integers do not mean the same patients in tempDb and slicer.dicomDatabase.

For example:

>>> tempDb.patients()
('1', '2', '3', '4', '5', '6', '7')
>>> 
>>> slicer.dicomDatabase.patients()
('1', '6', '7', '8')

PatientID=1 in the two databases may refer to completely different patients, while PatientID=2 in one database may be the same as PatientID=6 in the other.

Study and series ID's use DICOM universally unique identifiers StudyInstanceUID and SeriesInstanceUID, and therefore they are the same across all databases. Unfortunately, DICOM does not provide UUID for patients.

[rbumm]

Thanks Andras, all I can say is - relogged to our GEPACS to confirm - test cases in Slicers main database receive the correct GEPACS and hospitalwide patientID upon retrieve and repeated retrieve does not produce additional datasets in the main database

[lassoan]

"Patient ID" DICOM tag is different from the "PatientUID" database field. "Patient ID" may be anything (defined by the hospital) and is not guaranteed to uniquely identify a person (it may be just an internal ID used in the hospital). "PatientUID" database field is guaranteed to uniquely identify the patient, but only within the database.

[rbumm]

Good progress, I was able to use the code committed today for the en bloc retrieval of 15 cases from a GEPACS with seriesDescription="Lung".
The process lasted 5777 s, retrieved only lung series from the CT studies as planned and produced a DICOM database directory of 18 GB, which is something we still can handle. 
I a first attempt I chained the patientID  tags seperated by semicolons like xxxxxxx;xxxxxxx;xxxxxxx; still aware of the fact that in some cases people might use the semicolon in patientID (will they ?) . 
Question 1: Is it safe to automatically delete the "incoming" directory in the DICOM database directory ? "dicom" and "incoming" have the same size of 8.72 GB. In a previous test I did not see a problem after a delete.

[rbumm]

Question 2 was: Can I speed this up? But I withdraw the question :-)

[pieper]

Question 1: Is it safe to automatically delete the "incoming" directory in the DICOM database directory ? "dicom" and "incoming" have the same size of 8.72 GB. In a previous test I did not see a problem after a delete.

yes, that should be no problem. It probably should happen automatically but perhaps it was left in by mistake during debugging.

Question 2 was: Can I speed this up? But I withdraw the question :-)

Did you withdraw because the current speed is fast enough? Our implementation is not optimized and there probably tricks that production systems used to speed things up, like overlapping network requests.

Anyway glad things are basically working for you!

[rbumm]

Thanks Steve, that was a little irony after reading the Slicer on 25000 Mayo computers thread (which is highly interesting). A retrieval could be a bit faster, but basically no problem because if it is all nicely organized in the DICOM database after 90 min one can plan ahead.

[lassoan]

Question 1: Is it safe to automatically delete the "incoming" directory in the DICOM database directory ? "dicom" and "incoming" have the same size of 8.72 GB. In a previous test I did not see a problem after a delete.

yes, that should be no problem. It probably should happen automatically but perhaps it was left in by mistake during debugging.

@rbumm Please check if files are left over if you use the GUI. If they are then please file a bug report, as only one instance should be kept of each file.

[lassoan]

@rbumm What is the status of this extension?

[rbumm]

@rbumm What is the status of this extension?

This is working and I am using it from time to time if I need to batch retrieve DICOM data from our local PACS. The public test server access is functional.

[rbumm]

I checked this extension again and see that the output handling using the Python console is not very elegant. On the other hand, it is helpful for debugging and retrieving batch DICOM data. Any suggestions on what to do with this are welcome. If I invest time in a better query output - where should I put it?

[jcfr]

Before we cab move forward with the integration, I suggest you review and address the following issues:

• Missing License rbumm/SlicerPACSConnector#2
• Selecting the PACSConnector module reports invalid QLineEdit connections rbumm/SlicerPACSConnector#3
• Clicking on Query PACS only with default value reports "Failed to establish association" rbumm/SlicerPACSConnector#4

[lassoan]

@Punzo is working on a new web browser that can query a remote server, show results along with thumbnails, and automatically retrieve data. Maybe it is worth for waiting for this browser to land in Slicer and then check if a separate extension is still needed or the features implemented in that can be integrated into this new browser.

@Punzo It seems that this extension supports retrieving multiple patients (e.g., patients that are scheduled to be reviewed on that day), which we would be needed for the funding project, too. So, in addition to the automatic retrieve while browsing data in the currently selected patient, it should be possible to explicitly request retrieve of complete patient, studies, or series (they would be low-priority background tasks).

[Punzo]

@Punzo is working on a new web browser that can query a remote server, show results along with thumbnails, and automatically retrieve data. Maybe it is worth for waiting for this browser to land in Slicer and then check if a separate extension is still needed or the features implemented in that can be integrated into this new browser.

we are currently testing the new browser and adding the last features. Probably it will take few more weeks to have a working PR (current branch at https://github.com/Punzo/CTK/tree/addDICOMVisualNavigation)

@Punzo It seems that this extension supports retrieving multiple patients (e.g., patients that are scheduled to be reviewed on that day), which we would be needed for the funding project, too. So, in addition to the automatic retrieve while browsing data in the currently selected patient, it should be possible to explicitly request retrieve of complete patient, studies, or series (they would be low-priority background tasks).

ok I will add this in my to do list