upload-fhir-bundle.py

import json
import os
import requests


# FHIR server endpoint
URL = "http://localhost:8080/fhir/"

# FHIR server json header content
headers = {"Content-Type": "application/fhir+json;charset=utf-8"}

# This path is specific to my setup
dirTestPatients = '../../Synthea/output/fhir/'
# Loop over all files in the output folder in order to upload each json file for each patient.
for dirpath, dirnames, files in os.walk(dirTestPatients):

    for file_name in files:
        # Make sure the encoding is utf-8
        with open(dirTestPatients+file_name, "r", encoding='utf-8') as bundle_file:
                data = bundle_file.read()

        r = requests.post(url = URL, data = data, headers = headers)

        # Output file name that was processed + the status code
        print(file_name)
        print(r.status_code)

Hey Mr. Mayoral,

I have changed the script a little bit. I have added argument parsing, so the script won't contain any hard coded values anymore. Furthermore I have made the script create hospitals first, practitioners second and patients last. Because if you run your script the first time, the server will throw a lot of 500 errors (At least it did for me). I think that is because the "Patient" bundles reference resources that don't exist yet.

If you are curious I get the following response:

{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Failed to call access method: org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [insert into HFJ_RES_TAG (PARTITION_DATE, PARTITION_ID, TAG_ID, RES_ID, RES_TYPE, PID) values (?, ?, ?, ?, ?, ?)]; constraint [FKBFCJBAFTMIWR3RXKWSY23VNEO]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Failed to call access method: org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [insert into HFJ_RES_TAG (PARTITION_DATE, PARTITION_ID, TAG_ID, RES_ID, RES_TYPE, PID) values (?, ?, ?, ?, ?, ?)]; constraint [FKBFCJBAFTMIWR3RXKWSY23VNEO]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch"
  } ]
}

Modified Code:

import json
import os
import requests
import argparse

def send_data_to_fhir(file_name):
    """
    This function will open a file and send the json data to the FHIR server.
    """

    # Make sure the encoding is utf-8
    with open(dirTestPatients+file_name, "r", encoding='utf-8') as bundle_file:
        data = bundle_file.read()

    r = requests.post(url = URL, data = data, headers = headers)

    # Output file name that was processed + the status code
    print(file_name)
    print(r.status_code)

# Setting up argparse
parser = argparse.ArgumentParser(description='Uploads dummy data to a FHIR server.')
parser.add_argument('path', type=str, help='Path to the output directory of Synthea')
parser.add_argument('--url', default="http://localhost:8080/fhir/", help='URL to the FHIR server (default: "http://localhost:8080/fhir/")')
args = parser.parse_args()

# FHIR server endpoint
URL = args.url

# FHIR server json header content
headers = {"Content-Type": "application/fhir+json;charset=utf-8"}

# This path is specific to my setup
dirTestPatients = args.path
# Loop over all files in the output folder in order to upload each json file for each patient.
for dirpath, dirnames, files in os.walk(dirTestPatients):

    # Finding hospital information and creating the bundle first
    for file_name in files:
        if "hospitalInformation" in file_name:
            send_data_to_fhir(file_name)
            break

    # Finding practioners and creating the bundle second
    for file_name in files:
        if "practitionerInformation" in file_name:
            send_data_to_fhir(file_name)
            break

    # Creating the patients
    for file_name in files:
        if ("hospitalInformation" not in file_name) and ("practitionerInformation" not in file_name):
            send_data_to_fhir(file_name)

Then, it is possible to run the script with arguments.

Example (in my case the output of Synthea is located at "./output/fhir"):

python upload-fhir-bundle-modifed.py --url 'http://localhost:8080/fhir/' output/fhir/

After the script has finished, all resources should be created first try.

Diff between the two scripts:

┌─[✗]─[user@user-tobefilledbyoem]─[~/Projects/eHealth]
└──╼ $diff upload-fhir-bundle.py upload-fhir-bundle-modifed.py 
3a4
> import argparse
4a6,25
> def send_data_to_fhir(file_name):
>     """
>     This function will open a file and send the json data to the FHIR server.
>     """
> 
>     # Make sure the encoding is utf-8
>     with open(dirTestPatients+file_name, "r", encoding='utf-8') as bundle_file:
>         data = bundle_file.read()
> 
>     r = requests.post(url = URL, data = data, headers = headers)
> 
>     # Output file name that was processed + the status code
>     print(file_name)
>     print(r.status_code)
> 
> # Setting up argparse
> parser = argparse.ArgumentParser(description='Uploads dummy data to a FHIR server.')
> parser.add_argument('path', type=str, help='Path to the output directory of Synthea')
> parser.add_argument('--url', default="http://localhost:8080/fhir/", help='URL to the FHIR server (default: "http://localhost:8080/fhir/")')
> args = parser.parse_args()
7c28
< URL = "http://localhost:8080/fhir/"
---
> URL = args.url
13c34
< dirTestPatients = 'output/fhir/'
---
> dirTestPatients = args.path
16a38
>     # Finding hospital information and creating the bundle first
18,29c40,53
<         # Make sure the encoding is utf-8
<         with open(dirTestPatients+file_name, "r", encoding='utf-8') as bundle_file:
<                 data = bundle_file.read()
< 
<         r = requests.post(url = URL, data = data, headers = headers)
< 
<         # Output file name that was processed + the status code
<         print(file_name)
<         print(r.status_code)
<         if r.status_code == 500:
<             print(r.json())
<             #break
\ No newline at end of file
---
>         if "hospitalInformation" in file_name:
>             send_data_to_fhir(file_name)
>             break
> 
>     # Finding practioners and creating the bundle second
>     for file_name in files:
>         if "practitionerInformation" in file_name:
>             send_data_to_fhir(file_name)
>             break
> 
>     # Creating the patients
>     for file_name in files:
>         if ("hospitalInformation" not in file_name) and ("practitionerInformation" not in file_name):
>             send_data_to_fhir(file_name)
\ No newline at end of file

Hi Korkmatik,
Thanks! Much nicer now.
I did have the 500 errors as well. And the explanation makes sense. In a second try it worked, surely because the Organizations and Practitioners had already been created during the first run.
Cheers

Hi Korkmatik,
Thanks for the update code, this works for me now!!

Running a docker HAPI FHIR container
When I tried the code, it lists all JSON files followed by a 404 error.
Restart the container and find only entries for “Location” and “Organization”
Any suggestions as to why the 404 errors.

Hi,
was anything loaded into the FHIR-Server?
Is there a / missing in your url to the server? I think it should be http://localhost:8080/fhir/. That's what the code is using as default. You could also try leaving out the argument.
Cheers

Hi
Many thanks for taking the time to reply.
-The –url argument is defined as “http://localhost:8080/fhir/ “ (if a copy this URL to the browser, I get the Swagger UI) so it is correct and running
-The Synthea output is correct because it is reading the correct files before the 404 error. It does create
Locations and Organization but should create more resources that that (including Patients)
Rebuild the docker image and rerun script
First time
FileNotFoundError: [Errno 2] No such file or directory: 'H:/FHIRE/synthea/output/fhirhospitalInformation1683749514854.json'
Runnit again
hospitalInformation1683749514854.json
200
practitionerInformation1683749514854.json
400
Adam_Hirthe_1b039e27-16f0-85f6-fea3-83f94a66a397.json
404
Adan_Kshlerin_703f7f12-396b-fd3d-11b4-62607351522e.json
404
..etc

was this ever fixed, getting same error where it just creates “Location” and “Organization”