3d image patch extraction

.gitignore

**/.ipynb_checkpoints/*
ok.png
ok.nii.gz
workdir/*

README.md

problem statement

When moving the patches generated by tf.extract_volume_patches or tf.image.extract_patches back to its original shape with tf.reshape or np.reshape methods, the result will be incorrect.

Above issue is explained in below extensively.

https://stackoverflow.com/questions/44047753/reconstructing-an-image-after-using-extract-image-patches

Note Stop right here if just need non overlapping patches:

• see if np.reshape fits your bill - which usually does, then use that! (np-extract-patches.ipynb)

https://stackoverflow.com/questions/31527755/extract-blocks-or-patches-from-numpy-array

• also for 2d images, for non-overlapping patches, one can use tf.nn.space_to_depth and tf.nn.depth_to_space. See tf-space-to-depth.ipynb.

• alternatively if you prefer c++, git ITK a try.

https://discourse.itk.org/t/patch-extraction/3630

https://itk.org/ITKExamples/src/Filtering/ImageGrid/ExtractRegionOfInterestInOneImage/Documentation.html

solutions

Several solutions to resolve the above issue are listed below:

• for 2d or 3d images, and overlapping patches, one could reorder the pixels/voxels, see tf-extract-patches-index.ipynb

• another solution by Marco Ancona is "to realize that the inverse of tf.extract_image_patches is its gradient" provided in below link. (see `tf-extract-patches-gradient.ipynb) https://gist.github.com/aewhite/14db960f9e832bce4041bf185cdc9615

• I have not tried this one yet - there is a pytorch solution by blanNL https://stackoverflow.com/questions/68150248/how-to-extract-overlapping-patches-from-a-3d-volume-and-recreate-the-input-shape

notebooks can be executed following below prep script within tf container.

docker run -it -p 8888:8888 -w /workdir -v $PWD:/workdir tensorflow/tensorflow:2.6.1-gpu-jupyter bash
bash prep.sh
jupyter notebook --ip=* --allow-root

additional thoughts

whatever solutions you end up using, doesn't really matter during training.

You will for sure get a performance hit during inference. For problems dealing with 3d patches, tf.extract_volume_patches may be one method, but for medical images, depending on hardware, OOM may occur.

• thus now you are back to for loops with python/numpy for preprocessing -> batch wise inference -> reshape

• alternatively, chop up the volume to cubes (faster than for loops) -> feed that asyncronously to multiple serving end points -> gather the results and do the reshape.

• wait until you get the budget to run jumbo gpus for inference 24/7.

reference

https://stackoverflow.com/questions/41564321/split-image-tensor-into-small-patches
https://stackoverflow.com/questions/44047753/reconstructing-an-image-after-using-extract-image-patches
https://stackoverflow.com/questions/40731433/understanding-tf-extract-image-patches-for-extracting-patches-from-an-image

https://stackoverflow.com/questions/53451728/n-dimensional-sliding-window-operation-in-python-using-gpu-accelerated-libraries
https://stackoverflow.com/questions/35691947/3d-sliding-window-operation-in-theano


vae patch
Discovering Digital Tumor Signatures—Using Latent Code Representations to Manipulate and Classify Liver Lesions
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8269051

pediatric-ct-seg
https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=89096588#89096588bcab02c187174a288dbcbf95d26179e8

extract-patch-lung.ipynb

np-extract-patches.ipynb

prep.sh

echo 'pip intalling stuff'
pip install -r requirements.txt

echo 'downloading 2d image'
curl --output ok.png https://i.imgur.com/euaZxKf.png 


echo 'downloading 3d image'
python utils.py

requirements.txt

pydicom
SimpleITK
imageio
scipy

tf-extract-patches-gradient.ipynb

tf-extract-patches-index.ipynb

tf-space-to-depth.ipynb

utils.py

import os
import sys
import requests
import json
import zipfile
import pydicom
import SimpleITK as sitk


def download_images(series_instance_uid,image_root="."):
    url = f"https://services.cancerimagingarchive.net/services/v4/TCIA/query/getImage?SeriesInstanceUID={series_instance_uid}"
    zip_file_path = os.path.join(image_root,series_instance_uid+'.zip')
    r = requests.get(url, allow_redirects=True)
    print(r.status_code)
    if r.status_code != 200:
        raise LookupError(f"ohoh {r.status_code}!")
    open(zip_file_path, 'wb').write(r.content)
    folder_path = os.path.join(image_root,series_instance_uid)
    with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
        zip_ref.extractall(folder_path)
    return folder_path


def imread(folder_path):
    file_list = [os.path.join(folder_path,x) for x in os.listdir(folder_path)]
    dicom_list = []
    for image_file in file_list:
        ds=pydicom.dcmread(image_file)
        dicom_list.append((ds.InstanceNumber,image_file))
        dicom_list = sorted(dicom_list,key=lambda x:x[0])
    dicom_names = [x[1] for x in dicom_list]
    reader = sitk.ImageSeriesReader()
    reader.SetFileNames(dicom_names)        
    img_obj = reader.Execute()
    return img_obj

if __name__ == "__main__":
    series_instance_uid='1.3.6.1.4.1.14519.5.2.1.6279.6001.113679818447732724990336702075'
    image_root = "/tmp/myimages"
    os.makedirs(image_root,exist_ok=True)

    folder_path = download_images(series_instance_uid,image_root=image_root)
    img_obj = imread(folder_path)
    writer = sitk.ImageFileWriter()    
    writer.SetFileName('ok.nii.gz')
    writer.SetUseCompression(True)
    writer.Execute(img_obj)

    ksizes = (1,100,100,1)
    strides = (1,5,5,1)
    rates = [1,1,1,1]
    padding = "SAME"
    
    input_shape = (1,100,100,1)
    volume = np.random.random(input_shape)
    print(volume.shape)
    i = tf.image.extract_patches(
        volume, ksizes, strides, rates, padding, name=None
    )


    print('input_shape',input_shape)
    print('ksizes',ksizes)
    print('strides',strides)
    print('input_shape[1]*input_shape[2]',float(input_shape[1])*float(input_shape[2]))
    print('output_shape',i.shape)
    
input_shape (1, 100, 100, 1)
ksizes (1, 100, 100, 1)
strides (1, 5, 5, 1)
input_shape[1]*input_shape[2] 10000.0
output_shape (1, 20, 20, 10000)
    

nn-unet inference implementation

https://github.com/MIC-DKFZ/nnUNet/blob/5e5a151649af7386a2885fe2529fe6aa49bd2866/nnunet/network_architecture/neural_network.py#L154

https://arxiv.org/pdf/2004.08108.pdf
method section F