dm_out.txt

2020-10-29 7:15:17.47: =============================== logger created =======================================
2020-10-29 7:15:17.47: 
2020-10-29 7:15:17.47: ======================== Starting new session ============================
2020-10-29 7:15:17.47: Command line arguments given: 
Namespace(device='cuda', model_cfg='./outputs/dm_e1/holdout_0/0/modelConfig.cfg', reset_trainer=False, saved_model=None, test_cfg=None, train_cfg='./outputs/dm_e1/holdout_0/0/trainConfig.cfg')
2020-10-29 7:15:18.08: Available devices to Tensorflow:
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 6632431662447816985
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 14224203377801602860
physical_device_desc: "device: XLA_CPU device"
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 11795752551
locality {
  bus_id: 1
  links {
  }
}
incarnation: 5277204814880853623
physical_device_desc: "device: 0, name: Tesla P100-PCIE-12GB, pci bus id: 0000:58:00.0, compute capability: 6.0"
, name: "/device:XLA_GPU:0"
device_type: "XLA_GPU"
memory_limit: 17179869184
locality {
}
incarnation: 16639232827135720756
physical_device_desc: "device: XLA_GPU device"
]
2020-10-29 7:15:18.08: CONFIG: The configuration file for the [model] given is: /gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/modelConfig.cfg
2020-10-29 7:15:18.08: =============================================================
2020-10-29 7:15:18.08: ========== PARAMETERS FOR MAKING THE ARCHITECTURE ===========
2020-10-29 7:15:18.08: =============================================================
2020-10-29 7:15:18.08: CNN model's name = deepMedic
2020-10-29 7:15:18.08: ~~~~~~~~~~~~~~~~~~Model parameters~~~~~~~~~~~~~~~~
2020-10-29 7:15:18.08: Number of Classes (including background) = 2
2020-10-29 7:15:18.08: ~~Normal Pathway~~
2020-10-29 7:15:18.08: Number of Input Channels = 1
2020-10-29 7:15:18.08: Number of Layers = 8
2020-10-29 7:15:18.08: Number of Feature Maps per layer = [30, 30, 40, 40, 40, 40, 50, 50]
2020-10-29 7:15:18.08: Kernel Dimensions per layer = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]
2020-10-29 7:15:18.08: Receptive Field = [17, 17, 17]
2020-10-29 7:15:18.08: Residual connections added at the output of layers (indices from 0) = [3, 5, 7]
2020-10-29 7:15:18.08: Layers that will be made of Lower Rank (indices from 0) = []
2020-10-29 7:15:18.08: Lower Rank layers will be made of rank = []
2020-10-29 7:15:18.08: ~~Subsampled Pathway~~
2020-10-29 7:15:18.09: Use subsampled Pathway = True
2020-10-29 7:15:18.09: Number of subsampled pathways that will be built = 2
2020-10-29 7:15:18.09: Number of Layers (per sub-pathway) = [8, 8]
2020-10-29 7:15:18.09: Number of Feature Maps per layer (per sub-pathway) = [[30, 30, 40, 40, 40, 40, 50, 50], [30, 30, 40, 40, 40, 40, 50, 50]]
2020-10-29 7:15:18.09: Kernel Dimensions per layer = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]]
2020-10-29 7:15:18.09: Receptive Field = [17, 17, 17]
2020-10-29 7:15:18.09: Subsampling Factor per dimension (per sub-pathway) = [[3, 3, 3], [5, 5, 5]]
2020-10-29 7:15:18.09: Residual connections added at the output of layers (indices from 0) = [3, 5, 7]
2020-10-29 7:15:18.09: Layers that will be made of Lower Rank (indices from 0) = []
2020-10-29 7:15:18.09: Lower Rank layers will be made of rank = []
2020-10-29 7:15:18.09: ~~Fully Connected Pathway~~
2020-10-29 7:15:18.09: Number of additional FC layers (Excluding the Classif. Layer) = 2
2020-10-29 7:15:18.09: Number of Feature Maps in the additional FC layers = [250, 250]
2020-10-29 7:15:18.09: Residual connections added at the output of layers (indices from 0) = [1]
2020-10-29 7:15:18.09: Layers that will be made of Lower Rank (indices from 0) = []
2020-10-29 7:15:18.09: Dimensions of Kernels in the 1st FC layer (Classif. layer if no hidden FCs used) = [1, 1, 1]
2020-10-29 7:15:18.09: ~~Size Of Image Segments~~
2020-10-29 7:15:18.09: Size of Segments for Training = [37, 37, 37]
2020-10-29 7:15:18.09: Size of Segments for Validation = [17, 17, 17]
2020-10-29 7:15:18.09: Size of Segments for Testing = [45, 45, 45]
2020-10-29 7:15:18.09: ~~Dropout Rates~~
2020-10-29 7:15:18.09: Drop.R. for each layer in Normal Pathway = []
2020-10-29 7:15:18.09: Drop.R. for each layer in Subsampled Pathway = []
2020-10-29 7:15:18.09: Drop.R. for each layer in FC Pathway (additional FC layers + Classific.Layer at end) = [0.0, 0.5, 0.5]
2020-10-29 7:15:18.09: ~~Weight Initialization~~
2020-10-29 7:15:18.09: Initialization method and params for the conv kernel weights = ['fanIn', 2]
2020-10-29 7:15:18.09: ~~Activation Function~~
2020-10-29 7:15:18.09: Activation function to use = prelu
2020-10-29 7:15:18.09: ~~Batch Normalization~~
2020-10-29 7:15:18.09: Apply BN straight on pathways' inputs (eg straight on segments) = [False, False, True]
2020-10-29 7:15:18.09: Batch Normalization uses a rolling average for inference, over this many batches = 60
2020-10-29 7:15:18.09: ========== Done with printing session's parameters ==========
2020-10-29 7:15:18.09: =============================================================
2020-10-29 7:15:18.09: CONFIG: The configuration file for the [session] was loaded from: /gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/trainConfig.cfg
2020-10-29 7:15:18.12: 
2020-10-29 7:15:18.13: =============   NEW TRAINING SESSION     ==============

2020-10-29 7:15:18.13: 
2020-10-29 7:15:18.13: =============================================================
2020-10-29 7:15:18.13: ========= PARAMETERS FOR THIS TRAINING SESSION ==============
2020-10-29 7:15:18.13: =============================================================
2020-10-29 7:15:18.13: Session's name = trainSessionDm
2020-10-29 7:15:18.13: Model will be loaded from save = None
2020-10-29 7:15:18.13: ~~Output~~
2020-10-29 7:15:18.13: Main output folder = /gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0
2020-10-29 7:15:18.13: Log performance metrics for tensorboard = True
2020-10-29 7:15:18.13: Path and filename to save trained models = /gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/saved_models//trainSessionDm//deepMedic.trainSessionDm
2020-10-29 7:15:18.13: ~~~~~~~~~~~~~~~~~~Generic Information~~~~~~~~~~~~~~~~
2020-10-29 7:15:18.13: Number of Cases for Training = 103
2020-10-29 7:15:18.13: Number of Cases for Validation = 26
2020-10-29 7:15:18.13: ~~~~~~~~~~~~~~~~~~Training parameters~~~~~~~~~~~~~~~~
2020-10-29 7:15:18.13: Filepaths to Channels of the Training Cases = [['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1127/ISPY1_1127_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1127/ISPY1_1127_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1127/ISPY1_1127_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1166/ISPY1_1166_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1166/ISPY1_1166_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1166/ISPY1_1166_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1034/ISPY1_1034_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1034/ISPY1_1034_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1034/ISPY1_1034_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1117/ISPY1_1117_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1117/ISPY1_1117_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1117/ISPY1_1117_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1083/ISPY1_1083_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1083/ISPY1_1083_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1083/ISPY1_1083_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1003/ISPY1_1003_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1003/ISPY1_1003_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1003/ISPY1_1003_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1002/ISPY1_1002_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1002/ISPY1_1002_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1002/ISPY1_1002_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1239/ISPY1_1239_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1239/ISPY1_1239_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1239/ISPY1_1239_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1227/ISPY1_1227_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1227/ISPY1_1227_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1227/ISPY1_1227_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1074/ISPY1_1074_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1074/ISPY1_1074_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1074/ISPY1_1074_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1025/ISPY1_1025_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1025/ISPY1_1025_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1025/ISPY1_1025_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1008/ISPY1_1008_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1008/ISPY1_1008_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1008/ISPY1_1008_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1197/ISPY1_1197_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1197/ISPY1_1197_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1197/ISPY1_1197_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1209/ISPY1_1209_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1209/ISPY1_1209_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1209/ISPY1_1209_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1041/ISPY1_1041_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1041/ISPY1_1041_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1041/ISPY1_1041_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1175/ISPY1_1175_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1175/ISPY1_1175_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1175/ISPY1_1175_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1158/ISPY1_1158_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1158/ISPY1_1158_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1158/ISPY1_1158_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1152/ISPY1_1152_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1152/ISPY1_1152_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1152/ISPY1_1152_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1016/ISPY1_1016_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1016/ISPY1_1016_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1016/ISPY1_1016_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1090/ISPY1_1090_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1090/ISPY1_1090_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1090/ISPY1_1090_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1187/ISPY1_1187_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1187/ISPY1_1187_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1187/ISPY1_1187_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1049/ISPY1_1049_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1049/ISPY1_1049_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1049/ISPY1_1049_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1161/ISPY1_1161_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1161/ISPY1_1161_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1161/ISPY1_1161_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1032/ISPY1_1032_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1032/ISPY1_1032_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1032/ISPY1_1032_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1212/ISPY1_1212_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1212/ISPY1_1212_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1212/ISPY1_1212_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1189/ISPY1_1189_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1189/ISPY1_1189_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1189/ISPY1_1189_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1038/ISPY1_1038_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1038/ISPY1_1038_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1038/ISPY1_1038_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1220/ISPY1_1220_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1220/ISPY1_1220_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1220/ISPY1_1220_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1167/ISPY1_1167_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1167/ISPY1_1167_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1167/ISPY1_1167_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1149/ISPY1_1149_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1149/ISPY1_1149_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1149/ISPY1_1149_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1222/ISPY1_1222_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1222/ISPY1_1222_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1222/ISPY1_1222_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1042/ISPY1_1042_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1042/ISPY1_1042_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1042/ISPY1_1042_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1172/ISPY1_1172_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1172/ISPY1_1172_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1172/ISPY1_1172_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1124/ISPY1_1124_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1124/ISPY1_1124_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1124/ISPY1_1124_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1062/ISPY1_1062_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1062/ISPY1_1062_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1062/ISPY1_1062_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1114/ISPY1_1114_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1114/ISPY1_1114_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1114/ISPY1_1114_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1107/ISPY1_1107_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1107/ISPY1_1107_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1107/ISPY1_1107_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1137/ISPY1_1137_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1137/ISPY1_1137_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1137/ISPY1_1137_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1165/ISPY1_1165_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1165/ISPY1_1165_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1165/ISPY1_1165_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1070/ISPY1_1070_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1070/ISPY1_1070_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1070/ISPY1_1070_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1094/ISPY1_1094_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1094/ISPY1_1094_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1094/ISPY1_1094_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1030/ISPY1_1030_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1030/ISPY1_1030_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1030/ISPY1_1030_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1013/ISPY1_1013_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1013/ISPY1_1013_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1013/ISPY1_1013_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1156/ISPY1_1156_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1156/ISPY1_1156_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1156/ISPY1_1156_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1123/ISPY1_1123_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1123/ISPY1_1123_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1123/ISPY1_1123_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1184/ISPY1_1184_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1184/ISPY1_1184_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1184/ISPY1_1184_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1089/ISPY1_1089_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1089/ISPY1_1089_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1089/ISPY1_1089_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1203/ISPY1_1203_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1203/ISPY1_1203_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1203/ISPY1_1203_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1147/ISPY1_1147_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1147/ISPY1_1147_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1147/ISPY1_1147_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1168/ISPY1_1168_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1168/ISPY1_1168_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1168/ISPY1_1168_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1077/ISPY1_1077_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1077/ISPY1_1077_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1077/ISPY1_1077_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1088/ISPY1_1088_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1088/ISPY1_1088_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1088/ISPY1_1088_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1210/ISPY1_1210_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1210/ISPY1_1210_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1210/ISPY1_1210_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1176/ISPY1_1176_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1176/ISPY1_1176_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1176/ISPY1_1176_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1179/ISPY1_1179_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1179/ISPY1_1179_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1179/ISPY1_1179_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1188/ISPY1_1188_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1188/ISPY1_1188_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1188/ISPY1_1188_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1132/ISPY1_1132_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1132/ISPY1_1132_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1132/ISPY1_1132_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1128/ISPY1_1128_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1128/ISPY1_1128_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1128/ISPY1_1128_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1075/ISPY1_1075_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1075/ISPY1_1075_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1075/ISPY1_1075_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1225/ISPY1_1225_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1225/ISPY1_1225_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1225/ISPY1_1225_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1101/ISPY1_1101_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1101/ISPY1_1101_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1101/ISPY1_1101_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1009/ISPY1_1009_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1009/ISPY1_1009_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1009/ISPY1_1009_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1218/ISPY1_1218_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1218/ISPY1_1218_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1218/ISPY1_1218_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1207/ISPY1_1207_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1207/ISPY1_1207_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1207/ISPY1_1207_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1160/ISPY1_1160_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1160/ISPY1_1160_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1160/ISPY1_1160_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1196/ISPY1_1196_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1196/ISPY1_1196_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1196/ISPY1_1196_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1012/ISPY1_1012_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1012/ISPY1_1012_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1012/ISPY1_1012_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1163/ISPY1_1163_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1163/ISPY1_1163_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1163/ISPY1_1163_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1112/ISPY1_1112_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1112/ISPY1_1112_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1112/ISPY1_1112_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1087/ISPY1_1087_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1087/ISPY1_1087_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1087/ISPY1_1087_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1134/ISPY1_1134_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1134/ISPY1_1134_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1134/ISPY1_1134_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1224/ISPY1_1224_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1224/ISPY1_1224_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1224/ISPY1_1224_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1058/ISPY1_1058_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1058/ISPY1_1058_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1058/ISPY1_1058_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1053/ISPY1_1053_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1053/ISPY1_1053_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1053/ISPY1_1053_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1064/ISPY1_1064_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1064/ISPY1_1064_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1064/ISPY1_1064_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1236/ISPY1_1236_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1236/ISPY1_1236_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1236/ISPY1_1236_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1193/ISPY1_1193_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1193/ISPY1_1193_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1193/ISPY1_1193_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1017/ISPY1_1017_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1017/ISPY1_1017_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1017/ISPY1_1017_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1060/ISPY1_1060_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1060/ISPY1_1060_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1060/ISPY1_1060_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1073/ISPY1_1073_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1073/ISPY1_1073_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1073/ISPY1_1073_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1150/ISPY1_1150_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1150/ISPY1_1150_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1150/ISPY1_1150_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1155/ISPY1_1155_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1155/ISPY1_1155_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1155/ISPY1_1155_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1056/ISPY1_1056_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1056/ISPY1_1056_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1056/ISPY1_1056_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1234/ISPY1_1234_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1234/ISPY1_1234_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1234/ISPY1_1234_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1173/ISPY1_1173_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1173/ISPY1_1173_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1173/ISPY1_1173_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1106/ISPY1_1106_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1106/ISPY1_1106_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1106/ISPY1_1106_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1228/ISPY1_1228_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1228/ISPY1_1228_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1228/ISPY1_1228_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1072/ISPY1_1072_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1072/ISPY1_1072_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1072/ISPY1_1072_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1061/ISPY1_1061_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1061/ISPY1_1061_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1061/ISPY1_1061_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1223/ISPY1_1223_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1223/ISPY1_1223_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1223/ISPY1_1223_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1226/ISPY1_1226_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1226/ISPY1_1226_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1226/ISPY1_1226_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1235/ISPY1_1235_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1235/ISPY1_1235_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1235/ISPY1_1235_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1004/ISPY1_1004_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1004/ISPY1_1004_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1004/ISPY1_1004_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1174/ISPY1_1174_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1174/ISPY1_1174_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1174/ISPY1_1174_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1021/ISPY1_1021_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1021/ISPY1_1021_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1021/ISPY1_1021_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1096/ISPY1_1096_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1096/ISPY1_1096_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1096/ISPY1_1096_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1010/ISPY1_1010_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1010/ISPY1_1010_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1010/ISPY1_1010_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1205/ISPY1_1205_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1205/ISPY1_1205_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1205/ISPY1_1205_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1191/ISPY1_1191_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1191/ISPY1_1191_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1191/ISPY1_1191_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1144/ISPY1_1144_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1144/ISPY1_1144_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1144/ISPY1_1144_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1159/ISPY1_1159_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1159/ISPY1_1159_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1159/ISPY1_1159_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1136/ISPY1_1136_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1136/ISPY1_1136_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1136/ISPY1_1136_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1051/ISPY1_1051_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1051/ISPY1_1051_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1051/ISPY1_1051_DCE_0002_N3_zscored.nii.gz']]
2020-10-29 7:15:18.13: Filepaths to Ground-Truth labels of the Training Cases = ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1127/ISPY1_1127_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1166/ISPY1_1166_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1034/ISPY1_1034_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1117/ISPY1_1117_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1083/ISPY1_1083_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1003/ISPY1_1003_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1002/ISPY1_1002_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1239/ISPY1_1239_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1227/ISPY1_1227_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1074/ISPY1_1074_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1025/ISPY1_1025_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1008/ISPY1_1008_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1197/ISPY1_1197_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1209/ISPY1_1209_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1041/ISPY1_1041_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1175/ISPY1_1175_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1158/ISPY1_1158_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1152/ISPY1_1152_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1016/ISPY1_1016_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1090/ISPY1_1090_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1187/ISPY1_1187_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1049/ISPY1_1049_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1161/ISPY1_1161_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1032/ISPY1_1032_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1212/ISPY1_1212_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1189/ISPY1_1189_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1038/ISPY1_1038_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1220/ISPY1_1220_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1167/ISPY1_1167_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1149/ISPY1_1149_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1222/ISPY1_1222_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1042/ISPY1_1042_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1172/ISPY1_1172_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1124/ISPY1_1124_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1062/ISPY1_1062_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1114/ISPY1_1114_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1107/ISPY1_1107_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1137/ISPY1_1137_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1165/ISPY1_1165_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1070/ISPY1_1070_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1094/ISPY1_1094_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1030/ISPY1_1030_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1013/ISPY1_1013_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1156/ISPY1_1156_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1123/ISPY1_1123_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1184/ISPY1_1184_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1089/ISPY1_1089_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1203/ISPY1_1203_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1147/ISPY1_1147_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1168/ISPY1_1168_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1077/ISPY1_1077_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1088/ISPY1_1088_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1210/ISPY1_1210_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1176/ISPY1_1176_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1179/ISPY1_1179_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1188/ISPY1_1188_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1132/ISPY1_1132_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1128/ISPY1_1128_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1075/ISPY1_1075_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1225/ISPY1_1225_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1101/ISPY1_1101_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1009/ISPY1_1009_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1218/ISPY1_1218_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1207/ISPY1_1207_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1160/ISPY1_1160_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1196/ISPY1_1196_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1012/ISPY1_1012_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1163/ISPY1_1163_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1112/ISPY1_1112_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1087/ISPY1_1087_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1134/ISPY1_1134_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1224/ISPY1_1224_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1058/ISPY1_1058_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1053/ISPY1_1053_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1064/ISPY1_1064_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1236/ISPY1_1236_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1193/ISPY1_1193_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1017/ISPY1_1017_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1060/ISPY1_1060_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1073/ISPY1_1073_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1150/ISPY1_1150_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1155/ISPY1_1155_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1056/ISPY1_1056_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1234/ISPY1_1234_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1173/ISPY1_1173_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1106/ISPY1_1106_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1228/ISPY1_1228_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1072/ISPY1_1072_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1061/ISPY1_1061_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1223/ISPY1_1223_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1226/ISPY1_1226_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1235/ISPY1_1235_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1004/ISPY1_1004_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1174/ISPY1_1174_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1021/ISPY1_1021_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1096/ISPY1_1096_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1010/ISPY1_1010_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1205/ISPY1_1205_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1191/ISPY1_1191_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1144/ISPY1_1144_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1159/ISPY1_1159_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1136/ISPY1_1136_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1051/ISPY1_1051_mask.nii']
2020-10-29 7:15:18.13: ~~ Sampling (train) ~~
2020-10-29 7:15:18.13: Filepaths to ROI Masks of the Training Cases = None
2020-10-29 7:15:18.13: Type of Sampling = Per-Class (3)
2020-10-29 7:15:18.13: Sampling Categories = ['Class-0', 'Class-1']
2020-10-29 7:15:18.13: Percent of Samples to extract per Sampling Category = [0.5 0.5]
2020-10-29 7:15:18.13: Paths to weight-Maps for sampling of each category = None
2020-10-29 7:15:18.13: ~~Training Cycle~~
2020-10-29 7:15:18.13: Number of Epochs = 35
2020-10-29 7:15:18.13: Number of Subepochs per epoch = 20
2020-10-29 7:15:18.13: Number of cases to load per Subepoch (for extracting the samples for this subepoch) = 50
2020-10-29 7:15:18.13: Number of Segments loaded per subepoch for Training = 1000. NOTE: This number of segments divided by the batch-size defines the number of optimization-iterations that will be performed every subepoch!
2020-10-29 7:15:18.13: Batch size (train) = 10
2020-10-29 7:15:18.13: Number of parallel processes for sampling = 0
2020-10-29 7:15:18.13: ~~Learning Rate Schedule~~
2020-10-29 7:15:18.13: Type of schedule = predef
2020-10-29 7:15:18.13: [Predef] Predefined schedule of epochs when the LR will be lowered = [17, 22, 27, 30, 33]
2020-10-29 7:15:18.13: [Predef] When decreasing Learning Rate, divide LR by = 2.0
2020-10-29 7:15:18.13: [Poly] Initial epochs to wait before lowering LR = 10
2020-10-29 7:15:18.13: [Poly] Final epoch for the schedule = 35
2020-10-29 7:15:18.13: [Auto] Initial epochs to wait before lowering LR = 10
2020-10-29 7:15:18.13: [Auto] When decreasing Learning Rate, divide LR by = 2.0
2020-10-29 7:15:18.13: [Auto] Minimum increase in validation accuracy (0. to 1.) that resets the waiting counter = 0.0
2020-10-29 7:15:18.13: [Expon] (Deprecated) parameters = {'epochs_wait_before_decr': 10, 'final_ep_for_sch': 35, 'lr_to_reach_at_last_ep': 0.00390625, 'mom_to_reach_at_last_ep': 0.9}
2020-10-29 7:15:18.13: ~~Data Augmentation During Training~~
2020-10-29 7:15:18.13: Image level augmentation:
2020-10-29 7:15:18.13: Parameters for image-level augmentation: {'affine': <deepmedic.dataManagement.augmentImage.AugmenterAffineParams object at 0x7f2a7e357080>}
2020-10-29 7:15:18.13: 	 affine: OrderedDict([('prob', 0.0), ('max_rot_xyz', (45.0, 45.0, 45.0)), ('max_scaling', 0.1), ('seed', None), ('interp_order_imgs', 1), ('interp_order_lbls', 0), ('interp_order_roi', 0), ('interp_order_wmaps', 1), ('boundary_mode', 'nearest'), ('cval', 0.0)])
2020-10-29 7:15:18.13: Patch level augmentation:
2020-10-29 7:15:18.13: Mu and std for shift and scale of histograms = {'shift': {'mu': 0.0, 'std': 0.05}, 'scale': {'mu': 1.0, 'std': 0.01}}
2020-10-29 7:15:18.13: Probabilities of reflecting each axis = (0.5, 0.0, 0.0)
2020-10-29 7:15:18.13: Probabilities of rotating planes 0/90/180/270 degrees = {'xy': {'0': 0.0, '90': 0.0, '180': 0.0, '270': 0.0}, 'yz': {'0': 0.0, '90': 0.0, '180': 0.0, '270': 0.0}, 'xz': {'0': 0.0, '90': 0.0, '180': 0.0, '270': 0.0}}
2020-10-29 7:15:18.13: ~~~~~~~~~~~~~~~~~~Validation parameters~~~~~~~~~~~~~~~~
2020-10-29 7:15:18.13: Perform Validation on Samples throughout training? = True
2020-10-29 7:15:18.13: Perform Full Inference on validation cases every few epochs? = True
2020-10-29 7:15:18.13: Filepaths to Channels of the Validation Cases (Req for either of the above) = [['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_DCE_0002_N3_zscored.nii.gz'], ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_DCE_0000_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_DCE_0001_N3_zscored.nii.gz', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_DCE_0002_N3_zscored.nii.gz']]
2020-10-29 7:15:18.13: Filepaths to Ground-Truth labels of the Validation Cases = ['/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_mask.nii', '/cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_mask.nii']
2020-10-29 7:15:18.13: Filepaths to ROI masks for Validation Cases = None
2020-10-29 7:15:18.13: ~~~~~~~Validation on Samples throughout Training~~~~~~~
2020-10-29 7:15:18.13: Number of Segments loaded per subepoch for Validation = 5000
2020-10-29 7:15:18.13: Batch size (val on samples) = 50
2020-10-29 7:15:18.13: ~~ Sampling (val) ~~
2020-10-29 7:15:18.13: Type of Sampling = Uniform (1)
2020-10-29 7:15:18.13: Sampling Categories = ['Uniform']
2020-10-29 7:15:18.13: Percent of Samples to extract per Sampling Category = [1.0]
2020-10-29 7:15:18.13: Paths to weight-maps for sampling of each category = None
2020-10-29 7:15:18.13: ~~~~~Validation with Full Inference on Validation Cases~~~~~
2020-10-29 7:15:18.13: Perform Full-Inference on Val. cases every that many epochs = 5
2020-10-29 7:15:18.13: Batch size (val on whole volumes) = 10
2020-10-29 7:15:18.13: ~~Predictions (segmentations and prob maps on val. cases)~~
2020-10-29 7:15:18.13: Save Segmentations = True
2020-10-29 7:15:18.13: Save Probability Maps for each class = [True, True, True, True, True]
2020-10-29 7:15:18.13: Filepaths to save results per case = ['/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/predictions///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_mask.nii']
2020-10-29 7:15:18.13: Suffixes with which to save segmentations and probability maps = {'segm': 'Segm', 'prob': 'ProbMapClass'}
2020-10-29 7:15:18.13: ~~Feature Maps~~
2020-10-29 7:15:18.13: Save Feature Maps = False
2020-10-29 7:15:18.13: Min/Max Indices of FMs to visualise per pathway-type and per layer = None
2020-10-29 7:15:18.13: Filepaths to save FMs per case = ['/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_mask.nii', '/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/predictions/trainSessionDm/features///cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_mask.nii']
2020-10-29 7:15:18.13: ~~Optimization~~
2020-10-29 7:15:18.13: Initial Learning rate = 0.001
2020-10-29 7:15:18.13: Optimizer to use: SGD(0), Adam(1), RmsProp(2) = 2
2020-10-29 7:15:18.13: Parameters for Adam: b1= placeholder, b2=placeholder, e= placeholder
2020-10-29 7:15:18.13: Parameters for RmsProp: rho= 0.9, e= 0.0001
2020-10-29 7:15:18.13: Momentum Type: Classic (0) or Nesterov (1) = 1
2020-10-29 7:15:18.13: Momentum Non-Normalized (0) or Normalized (1) = 0
2020-10-29 7:15:18.13: Momentum Value = 0.6
2020-10-29 7:15:18.13: ~~Costs~~
2020-10-29 7:15:18.13: Loss functions and their weights = {'xentr': 1.0, 'iou': None, 'dsc': None}
2020-10-29 7:15:18.13: Reweight samples in cost on a per-class basis = {'type': None, 'prms': None, 'schedule': [0, 35]}
2020-10-29 7:15:18.14: L1 Regularization term = 1e-06
2020-10-29 7:15:18.14: L2 Regularization term = 0.0001
2020-10-29 7:15:18.14: ~~Freeze Weights of Certain Layers~~
2020-10-29 7:15:18.14: Indices of layers from each type of pathway that will be kept fixed (first layer is 0):
2020-10-29 7:15:18.14: Normal pathway's layers to freeze = []
2020-10-29 7:15:18.14: Subsampled pathway's layers to freeze = []
2020-10-29 7:15:18.14: FC pathway's layers to freeze = []
2020-10-29 7:15:18.14: ~~~~~~~~~~~~~~~~~~ PRE-PROCESSING ~~~~~~~~~~~~~~~~
2020-10-29 7:15:18.14: ~~Data Compabitibility Checks~~
2020-10-29 7:15:18.14: Check whether input data has correct format (can slow down process) = True
2020-10-29 7:15:18.14: ~~Padding~~
2020-10-29 7:15:18.14: Pad Input Images = True
2020-10-29 7:15:18.14: ~~Intensity Normalization~~
2020-10-29 7:15:18.14: Verbosity level = 0
2020-10-29 7:15:18.14: Z-Score parameters = {'apply_to_all_channels': True, 'apply_per_channel': None, 'cutoff_percents': [5.0, 95.0], 'cutoff_times_std': [3.0, 3.0], 'cutoff_below_mean': False}
2020-10-29 7:15:18.14: ========== Done with printing session's parameters ==========
2020-10-29 7:15:18.14: =============================================================

2020-10-29 7:15:18.14: =======================================================

2020-10-29 7:15:18.14: =========== Making the CNN graph... ===============
2020-10-29 7:15:18.14: ...Building the CNN model...
2020-10-29 7:15:18.14: [Pathway_NORMAL] is being built...
2020-10-29 7:15:18.14: 	[Pathway_NORMAL]: Input's Shape: (Train) [None, 1, 37, 37, 37], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 45, 45, 45]
2020-10-29 7:15:18.14: 	[Conv.Layer_0], Filter Shape: [30, 1, 3, 3, 3]
2020-10-29 7:15:18.14: 	[Conv.Layer_0], Input's Shape: (Train) [None, 1, 37, 37, 37], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 45, 45, 45]
2020-10-29 7:15:18.16: 	[Conv.Layer_1], Filter Shape: [30, 30, 3, 3, 3]
2020-10-29 7:15:18.16: 	[Conv.Layer_1], Input's Shape: (Train) [None, 30, 35, 35, 35], (Val) [None, 30, 15, 15, 15], (Test) [None, 30, 43, 43, 43]
2020-10-29 7:15:18.24: 	[Conv.Layer_2], Filter Shape: [40, 30, 3, 3, 3]
2020-10-29 7:15:18.24: 	[Conv.Layer_2], Input's Shape: (Train) [None, 30, 33, 33, 33], (Val) [None, 30, 13, 13, 13], (Test) [None, 30, 41, 41, 41]
2020-10-29 7:15:18.32: 	[Conv.Layer_3], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:18.32: 	[Conv.Layer_3], Input's Shape: (Train) [None, 40, 31, 31, 31], (Val) [None, 40, 11, 11, 11], (Test) [None, 40, 39, 39, 39]
2020-10-29 7:15:18.39: 	[Pathway_NORMAL]: making Residual Connection between output of [Layer_3] to input of previous layer.
2020-10-29 7:15:18.56: 	[Conv.Layer_4], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:18.56: 	[Conv.Layer_4], Input's Shape: (Train) [None, 40, 29, 29, 29], (Val) [None, 40, 9, 9, 9], (Test) [None, 40, 37, 37, 37]
2020-10-29 7:15:18.63: 	[Conv.Layer_5], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:18.63: 	[Conv.Layer_5], Input's Shape: (Train) [None, 40, 27, 27, 27], (Val) [None, 40, 7, 7, 7], (Test) [None, 40, 35, 35, 35]
2020-10-29 7:15:18.70: 	[Pathway_NORMAL]: making Residual Connection between output of [Layer_5] to input of previous layer.
2020-10-29 7:15:18.87: 	[Conv.Layer_6], Filter Shape: [50, 40, 3, 3, 3]
2020-10-29 7:15:18.87: 	[Conv.Layer_6], Input's Shape: (Train) [None, 40, 25, 25, 25], (Val) [None, 40, 5, 5, 5], (Test) [None, 40, 33, 33, 33]
2020-10-29 7:15:18.94: 	[Conv.Layer_7], Filter Shape: [50, 50, 3, 3, 3]
2020-10-29 7:15:18.94: 	[Conv.Layer_7], Input's Shape: (Train) [None, 50, 23, 23, 23], (Val) [None, 50, 3, 3, 3], (Test) [None, 50, 31, 31, 31]
2020-10-29 7:15:19.01: 	[Pathway_NORMAL]: making Residual Connection between output of [Layer_7] to input of previous layer.
2020-10-29 7:15:19.22: 	[Pathway_NORMAL]: Output's Shape: (Train) [None, 50, 21, 21, 21], (Val) [None, 50, 1, 1, 1], (Test) [None, 50, 29, 29, 29]
2020-10-29 7:15:19.22: [Pathway_NORMAL] done.
2020-10-29 7:15:19.22: [Pathway_SUBSAMPLED[3, 3, 3]] is being built...
2020-10-29 7:15:19.22: 	[Pathway_SUBSAMPLED[3, 3, 3]]: Input's Shape: (Train) [None, 1, 23, 23, 23], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 26, 26, 26]
2020-10-29 7:15:19.22: 	[Conv.Layer_0], Filter Shape: [30, 1, 3, 3, 3]
2020-10-29 7:15:19.22: 	[Conv.Layer_0], Input's Shape: (Train) [None, 1, 23, 23, 23], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 26, 26, 26]
2020-10-29 7:15:19.24: 	[Conv.Layer_1], Filter Shape: [30, 30, 3, 3, 3]
2020-10-29 7:15:19.24: 	[Conv.Layer_1], Input's Shape: (Train) [None, 30, 21, 21, 21], (Val) [None, 30, 15, 15, 15], (Test) [None, 30, 24, 24, 24]
2020-10-29 7:15:19.31: 	[Conv.Layer_2], Filter Shape: [40, 30, 3, 3, 3]
2020-10-29 7:15:19.31: 	[Conv.Layer_2], Input's Shape: (Train) [None, 30, 19, 19, 19], (Val) [None, 30, 13, 13, 13], (Test) [None, 30, 22, 22, 22]
2020-10-29 7:15:19.38: 	[Conv.Layer_3], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:19.38: 	[Conv.Layer_3], Input's Shape: (Train) [None, 40, 17, 17, 17], (Val) [None, 40, 11, 11, 11], (Test) [None, 40, 20, 20, 20]
2020-10-29 7:15:19.45: 	[Pathway_SUBSAMPLED[3, 3, 3]]: making Residual Connection between output of [Layer_3] to input of previous layer.
2020-10-29 7:15:19.62: 	[Conv.Layer_4], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:19.62: 	[Conv.Layer_4], Input's Shape: (Train) [None, 40, 15, 15, 15], (Val) [None, 40, 9, 9, 9], (Test) [None, 40, 18, 18, 18]
2020-10-29 7:15:19.69: 	[Conv.Layer_5], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:19.69: 	[Conv.Layer_5], Input's Shape: (Train) [None, 40, 13, 13, 13], (Val) [None, 40, 7, 7, 7], (Test) [None, 40, 16, 16, 16]
2020-10-29 7:15:19.76: 	[Pathway_SUBSAMPLED[3, 3, 3]]: making Residual Connection between output of [Layer_5] to input of previous layer.
2020-10-29 7:15:19.93: 	[Conv.Layer_6], Filter Shape: [50, 40, 3, 3, 3]
2020-10-29 7:15:19.93: 	[Conv.Layer_6], Input's Shape: (Train) [None, 40, 11, 11, 11], (Val) [None, 40, 5, 5, 5], (Test) [None, 40, 14, 14, 14]
2020-10-29 7:15:20.00: 	[Conv.Layer_7], Filter Shape: [50, 50, 3, 3, 3]
2020-10-29 7:15:20.00: 	[Conv.Layer_7], Input's Shape: (Train) [None, 50, 9, 9, 9], (Val) [None, 50, 3, 3, 3], (Test) [None, 50, 12, 12, 12]
2020-10-29 7:15:20.07: 	[Pathway_SUBSAMPLED[3, 3, 3]]: making Residual Connection between output of [Layer_7] to input of previous layer.
2020-10-29 7:15:20.25: 	[Pathway_SUBSAMPLED[3, 3, 3]]: Output's Shape: (Train) [None, 50, 7, 7, 7], (Val) [None, 50, 1, 1, 1], (Test) [None, 50, 10, 10, 10]
2020-10-29 7:15:20.25: [Pathway_SUBSAMPLED[3, 3, 3]] done.
2020-10-29 7:15:20.48: [Pathway_SUBSAMPLED[5, 5, 5]] is being built...
2020-10-29 7:15:20.48: 	[Pathway_SUBSAMPLED[5, 5, 5]]: Input's Shape: (Train) [None, 1, 21, 21, 21], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 22, 22, 22]
2020-10-29 7:15:20.48: 	[Conv.Layer_0], Filter Shape: [30, 1, 3, 3, 3]
2020-10-29 7:15:20.48: 	[Conv.Layer_0], Input's Shape: (Train) [None, 1, 21, 21, 21], (Val) [None, 1, 17, 17, 17], (Test) [None, 1, 22, 22, 22]
2020-10-29 7:15:20.49: 	[Conv.Layer_1], Filter Shape: [30, 30, 3, 3, 3]
2020-10-29 7:15:20.49: 	[Conv.Layer_1], Input's Shape: (Train) [None, 30, 19, 19, 19], (Val) [None, 30, 15, 15, 15], (Test) [None, 30, 20, 20, 20]
2020-10-29 7:15:20.56: 	[Conv.Layer_2], Filter Shape: [40, 30, 3, 3, 3]
2020-10-29 7:15:20.56: 	[Conv.Layer_2], Input's Shape: (Train) [None, 30, 17, 17, 17], (Val) [None, 30, 13, 13, 13], (Test) [None, 30, 18, 18, 18]
2020-10-29 7:15:20.68: 	[Conv.Layer_3], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:20.68: 	[Conv.Layer_3], Input's Shape: (Train) [None, 40, 15, 15, 15], (Val) [None, 40, 11, 11, 11], (Test) [None, 40, 16, 16, 16]
2020-10-29 7:15:20.75: 	[Pathway_SUBSAMPLED[5, 5, 5]]: making Residual Connection between output of [Layer_3] to input of previous layer.
2020-10-29 7:15:20.92: 	[Conv.Layer_4], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:20.92: 	[Conv.Layer_4], Input's Shape: (Train) [None, 40, 13, 13, 13], (Val) [None, 40, 9, 9, 9], (Test) [None, 40, 14, 14, 14]
2020-10-29 7:15:20.99: 	[Conv.Layer_5], Filter Shape: [40, 40, 3, 3, 3]
2020-10-29 7:15:20.99: 	[Conv.Layer_5], Input's Shape: (Train) [None, 40, 11, 11, 11], (Val) [None, 40, 7, 7, 7], (Test) [None, 40, 12, 12, 12]
2020-10-29 7:15:21.06: 	[Pathway_SUBSAMPLED[5, 5, 5]]: making Residual Connection between output of [Layer_5] to input of previous layer.
2020-10-29 7:15:21.24: 	[Conv.Layer_6], Filter Shape: [50, 40, 3, 3, 3]
2020-10-29 7:15:21.24: 	[Conv.Layer_6], Input's Shape: (Train) [None, 40, 9, 9, 9], (Val) [None, 40, 5, 5, 5], (Test) [None, 40, 10, 10, 10]
2020-10-29 7:15:21.31: 	[Conv.Layer_7], Filter Shape: [50, 50, 3, 3, 3]
2020-10-29 7:15:21.31: 	[Conv.Layer_7], Input's Shape: (Train) [None, 50, 7, 7, 7], (Val) [None, 50, 3, 3, 3], (Test) [None, 50, 8, 8, 8]
2020-10-29 7:15:21.38: 	[Pathway_SUBSAMPLED[5, 5, 5]]: making Residual Connection between output of [Layer_7] to input of previous layer.
2020-10-29 7:15:21.55: 	[Pathway_SUBSAMPLED[5, 5, 5]]: Output's Shape: (Train) [None, 50, 5, 5, 5], (Val) [None, 50, 1, 1, 1], (Test) [None, 50, 6, 6, 6]
2020-10-29 7:15:21.55: [Pathway_SUBSAMPLED[5, 5, 5]] done.
2020-10-29 7:15:21.79: DEBUG: Shape of the kernel of the first FC layer is : [1, 1, 1]
2020-10-29 7:15:21.79: DEBUG: Input to the FC Pathway will be padded by that many voxels per dimension: [0, 0, 0]
2020-10-29 7:15:21.79: [Pathway_FC] is being built...
2020-10-29 7:15:21.79: 	[Pathway_FC]: Input's Shape: (Train) [None, 150, 21, 21, 21], (Val) [None, 150, 1, 1, 1], (Test) [None, 150, 29, 29, 29]
2020-10-29 7:15:21.79: 	[Conv.Layer_0], Filter Shape: [250, 150, 1, 1, 1]
2020-10-29 7:15:21.79: 	[Conv.Layer_0], Input's Shape: (Train) [None, 150, 21, 21, 21], (Val) [None, 150, 1, 1, 1], (Test) [None, 150, 29, 29, 29]
2020-10-29 7:15:21.86: 	[Conv.Layer_1], Filter Shape: [250, 250, 1, 1, 1]
2020-10-29 7:15:21.86: 	[Conv.Layer_1], Input's Shape: (Train) [None, 250, 21, 21, 21], (Val) [None, 250, 1, 1, 1], (Test) [None, 250, 29, 29, 29]
2020-10-29 7:15:21.94: 	[Pathway_FC]: making Residual Connection between output of [Layer_1] to input of previous layer.
2020-10-29 7:15:22.12: 	[Conv.Layer_2], Filter Shape: [2, 250, 1, 1, 1]
2020-10-29 7:15:22.12: 	[Conv.Layer_2], Input's Shape: (Train) [None, 250, 21, 21, 21], (Val) [None, 250, 1, 1, 1], (Test) [None, 250, 29, 29, 29]
2020-10-29 7:15:22.19: 	[Pathway_FC]: Output's Shape: (Train) [None, 2, 21, 21, 21], (Val) [None, 2, 1, 1, 1], (Test) [None, 2, 29, 29, 29]
2020-10-29 7:15:22.19: [Pathway_FC] done.
2020-10-29 7:15:22.19: Adding the final Softmax Target layer...
2020-10-29 7:15:22.31: Finished building the CNN's model.
2020-10-29 7:15:22.31: =========== Building Trainer ===========

2020-10-29 7:15:22.31: Building Trainer.
2020-10-29 7:15:22.31: COST: Using cross entropy with weight: 1.0
2020-10-29 7:15:22.45: ...Initializing state of the optimizer...
2020-10-29 7:15:24.46: ----------- Creating Tensorboard Loggers -----------
2020-10-29 7:15:26.58: Loggers created successfully
2020-10-29 7:15:26.58: -----------=============================-----------
2020-10-29 7:15:26.58: =========== Compiling the Training Function ===========
2020-10-29 7:15:26.58: =======================================================

2020-10-29 7:15:30.90: ...Building the training function...
2020-10-29 7:15:30.94: ...Collecting ops and feeds for training...
2020-10-29 7:15:30.96: Done.
2020-10-29 7:15:30.96: =========== Compiling the Validation Function =========
2020-10-29 7:15:30.96: ...Building the validation function...
2020-10-29 7:15:30.96: ...Collecting ops and feeds for validation...
2020-10-29 7:15:30.97: Done.
2020-10-29 7:15:30.97: =========== Compiling the Testing Function ============
2020-10-29 7:15:30.97: ...Building the function for testing and visualisation of FMs...
2020-10-29 7:15:30.97: ...Collecting ops and feeds for testing...
2020-10-29 7:15:30.97: Done.
2020-10-29 7:15:31.55: =========== Initializing network and trainer variables  ===============
2020-10-29 7:16:50.10: All variables were initialized.
2020-10-29 7:16:50.10: Saving the initial model at:/gpfs/fs001/cbica/comp_space/bhaleram/DeepMedicWrapper/outputs/dm_e1/holdout_0/0/saved_models//trainSessionDm//deepMedic.trainSessionDm.initial.2020-10-29.07.16.50.098816
2020-10-29 7:16:50.25: 
2020-10-29 7:16:50.25: =======================================================
2020-10-29 7:16:50.25: ============== Training the CNN model =================
2020-10-29 7:16:50.25: =======================================================
2020-10-29 7:16:50.38: 
2020-10-29 7:16:50.38: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-29 7:16:50.38: ~~			 Starting new Epoch! Epoch #0/35  	~~
2020-10-29 7:16:50.38: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-29 7:16:50.38: 
2020-10-29 7:16:50.38: ***********************************************************************************
2020-10-29 7:16:50.38: *			 Starting new Subepoch: #0/20 		*
2020-10-29 7:16:50.38: ***********************************************************************************
2020-10-29 7:16:50.38: [MAIN|PID:131785] MULTIPROC: Before Validation in subepoch #0, submitting sampling job for next [VALIDATION].
2020-10-29 7:16:50.38: [VAL|SAMPLER|PID:131785] :=:=:=:=:=:=: Starting to sample for next [Validation]... :=:=:=:=:=:=:
2020-10-29 7:16:50.38: [VAL|SAMPLER|PID:131785] Out of [26] subjects given for [Validation], we will sample from maximum [50] per subepoch.
2020-10-29 7:16:50.38: [VAL|SAMPLER|PID:131785] Shuffled indices of subjects that were randomly chosen: [17, 16, 12, 14, 19, 6, 7, 13, 23, 21, 8, 3, 18, 24, 10, 11, 2, 1, 20, 4, 5, 15, 25, 9, 22, 0]
2020-10-29 7:16:50.38: [VAL|SAMPLER|PID:131785] Will sample from [26] subjects for next Validation...
2020-10-29 7:16:50.38: [VAL|JOB:0|PID:131785] Started. (#0/26) sampling job. Load & sample from subject of index (in user's list): 17
2020-10-29 7:16:50.38: [VAL|JOB:0|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1069/ISPY1_1069_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:16:50.92: [VAL|JOB:0|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:16:52.24: [VAL|JOB:0|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:16:52.24: [VAL|JOB:0|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:16:52.24: [VAL|JOB:1|PID:131785] Started. (#1/26) sampling job. Load & sample from subject of index (in user's list): 16
2020-10-29 7:16:52.24: [VAL|JOB:1|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1035/ISPY1_1035_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:16:52.61: [VAL|JOB:1|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:16:53.91: [VAL|JOB:1|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:16:53.91: [VAL|JOB:1|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:16:53.91: [VAL|JOB:2|PID:131785] Started. (#2/26) sampling job. Load & sample from subject of index (in user's list): 12
2020-10-29 7:16:53.91: [VAL|JOB:2|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1125/ISPY1_1125_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:16:54.53: [VAL|JOB:2|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:16:55.62: [VAL|JOB:2|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:16:55.62: [VAL|JOB:2|PID:131785] TIMING: [Load: 0.6] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:16:55.62: [VAL|JOB:3|PID:131785] Started. (#3/26) sampling job. Load & sample from subject of index (in user's list): 14
2020-10-29 7:16:55.62: [VAL|JOB:3|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1036/ISPY1_1036_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:16:56.23: [VAL|JOB:3|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:16:57.76: [VAL|JOB:3|PID:131785] Done. Samples per category: [Uniform: 195/195] 
2020-10-29 7:16:57.76: [VAL|JOB:3|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.2] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:16:57.76: [VAL|JOB:4|PID:131785] Started. (#4/26) sampling job. Load & sample from subject of index (in user's list): 19
2020-10-29 7:16:57.76: [VAL|JOB:4|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1099/ISPY1_1099_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:16:58.27: [VAL|JOB:4|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:16:59.55: [VAL|JOB:4|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:16:59.55: [VAL|JOB:4|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:16:59.55: [VAL|JOB:5|PID:131785] Started. (#5/26) sampling job. Load & sample from subject of index (in user's list): 6
2020-10-29 7:16:59.55: [VAL|JOB:5|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1104/ISPY1_1104_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:0.09: [VAL|JOB:5|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:1.43: [VAL|JOB:5|PID:131785] Done. Samples per category: [Uniform: 193/193] 
2020-10-29 7:17:1.43: [VAL|JOB:5|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:1.43: [VAL|JOB:6|PID:131785] Started. (#6/26) sampling job. Load & sample from subject of index (in user's list): 7
2020-10-29 7:17:1.43: [VAL|JOB:6|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1037/ISPY1_1037_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:2.15: [VAL|JOB:6|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:3.87: [VAL|JOB:6|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:3.87: [VAL|JOB:6|PID:131785] TIMING: [Load: 0.8] [Preproc: 1.4] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:3.87: [VAL|JOB:7|PID:131785] Started. (#7/26) sampling job. Load & sample from subject of index (in user's list): 13
2020-10-29 7:17:3.87: [VAL|JOB:7|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1145/ISPY1_1145_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:4.54: [VAL|JOB:7|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:6.20: [VAL|JOB:7|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:6.20: [VAL|JOB:7|PID:131785] TIMING: [Load: 0.7] [Preproc: 1.3] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:6.21: [VAL|JOB:8|PID:131785] Started. (#8/26) sampling job. Load & sample from subject of index (in user's list): 23
2020-10-29 7:17:6.21: [VAL|JOB:8|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1078/ISPY1_1078_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:6.81: [VAL|JOB:8|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:8.23: [VAL|JOB:8|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:8.23: [VAL|JOB:8|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:8.23: [VAL|JOB:9|PID:131785] Started. (#9/26) sampling job. Load & sample from subject of index (in user's list): 21
2020-10-29 7:17:8.23: [VAL|JOB:9|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1130/ISPY1_1130_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:8.81: [VAL|JOB:9|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:10.04: [VAL|JOB:9|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:10.04: [VAL|JOB:9|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:10.04: [VAL|JOB:10|PID:131785] Started. (#10/26) sampling job. Load & sample from subject of index (in user's list): 8
2020-10-29 7:17:10.04: [VAL|JOB:10|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1146/ISPY1_1146_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:10.58: [VAL|JOB:10|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:11.88: [VAL|JOB:10|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:11.88: [VAL|JOB:10|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:11.88: [VAL|JOB:11|PID:131785] Started. (#11/26) sampling job. Load & sample from subject of index (in user's list): 3
2020-10-29 7:17:11.88: [VAL|JOB:11|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1129/ISPY1_1129_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:12.57: [VAL|JOB:11|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:13.98: [VAL|JOB:11|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:13.98: [VAL|JOB:11|PID:131785] TIMING: [Load: 0.7] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:13.98: [VAL|JOB:12|PID:131785] Started. (#12/26) sampling job. Load & sample from subject of index (in user's list): 18
2020-10-29 7:17:13.98: [VAL|JOB:12|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1238/ISPY1_1238_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:14.43: [VAL|JOB:12|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:15.78: [VAL|JOB:12|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:15.78: [VAL|JOB:12|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:15.78: [VAL|JOB:13|PID:131785] Started. (#13/26) sampling job. Load & sample from subject of index (in user's list): 24
2020-10-29 7:17:15.78: [VAL|JOB:13|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1198/ISPY1_1198_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:16.28: [VAL|JOB:13|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:17.48: [VAL|JOB:13|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:17.48: [VAL|JOB:13|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:17.48: [VAL|JOB:14|PID:131785] Started. (#14/26) sampling job. Load & sample from subject of index (in user's list): 10
2020-10-29 7:17:17.48: [VAL|JOB:14|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1031/ISPY1_1031_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:18.03: [VAL|JOB:14|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:19.34: [VAL|JOB:14|PID:131785] Done. Samples per category: [Uniform: 195/195] 
2020-10-29 7:17:19.34: [VAL|JOB:14|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:19.34: [VAL|JOB:15|PID:131785] Started. (#15/26) sampling job. Load & sample from subject of index (in user's list): 11
2020-10-29 7:17:19.34: [VAL|JOB:15|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1171/ISPY1_1171_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:20.08: [VAL|JOB:15|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:22.02: [VAL|JOB:15|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:22.02: [VAL|JOB:15|PID:131785] TIMING: [Load: 0.8] [Preproc: 1.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:22.02: [VAL|JOB:16|PID:131785] Started. (#16/26) sampling job. Load & sample from subject of index (in user's list): 2
2020-10-29 7:17:22.02: [VAL|JOB:16|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1135/ISPY1_1135_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:22.40: [VAL|JOB:16|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:23.70: [VAL|JOB:16|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:23.70: [VAL|JOB:16|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:23.70: [VAL|JOB:17|PID:131785] Started. (#17/26) sampling job. Load & sample from subject of index (in user's list): 1
2020-10-29 7:17:23.70: [VAL|JOB:17|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1202/ISPY1_1202_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:24.30: [VAL|JOB:17|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:25.65: [VAL|JOB:17|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:25.65: [VAL|JOB:17|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:25.65: [VAL|JOB:18|PID:131785] Started. (#18/26) sampling job. Load & sample from subject of index (in user's list): 20
2020-10-29 7:17:25.65: [VAL|JOB:18|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1231/ISPY1_1231_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:26.17: [VAL|JOB:18|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:28.10: [VAL|JOB:18|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:28.10: [VAL|JOB:18|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:28.10: [VAL|JOB:19|PID:131785] Started. (#19/26) sampling job. Load & sample from subject of index (in user's list): 4
2020-10-29 7:17:28.10: [VAL|JOB:19|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1221/ISPY1_1221_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:28.50: [VAL|JOB:19|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:29.76: [VAL|JOB:19|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:29.76: [VAL|JOB:19|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:29.77: [VAL|JOB:20|PID:131785] Started. (#20/26) sampling job. Load & sample from subject of index (in user's list): 5
2020-10-29 7:17:29.77: [VAL|JOB:20|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1122/ISPY1_1122_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:30.18: [VAL|JOB:20|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:31.53: [VAL|JOB:20|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:31.53: [VAL|JOB:20|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:31.53: [VAL|JOB:21|PID:131785] Started. (#21/26) sampling job. Load & sample from subject of index (in user's list): 15
2020-10-29 7:17:31.53: [VAL|JOB:21|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1007/ISPY1_1007_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:32.10: [VAL|JOB:21|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:33.33: [VAL|JOB:21|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:33.33: [VAL|JOB:21|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:33.33: [VAL|JOB:22|PID:131785] Started. (#22/26) sampling job. Load & sample from subject of index (in user's list): 25
2020-10-29 7:17:33.33: [VAL|JOB:22|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1084/ISPY1_1084_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:34.08: [VAL|JOB:22|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:35.53: [VAL|JOB:22|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:35.53: [VAL|JOB:22|PID:131785] TIMING: [Load: 0.8] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:35.53: [VAL|JOB:23|PID:131785] Started. (#23/26) sampling job. Load & sample from subject of index (in user's list): 9
2020-10-29 7:17:35.54: [VAL|JOB:23|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1181/ISPY1_1181_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:35.90: [VAL|JOB:23|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:37.01: [VAL|JOB:23|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:37.01: [VAL|JOB:23|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:37.01: [VAL|JOB:24|PID:131785] Started. (#24/26) sampling job. Load & sample from subject of index (in user's list): 22
2020-10-29 7:17:37.01: [VAL|JOB:24|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1005/ISPY1_1005_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:37.50: [VAL|JOB:24|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:38.50: [VAL|JOB:24|PID:131785] Done. Samples per category: [Uniform: 192/192] 
2020-10-29 7:17:38.50: [VAL|JOB:24|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:38.50: [VAL|JOB:25|PID:131785] Started. (#25/26) sampling job. Load & sample from subject of index (in user's list): 0
2020-10-29 7:17:38.50: [VAL|JOB:25|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1148/ISPY1_1148_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:38.86: [VAL|JOB:25|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:39.97: [VAL|JOB:25|PID:131785] Done. Samples per category: [Uniform: 193/193] 
2020-10-29 7:17:39.97: [VAL|JOB:25|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:39.98: [VAL|SAMPLER|PID:131785] TIMING: Sampling for next [Validation] lasted: 49.6 secs.
2020-10-29 7:17:39.98: [VAL|SAMPLER|PID:131785] :=:=:=:=:=:= Finished sampling for next [Validation] =:=:=:=:=:=:
2020-10-29 7:17:40.37: [MAIN|PID:131785] MULTIPROC: Before Validation in subepoch #0, submitting sampling job for next [TRAINING].
2020-10-29 7:17:40.37: V-V-V-V- Validating for subepoch before starting training iterations -V-V-V-V
2020-10-29 7:17:40.37: [TRA|SAMPLER|PID:131785] :=:=:=:=:=:=: Starting to sample for next [Training]... :=:=:=:=:=:=:
2020-10-29 7:17:40.37: [VALIDATION] Validated on 1/100 batches for this subepoch...
2020-10-29 7:17:40.37: [TRA|SAMPLER|PID:131785] Out of [103] subjects given for [Training], we will sample from maximum [50] per subepoch.
2020-10-29 7:17:40.37: [TRA|SAMPLER|PID:131785] Shuffled indices of subjects that were randomly chosen: [55, 95, 37, 5, 40, 8, 77, 42, 65, 27, 62, 36, 92, 82, 12, 45, 97, 98, 63, 96, 79, 0, 19, 61, 24, 15, 25, 51, 102, 85, 57, 68, 17, 84, 4, 26, 28, 11, 80, 38, 18, 54, 49, 94, 9, 90, 72, 67, 44, 69]
2020-10-29 7:17:40.37: [TRA|SAMPLER|PID:131785] Will sample from [50] subjects for next Training...
2020-10-29 7:17:40.37: [TRA|JOB:0|PID:131785] Started. (#0/50) sampling job. Load & sample from subject of index (in user's list): 55
2020-10-29 7:17:40.37: [TRA|JOB:0|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1188/ISPY1_1188_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:40.91: [TRA|JOB:0|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:41.69: 

 ERROR: Caught exception in do_training(): input and filter must have the same depth
	 [[node net/Conv3D_49 (defined at /cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py:189)  = Conv3D[T=DT_FLOAT, data_format="NDHWC", dilations=[1, 1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](net/transpose_115, net/transpose_112)]]
	 [[{{node Sum_12/_851}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2945_Sum_12", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'net/Conv3D_49', defined at:
  File "/cbica/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/bin/deepMedicRun", line 178, in <module>
    session.run_session(sess_device, model_params, args.reset_trainer)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/frontEnd/trainSession.py", line 111, in run_session
    cnn3d.make_cnn_model(*model_params.get_args_for_arch())
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/cnn3d.py", line 452, in make_cnn_model
    indicesOfLayersToConnectResidualsInOutput[thisPathwayType]
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/pathways.py", line 210, in makeLayersOfThisPathwayAndReturnDimensionsOfOutputFM
    dropoutRate=thisLayerDropoutRate
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/layers.py", line 302, in makeLayer
    inputToConvShapeTrain, inputToConvShapeVal, inputToConvShapeTest)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/layers.py", line 248, in _createWeightsTensorAndConvolve
    tupleWithOuputAndShapeTrValTest = convolveWithGivenWeightMatrix(self._W, filterShape, inputToConvTrain, inputToConvVal, inputToConvTest, inputToConvShapeTrain, inputToConvShapeVal, inputToConvShapeTest)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py", line 189, in convolveWithGivenWeightMatrix
    data_format = "NDHWC"
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1362, in conv3d
    name=name)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): input and filter must have the same depth
	 [[node net/Conv3D_49 (defined at /cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py:189)  = Conv3D[T=DT_FLOAT, data_format="NDHWC", dilations=[1, 1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](net/transpose_115, net/transpose_112)]]
	 [[{{node Sum_12/_851}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2945_Sum_12", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]


2020-10-29 7:17:41.69: Traceback (most recent call last):
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth
	 [[{{node net/Conv3D_49}} = Conv3D[T=DT_FLOAT, data_format="NDHWC", dilations=[1, 1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](net/transpose_115, net/transpose_112)]]
	 [[{{node Sum_12/_851}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2945_Sum_12", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/routines/training.py", line 304, in do_training
    lbls_samples_per_path_val)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/routines/training.py", line 90, in process_in_batches
    results_of_run = sessionTf.run(fetches=list_of_ops, feed_dict=feeds_dict)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth
	 [[node net/Conv3D_49 (defined at /cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py:189)  = Conv3D[T=DT_FLOAT, data_format="NDHWC", dilations=[1, 1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](net/transpose_115, net/transpose_112)]]
	 [[{{node Sum_12/_851}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2945_Sum_12", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'net/Conv3D_49', defined at:
  File "/cbica/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/bin/deepMedicRun", line 178, in <module>
    session.run_session(sess_device, model_params, args.reset_trainer)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/frontEnd/trainSession.py", line 111, in run_session
    cnn3d.make_cnn_model(*model_params.get_args_for_arch())
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/cnn3d.py", line 452, in make_cnn_model
    indicesOfLayersToConnectResidualsInOutput[thisPathwayType]
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/pathways.py", line 210, in makeLayersOfThisPathwayAndReturnDimensionsOfOutputFM
    dropoutRate=thisLayerDropoutRate
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/layers.py", line 302, in makeLayer
    inputToConvShapeTrain, inputToConvShapeVal, inputToConvShapeTest)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/layers.py", line 248, in _createWeightsTensorAndConvolve
    tupleWithOuputAndShapeTrValTest = convolveWithGivenWeightMatrix(self._W, filterShape, inputToConvTrain, inputToConvVal, inputToConvTest, inputToConvShapeTrain, inputToConvShapeVal, inputToConvShapeTest)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py", line 189, in convolveWithGivenWeightMatrix
    data_format = "NDHWC"
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1362, in conv3d
    name=name)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): input and filter must have the same depth
	 [[node net/Conv3D_49 (defined at /cbica/software/external/python/anaconda/3/envs/DeepMedic_2019-11-14_v0.8.0/lib/python3.6/site-packages/deepmedic/neuralnet/ops.py:189)  = Conv3D[T=DT_FLOAT, data_format="NDHWC", dilations=[1, 1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](net/transpose_115, net/transpose_112)]]
	 [[{{node Sum_12/_851}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2945_Sum_12", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]


2020-10-29 7:17:41.69: Terminating worker pool.
2020-10-29 7:17:42.44: [TRA|JOB:0|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:42.44: [TRA|JOB:0|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:42.44: [TRA|JOB:1|PID:131785] Started. (#1/50) sampling job. Load & sample from subject of index (in user's list): 95
2020-10-29 7:17:42.44: [TRA|JOB:1|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1096/ISPY1_1096_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:42.82: [TRA|JOB:1|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:43.63: [TRA|JOB:1|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:43.63: [TRA|JOB:1|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:43.63: [TRA|JOB:2|PID:131785] Started. (#2/50) sampling job. Load & sample from subject of index (in user's list): 37
2020-10-29 7:17:43.64: [TRA|JOB:2|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1137/ISPY1_1137_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:44.06: [TRA|JOB:2|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:45.52: [TRA|JOB:2|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:45.52: [TRA|JOB:2|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:45.52: [TRA|JOB:3|PID:131785] Started. (#3/50) sampling job. Load & sample from subject of index (in user's list): 5
2020-10-29 7:17:45.52: [TRA|JOB:3|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1003/ISPY1_1003_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:45.87: [TRA|JOB:3|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:47.11: [TRA|JOB:3|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:47.11: [TRA|JOB:3|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:47.11: [TRA|JOB:4|PID:131785] Started. (#4/50) sampling job. Load & sample from subject of index (in user's list): 40
2020-10-29 7:17:47.11: [TRA|JOB:4|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1094/ISPY1_1094_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:47.49: [TRA|JOB:4|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:48.72: [TRA|JOB:4|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:48.72: [TRA|JOB:4|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:48.72: [TRA|JOB:5|PID:131785] Started. (#5/50) sampling job. Load & sample from subject of index (in user's list): 8
2020-10-29 7:17:48.72: [TRA|JOB:5|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1227/ISPY1_1227_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:49.29: [TRA|JOB:5|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:51.90: [TRA|JOB:5|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:51.90: [TRA|JOB:5|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.6] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:51.90: [TRA|JOB:6|PID:131785] Started. (#6/50) sampling job. Load & sample from subject of index (in user's list): 77
2020-10-29 7:17:51.90: [TRA|JOB:6|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1017/ISPY1_1017_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:52.44: [TRA|JOB:6|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:53.67: [TRA|JOB:6|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:53.67: [TRA|JOB:6|PID:131785] TIMING: [Load: 0.6] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:53.67: [TRA|JOB:7|PID:131785] Started. (#7/50) sampling job. Load & sample from subject of index (in user's list): 42
2020-10-29 7:17:53.67: [TRA|JOB:7|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1013/ISPY1_1013_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:54.25: [TRA|JOB:7|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:55.46: [TRA|JOB:7|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:55.46: [TRA|JOB:7|PID:131785] TIMING: [Load: 0.6] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:55.46: [TRA|JOB:8|PID:131785] Started. (#8/50) sampling job. Load & sample from subject of index (in user's list): 65
2020-10-29 7:17:55.46: [TRA|JOB:8|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1196/ISPY1_1196_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:55.90: [TRA|JOB:8|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:17:57.28: [TRA|JOB:8|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:17:57.28: [TRA|JOB:8|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:17:57.28: [TRA|JOB:9|PID:131785] Started. (#9/50) sampling job. Load & sample from subject of index (in user's list): 27
2020-10-29 7:17:57.28: [TRA|JOB:9|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1220/ISPY1_1220_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:17:58.04: [TRA|JOB:9|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:1.43: [TRA|JOB:9|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:1.43: [TRA|JOB:9|PID:131785] TIMING: [Load: 0.8] [Preproc: 2.3] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:1.43: [TRA|JOB:10|PID:131785] Started. (#10/50) sampling job. Load & sample from subject of index (in user's list): 62
2020-10-29 7:18:1.43: [TRA|JOB:10|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1218/ISPY1_1218_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:2.10: [TRA|JOB:10|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:3.62: [TRA|JOB:10|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:3.62: [TRA|JOB:10|PID:131785] TIMING: [Load: 0.7] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:3.62: [TRA|JOB:11|PID:131785] Started. (#11/50) sampling job. Load & sample from subject of index (in user's list): 36
2020-10-29 7:18:3.62: [TRA|JOB:11|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1107/ISPY1_1107_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:3.95: [TRA|JOB:11|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:5.12: [TRA|JOB:11|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:5.12: [TRA|JOB:11|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:5.12: [TRA|JOB:12|PID:131785] Started. (#12/50) sampling job. Load & sample from subject of index (in user's list): 92
2020-10-29 7:18:5.12: [TRA|JOB:12|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1004/ISPY1_1004_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:5.39: [TRA|JOB:12|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:6.18: [TRA|JOB:12|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:6.18: [TRA|JOB:12|PID:131785] TIMING: [Load: 0.3] [Preproc: 0.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:6.18: [TRA|JOB:13|PID:131785] Started. (#13/50) sampling job. Load & sample from subject of index (in user's list): 82
2020-10-29 7:18:6.18: [TRA|JOB:13|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1056/ISPY1_1056_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:6.57: [TRA|JOB:13|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:7.92: [TRA|JOB:13|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:7.92: [TRA|JOB:13|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:7.92: [TRA|JOB:14|PID:131785] Started. (#14/50) sampling job. Load & sample from subject of index (in user's list): 12
2020-10-29 7:18:7.92: [TRA|JOB:14|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1197/ISPY1_1197_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:8.92: [TRA|JOB:14|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:12.41: [TRA|JOB:14|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:12.41: [TRA|JOB:14|PID:131785] TIMING: [Load: 1.1] [Preproc: 2.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:12.41: [TRA|JOB:15|PID:131785] Started. (#15/50) sampling job. Load & sample from subject of index (in user's list): 45
2020-10-29 7:18:12.41: [TRA|JOB:15|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1184/ISPY1_1184_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:12.84: [TRA|JOB:15|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:13.95: [TRA|JOB:15|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:13.95: [TRA|JOB:15|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.7] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:13.95: [TRA|JOB:16|PID:131785] Started. (#16/50) sampling job. Load & sample from subject of index (in user's list): 97
2020-10-29 7:18:13.95: [TRA|JOB:16|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1205/ISPY1_1205_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:14.42: [TRA|JOB:16|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:15.54: [TRA|JOB:16|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:15.54: [TRA|JOB:16|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.7] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:15.54: [TRA|JOB:17|PID:131785] Started. (#17/50) sampling job. Load & sample from subject of index (in user's list): 98
2020-10-29 7:18:15.54: [TRA|JOB:17|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1191/ISPY1_1191_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:15.86: [TRA|JOB:17|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:17.04: [TRA|JOB:17|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:17.04: [TRA|JOB:17|PID:131785] TIMING: [Load: 0.3] [Preproc: 0.7] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:17.04: [TRA|JOB:18|PID:131785] Started. (#18/50) sampling job. Load & sample from subject of index (in user's list): 63
2020-10-29 7:18:17.04: [TRA|JOB:18|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1207/ISPY1_1207_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:17.80: [TRA|JOB:18|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:21.14: [TRA|JOB:18|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:21.14: [TRA|JOB:18|PID:131785] TIMING: [Load: 0.9] [Preproc: 2.2] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:21.14: [TRA|JOB:19|PID:131785] Started. (#19/50) sampling job. Load & sample from subject of index (in user's list): 96
2020-10-29 7:18:21.14: [TRA|JOB:19|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1010/ISPY1_1010_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:21.56: [TRA|JOB:19|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:23.36: [TRA|JOB:19|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:23.36: [TRA|JOB:19|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.2] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:23.36: [TRA|JOB:20|PID:131785] Started. (#20/50) sampling job. Load & sample from subject of index (in user's list): 79
2020-10-29 7:18:23.36: [TRA|JOB:20|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1073/ISPY1_1073_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:23.73: [TRA|JOB:20|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:25.07: [TRA|JOB:20|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:25.07: [TRA|JOB:20|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:25.07: [TRA|JOB:21|PID:131785] Started. (#21/50) sampling job. Load & sample from subject of index (in user's list): 0
2020-10-29 7:18:25.07: [TRA|JOB:21|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1127/ISPY1_1127_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:25.43: [TRA|JOB:21|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:26.87: [TRA|JOB:21|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:26.87: [TRA|JOB:21|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:26.87: [TRA|JOB:22|PID:131785] Started. (#22/50) sampling job. Load & sample from subject of index (in user's list): 19
2020-10-29 7:18:26.87: [TRA|JOB:22|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1090/ISPY1_1090_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:27.30: [TRA|JOB:22|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:28.92: [TRA|JOB:22|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:28.92: [TRA|JOB:22|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:28.92: [TRA|JOB:23|PID:131785] Started. (#23/50) sampling job. Load & sample from subject of index (in user's list): 61
2020-10-29 7:18:28.92: [TRA|JOB:23|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1009/ISPY1_1009_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:29.25: [TRA|JOB:23|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:30.72: [TRA|JOB:23|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:30.72: [TRA|JOB:23|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:30.72: [TRA|JOB:24|PID:131785] Started. (#24/50) sampling job. Load & sample from subject of index (in user's list): 24
2020-10-29 7:18:30.72: [TRA|JOB:24|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1212/ISPY1_1212_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:31.14: [TRA|JOB:24|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:32.42: [TRA|JOB:24|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:32.42: [TRA|JOB:24|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:32.42: [TRA|JOB:25|PID:131785] Started. (#25/50) sampling job. Load & sample from subject of index (in user's list): 15
2020-10-29 7:18:32.42: [TRA|JOB:25|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1175/ISPY1_1175_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:32.89: [TRA|JOB:25|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:34.44: [TRA|JOB:25|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:34.44: [TRA|JOB:25|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:34.44: [TRA|JOB:26|PID:131785] Started. (#26/50) sampling job. Load & sample from subject of index (in user's list): 25
2020-10-29 7:18:34.44: [TRA|JOB:26|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1189/ISPY1_1189_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:34.99: [TRA|JOB:26|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:37.53: [TRA|JOB:26|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:37.53: [TRA|JOB:26|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:37.53: [TRA|JOB:27|PID:131785] Started. (#27/50) sampling job. Load & sample from subject of index (in user's list): 51
2020-10-29 7:18:37.53: [TRA|JOB:27|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1088/ISPY1_1088_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:38.05: [TRA|JOB:27|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:39.53: [TRA|JOB:27|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:39.53: [TRA|JOB:27|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:39.53: [TRA|JOB:28|PID:131785] Started. (#28/50) sampling job. Load & sample from subject of index (in user's list): 102
2020-10-29 7:18:39.53: [TRA|JOB:28|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1051/ISPY1_1051_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:40.10: [TRA|JOB:28|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:41.62: [TRA|JOB:28|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:41.62: [TRA|JOB:28|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:41.62: [TRA|JOB:29|PID:131785] Started. (#29/50) sampling job. Load & sample from subject of index (in user's list): 85
2020-10-29 7:18:41.62: [TRA|JOB:29|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1106/ISPY1_1106_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:42.23: [TRA|JOB:29|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:43.56: [TRA|JOB:29|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:43.56: [TRA|JOB:29|PID:131785] TIMING: [Load: 0.6] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:43.56: [TRA|JOB:30|PID:131785] Started. (#30/50) sampling job. Load & sample from subject of index (in user's list): 57
2020-10-29 7:18:43.56: [TRA|JOB:30|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1128/ISPY1_1128_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:43.93: [TRA|JOB:30|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:45.40: [TRA|JOB:30|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:45.40: [TRA|JOB:30|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:45.40: [TRA|JOB:31|PID:131785] Started. (#31/50) sampling job. Load & sample from subject of index (in user's list): 68
2020-10-29 7:18:45.40: [TRA|JOB:31|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1112/ISPY1_1112_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:45.76: [TRA|JOB:31|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:47.14: [TRA|JOB:31|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:47.14: [TRA|JOB:31|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:47.14: [TRA|JOB:32|PID:131785] Started. (#32/50) sampling job. Load & sample from subject of index (in user's list): 17
2020-10-29 7:18:47.14: [TRA|JOB:32|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1152/ISPY1_1152_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:47.73: [TRA|JOB:32|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:50.58: [TRA|JOB:32|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:50.58: [TRA|JOB:32|PID:131785] TIMING: [Load: 0.7] [Preproc: 1.7] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:50.58: [TRA|JOB:33|PID:131785] Started. (#33/50) sampling job. Load & sample from subject of index (in user's list): 84
2020-10-29 7:18:50.58: [TRA|JOB:33|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1173/ISPY1_1173_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:51.09: [TRA|JOB:33|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:52.62: [TRA|JOB:33|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:52.62: [TRA|JOB:33|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:52.62: [TRA|JOB:34|PID:131785] Started. (#34/50) sampling job. Load & sample from subject of index (in user's list): 4
2020-10-29 7:18:52.62: [TRA|JOB:34|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1083/ISPY1_1083_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:52.99: [TRA|JOB:34|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:54.38: [TRA|JOB:34|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:54.38: [TRA|JOB:34|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:54.38: [TRA|JOB:35|PID:131785] Started. (#35/50) sampling job. Load & sample from subject of index (in user's list): 26
2020-10-29 7:18:54.38: [TRA|JOB:35|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1038/ISPY1_1038_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:54.94: [TRA|JOB:35|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:18:57.08: [TRA|JOB:35|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:18:57.08: [TRA|JOB:35|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.4] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:18:57.08: [TRA|JOB:36|PID:131785] Started. (#36/50) sampling job. Load & sample from subject of index (in user's list): 28
2020-10-29 7:18:57.08: [TRA|JOB:36|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1167/ISPY1_1167_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:18:58.24: [TRA|JOB:36|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:2.06: [TRA|JOB:36|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:2.06: [TRA|JOB:36|PID:131785] TIMING: [Load: 1.3] [Preproc: 2.5] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:2.06: [TRA|JOB:37|PID:131785] Started. (#37/50) sampling job. Load & sample from subject of index (in user's list): 11
2020-10-29 7:19:2.06: [TRA|JOB:37|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1008/ISPY1_1008_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:2.42: [TRA|JOB:37|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:3.87: [TRA|JOB:37|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:3.87: [TRA|JOB:37|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:3.87: [TRA|JOB:38|PID:131785] Started. (#38/50) sampling job. Load & sample from subject of index (in user's list): 80
2020-10-29 7:19:3.87: [TRA|JOB:38|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1150/ISPY1_1150_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:4.31: [TRA|JOB:38|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:5.91: [TRA|JOB:38|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:5.91: [TRA|JOB:38|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.1] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:5.91: [TRA|JOB:39|PID:131785] Started. (#39/50) sampling job. Load & sample from subject of index (in user's list): 38
2020-10-29 7:19:5.91: [TRA|JOB:39|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1165/ISPY1_1165_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:6.35: [TRA|JOB:39|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:7.67: [TRA|JOB:39|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:7.67: [TRA|JOB:39|PID:131785] TIMING: [Load: 0.5] [Preproc: 0.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:7.67: [TRA|JOB:40|PID:131785] Started. (#40/50) sampling job. Load & sample from subject of index (in user's list): 18
2020-10-29 7:19:7.67: [TRA|JOB:40|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1016/ISPY1_1016_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:7.92: [TRA|JOB:40|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:8.80: [TRA|JOB:40|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:8.80: [TRA|JOB:40|PID:131785] TIMING: [Load: 0.3] [Preproc: 0.6] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:8.80: [TRA|JOB:41|PID:131785] Started. (#41/50) sampling job. Load & sample from subject of index (in user's list): 54
2020-10-29 7:19:8.80: [TRA|JOB:41|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1179/ISPY1_1179_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:9.24: [TRA|JOB:41|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:10.82: [TRA|JOB:41|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:10.82: [TRA|JOB:41|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:10.82: [TRA|JOB:42|PID:131785] Started. (#42/50) sampling job. Load & sample from subject of index (in user's list): 49
2020-10-29 7:19:10.82: [TRA|JOB:42|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1168/ISPY1_1168_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:11.38: [TRA|JOB:42|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:13.82: [TRA|JOB:42|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:13.82: [TRA|JOB:42|PID:131785] TIMING: [Load: 0.6] [Preproc: 1.6] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:13.82: [TRA|JOB:43|PID:131785] Started. (#43/50) sampling job. Load & sample from subject of index (in user's list): 94
2020-10-29 7:19:13.82: [TRA|JOB:43|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1021/ISPY1_1021_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:14.32: [TRA|JOB:43|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:15.83: [TRA|JOB:43|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:15.83: [TRA|JOB:43|PID:131785] TIMING: [Load: 0.5] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:15.83: [TRA|JOB:44|PID:131785] Started. (#44/50) sampling job. Load & sample from subject of index (in user's list): 9
2020-10-29 7:19:15.83: [TRA|JOB:44|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1074/ISPY1_1074_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:16.20: [TRA|JOB:44|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:17.51: [TRA|JOB:44|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:17.51: [TRA|JOB:44|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:17.51: [TRA|JOB:45|PID:131785] Started. (#45/50) sampling job. Load & sample from subject of index (in user's list): 90
2020-10-29 7:19:17.51: [TRA|JOB:45|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1226/ISPY1_1226_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:18.25: [TRA|JOB:45|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:20.87: [TRA|JOB:45|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:20.87: [TRA|JOB:45|PID:131785] TIMING: [Load: 0.8] [Preproc: 1.8] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:20.87: [TRA|JOB:46|PID:131785] Started. (#46/50) sampling job. Load & sample from subject of index (in user's list): 72
2020-10-29 7:19:20.87: [TRA|JOB:46|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1058/ISPY1_1058_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:21.44: [TRA|JOB:46|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:22.83: [TRA|JOB:46|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:22.83: [TRA|JOB:46|PID:131785] TIMING: [Load: 0.6] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:22.83: [TRA|JOB:47|PID:131785] Started. (#47/50) sampling job. Load & sample from subject of index (in user's list): 67
2020-10-29 7:19:22.83: [TRA|JOB:47|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1163/ISPY1_1163_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:23.20: [TRA|JOB:47|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:24.57: [TRA|JOB:47|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:24.57: [TRA|JOB:47|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:24.57: [TRA|JOB:48|PID:131785] Started. (#48/50) sampling job. Load & sample from subject of index (in user's list): 44
2020-10-29 7:19:24.57: [TRA|JOB:48|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1123/ISPY1_1123_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:24.94: [TRA|JOB:48|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:26.30: [TRA|JOB:48|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:26.30: [TRA|JOB:48|PID:131785] TIMING: [Load: 0.4] [Preproc: 0.9] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:26.30: [TRA|JOB:49|PID:131785] Started. (#49/50) sampling job. Load & sample from subject of index (in user's list): 69
2020-10-29 7:19:26.30: [TRA|JOB:49|PID:131785] Loading subject with 1st channel at: /cbica/home/bhaleram/comp_space/ispy-seg/data_resampled_isotropic_norm/ISPY1_1087/ISPY1_1087_DCE_0000_N3_zscored.nii.gz
2020-10-29 7:19:26.69: [TRA|JOB:49|PID:131785] WARN: Loaded labels are dtype [float32]. Rounding and casting to [int16]!
2020-10-29 7:19:28.21: [TRA|JOB:49|PID:131785] Done. Samples per category: [Class-0: 10/10] [Class-1: 10/10] 
2020-10-29 7:19:28.21: [TRA|JOB:49|PID:131785] TIMING: [Load: 0.4] [Preproc: 1.0] [Augm-Img: 0.0] [Augm-Samples: 0.0] secs
2020-10-29 7:19:28.21: [TRA|SAMPLER|PID:131785] TIMING: Sampling for next [Training] lasted: 107.8 secs.
2020-10-29 7:19:28.21: [TRA|SAMPLER|PID:131785] :=:=:=:=:=:= Finished sampling for next [Training] =:=:=:=:=:=:
2020-10-29 7:19:28.81: 
=======================================================
2020-10-29 7:19:28.81: =========== Training session finished =================
2020-10-29 7:19:28.81: =======================================================
2020-10-29 7:19:28.81: Finished.