Sometimes FSL tools do not immediately find a satisfactory transformation to bring two images in register. In that case, you can give the flirt algorithm a little nudge by (1.) bringing the two images roughly in the same space using itk snap's automatic and manual registration tools and only then (2.) running FSL flirt or epi_reg.
Since itksnap and FSL use different origins for their coordinate systems the transformation matrices that are produced by the two programs are not compatible. Luckily, there is c3d_affine_tool which can be used to convert transformation files from itk to FSL and the other way round.
Proceed in two steps:
Use itk snap's automatic or manual registration tools to bring the images in rough register. Save the resulting transformation matrix. It is important here that one does not use the default file format (itk transform file) but instead chooses the Convert3D Transform File:
Save itkTrfFile.mat
Use the c3d_affine_tool (http://www.itksnap.org/pmwiki/pmwiki.php?n=Downloads.C3D) to convert the the transformation matrix from itk's RAS format to the format required by FSL.
c3d_affine_tool -ref ref.nii -src src.nii itkTrfFile.mat -ras2fsl -o FslTrfFile
The FslTrfFile that just has been created can now be used with FSL command line tools, such as ApplyXFM or ConcatXFM. Using ConcatXFM, for example, one can concantenate FslTrfFile with the trf files that were created while using epi_reg. One can subsequently transform files from source to reference in one go using the concatenated tranformation matrix with ApplyXFM.
Thanks for your post. I was able to convert an ITK SNAP transformation *.txt file to a FSL *.mat file using a slightly different command:
c3d_affine_tool -ref ref.nii -src src.nii -itk itk_transfomation_text_file.txt -ras2fsl -o fsl_transformation_file.mat
I'm using ITK-SNAP 3.8.0