Sometimes FSL tools do not immediately find a satisfactory transformation to bring two images in register. In that case, you can give the flirt algorithm a little nudge by (1.) bringing the two images roughly in the same space using itk snap's automatic and manual registration tools and only then (2.) running FSL flirt or epi_reg.
Since itksnap and FSL use different origins for their coordinate systems the transformation matrices that are produced by the two programs are not compatible. Luckily, there is c3d_affine_tool which can be used to convert transformation files from itk to FSL and the other way round.
Proceed in two steps:
Use itk snap's automatic or manual registration tools to bring the images in rough register. Save the resulting transformation matrix. It is important here that one does not use the default file format (itk transform file) but instead chooses the Convert3D Transform File:
Save itkTrfFile.mat
Use the c3d_affine_tool (http://www.itksnap.org/pmwiki/pmwiki.php?n=Downloads.C3D) to convert the the transformation matrix from itk's RAS format to the format required by FSL.
c3d_affine_tool -ref ref.nii -src src.nii itkTrfFile.mat -ras2fsl -o FslTrfFile
The FslTrfFile that just has been created can now be used with FSL command line tools, such as ApplyXFM or ConcatXFM. Using ConcatXFM, for example, one can concantenate FslTrfFile with the trf files that were created while using epi_reg. One can subsequently transform files from source to reference in one go using the concatenated tranformation matrix with ApplyXFM.
not sure if you already found the solution,
you can do
c3d_affine_tool -ref ref.nii -src src.nii fsl_transfomation.mat -fsl2ras -oitk itk_transformation_file.txt
so, similar to @sufkes's comment above for the inverse conversion, when you give the itk file as input or output you have to specify it's an itk file (via -itk or -oitk respectively).
Hope that helps!