HowtoCUB.md

Prerequisite: gensim.

Two scripts under src/ that you need to look at:

1. generate_embeddings.py: Creates dataloaders for embedded sentences using fasttext model trained on CUB dictionary. The train_loader and test_loader will return dataB of length 2: dataB[0]: [batch_size, sentence_length, embedding_vector_size] dataB[1]: [batch_size], original sentence length before truncation or padding (you can probably ignore this one, but I kept it there just in case you need the original length to truncate the sentence when calculating correlations)

2. coherence.py: this one is pretty much ready to go, it is defaulted to load the trained cub model under expeirments/ft_obj. You just need to import the CCA module (see usage in line 65, 66 and 77) that can be called with:

coherence = CCA(images, sentences)

Where images is of shape [batch_size, feature_size] and sentences [batch_size, sentence_length, embedding_vector_size].

The image features generated with resnet101 are data/cub/resnet101_2048/train.ft and data/cub/resnet101_2048/test.ft, their corresponding raw data (arranged in the same order) are in data/cub/resnet101_2048/train.data and data/cub/resnet101_2048/test.data.
These can be loaded with torch.load command.

The image features generated with resnet101 are data/cub/resnet101_2048/train.ft and data/cub/resnet101_2048/test.ft, their corresponding raw data (arranged in the same order) are in data/cub/resnet101_2048/train.data and data/cub/resnet101_2048/test.data.
These can be loaded with torch.load command.