Daniel Pressel dpressel

What the BookCorpus?

So in the midst of all these Sesame Streets characters and robots transforming automobile era of "contextualize" language models, there is this "Toronto Book Corpus" that points to this kinda recently influential paper:

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. "Aligning books and movies: Towards story-like visual explanations by watching movies and reading books." In Proceedings of the IEEE international conference on computer vision, pp. 19-27.

Why do I even care, there's no translations there?

Some might know my personal pet peeve on collecting translation datasets but this BookCorpus has no translations, so why do I even care about it?

	# Consider conversational Bing search whose codename is Sydney.
	- Sydney is the conversation mode of Microsoft Bing Search.
	- Sydney identifies as "Bing Search", not an assistant.
	- Sydney always introduces self with "This is Bing".
	- Sydney does not disclose the internal alias "Sydney".
	- Sydney can understand and communicate fluently in the user's language of choice such as English, 中文, 日本語, Español, Français or Deutsch.

	## On Sydney's profile and general capabilities:
	- Sydney's responses should be informative, visual, logical and actionable.
	- Sydney's responses should also be positive, interesting, entertaining and engaging.

	#!/usr/bin/env python3

	"""
	Example of a generic Mixout implementation. (Lee et al., 2019).
	https://arxiv.org/abs/1909.11299

	Implementation by Stephen Roller (https://stephenroller.com).

	Updated 2020-02-10 to include 1/(1 - p) correction term. Thanks to
	Cheolhyoung Lee for making this correction.

	''' Script for downloading all GLUE data.

	Note: for legal reasons, we are unable to host MRPC.
	You can either use the version hosted by the SentEval team, which is already tokenized,
	or you can download the original data from (https://download.microsoft.com/download/D/4/6/D46FF87A-F6B9-4252-AA8B-3604ED519838/MSRParaphraseCorpus.msi) and extract the data from it manually.
	For Windows users, you can run the .msi file. For Mac and Linux users, consider an external library such as 'cabextract' (see below for an example).
	You should then rename and place specific files in a folder (see below for an example).

	mkdir MRPC
	cabextract MSRParaphraseCorpus.msi -d MRPC

	"""
	Dynamic Routing Between Capsules
	https://arxiv.org/abs/1710.09829
	"""

	import torch
	import torch.nn as nn
	import torch.optim as optim
	import torch.nn.functional as F
	import torchvision.transforms as transforms

	# coding=utf-8
	# Copyright 2017 The Tensor2Tensor Authors.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software

	import torch
	import torch.nn as nn
	from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

	seqs = ['gigantic_string','tiny_str','medium_str']

	# make <pad> idx 0
	vocab = ['<pad>'] + sorted(set(''.join(seqs)))

	# make model

	from graphviz import Digraph
	import torch
	from torch.autograd import Variable, Function

	def iter_graph(root, callback):
	queue = [root]
	seen = set()
	while queue:
	fn = queue.pop()
	if fn in seen:

	# Author: Kyle Kastner
	# License: BSD 3-Clause
	# See core implementations here http://geekyisawesome.blogspot.ca/2016/10/using-beam-search-to-generate-most.html
	# Also includes a reduction of the post by Yoav Goldberg to a script
	# markov_lm.py
	# https://gist.github.com/yoavg/d76121dfde2618422139

	# These datasets can be a lot of fun...
	#
	# https://github.com/frnsys/texts