Adam Golinski talesa

I am far from certain about the cause of the issue.

The symptoms are as follows (I haven't gathered extensive evidence yet, because I'm just setting up and testing sacred for the first time):

the start of the experiment is always logged
sometimes some heartbeats are sent at the beginning (because the stdout capture and metrics are showing up)
then they stop (stdout capture and metrics are not updated, and the experiment has 'Probably dead' status in sacredboard)
sometimes they show up again for a while (tens of minutes later, much more than what I would expect to see as reasonable delay)
and then they usually disappear again
when the experiment completes the appropriate event is sent to the database (and experiment is marked as completed in sacredboard), but the script freezes after the final message INFO - experiment_name - Completed after 0:14:29 and takes ages to finish (same as #273)
if I let the frozen script finish the output capture and metrics are collected

SSH agent forwarding and screen

When connecting to a remote server via SSH it is often convenient to use SSH agent forwarding so that you don't need a separate keypair on that server for connecting to further servers.

This is enabled by adding the

ForwardAgent yes

option to any of your Host entries in ~/.ssh/config (or alternatively with the -A option). Don't set this option in a wildcard Host * section since any user on the remote server that can bypass file permissions can now als use keys loaded in your SSH agent. So only use this with hosts you trust.

	import torch
	import torch.distributions as dist

	# Generates a sample from the generative model
	def gmm_generate_data(K, N, batch_size=10,
	upsilon=torch.Tensor([2.]).to('cpu'),
	mu_0=torch.Tensor([0., 0.]).to('cuda'),
	sigma2_0=torch.Tensor([2., 2.]).to('cuda')):

	# Sample parameters to distribution over mixture components

	"Dynamic plotting in matplotlib. Copy and paste into a Jupyter notebook."
	# written October 2016 by Sam Greydanus
	%matplotlib notebook
	import matplotlib.pyplot as plt
	import numpy as np
	import time

	def plt_dynamic(x, y, ax, colors=['b']):
	for color in colors:
	ax.plot(x, y, color)

	import seaborn as sns
	import pandas as pd

	data = pd.DataFrame(columns=["tolerance", "baseline", "run_no", "value"])

	for baseline_true, table in zip([True, False], [baseline, no_baseline]):
	for tol, row in zip(x, table):
	for run_no, val in zip(range(len(row)), row):
	data.loc[len(data)] = [tol, baseline_true, run_no, val]

	import torch
	from torch import tensor
	import torch.distributions as td
	import torch.optim as optim
	import matplotlib.pyplot as plt
	from torch import isinf, isnan
	from tqdm import tqdm as tqdm
	import numpy as np
	import scipy.stats

	import matplotlib
	import matplotlib.pyplot as plt
	import numpy as np
	import scipy.stats as stats

	import scipy.special as special

	import scipy

	plt.rcParams["figure.figsize"] = (16,10)

	import asyncio
	import ccxt.async as ccxt
	from collections import defaultdict

	exchanges = list()
	exchanges.append(('binance', apiKey, secret))
	exchanges.append(('livecoin', apiKey, secret))

	tasks = list()
	tasks += [ccxt.coinmarketcap({'timeout':100000}).fetch_tickers(params={'limit':0})]

	# you have to execute this code line by line so use jupyter notebook or hydrogen inside atom editor

	# import libraries
	import ccxt
	from datetime import datetime

	# create exchange API handle
	exchange = getattr(ccxt, 'binance')()
	# paste in your API key and secret here (if you're afraid they're gonna get stolen, inspect the ccxt library open source code on github)
	exchange.apiKey = ''