Here are some tanh units being self-normalising

self.py

# See "Self-Normalizing Neural Networks" https://arxiv.org/abs/1706.02515
# "SNNs cannot be derived with...tanh units..."
# So I'm probably missing the point somewhere...

import math
import numpy

# Magic number
lambda0 = 1.59254

n = 1000
nlayers = 100

# Incoming activiations have mean 0, variance 1
x = numpy.random.normal(0, 1, n)

# Applying 100 fully connected random layers of 1000 units each
for i in xrange(nlayers):
    w = numpy.random.normal(0, 1.0/math.sqrt(n), (n, n))
    x = lambda0*numpy.tanh(w.dot(x))

# Mean and variance remain around 0, 1
print numpy.mean(x), numpy.var(x)

Try things like x = numpy.random.normal(-0.2, 1.2, n)

You still end up with mean, variance around 0, 1

Note that you can push the lambda0 into the weights so that everyone who's been using tanh has been using self-normalising units all along. They just weren't initialising the weights appropriately :-)