Last active
April 4, 2018 19:03
-
-
Save JossWhittle/94dc4d566b50159c0c8fa630454c901c to your computer and use it in GitHub Desktop.
Cleaned up versions of Tensorflow's default weight initialization schemes, extracted from deep within the Tensorflow source code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| He_gain = 2.0 | |
| He_n = (fan_in * (kernel_size ** 2)) | |
| # Most stable performance in my experience. | |
| He_c_trunc_norm = np.sqrt(1.3 * He_gain / He_n) | |
| He_normal_init = tf.initializers.truncated_normal(stddev=He_c_trunc_norm) | |
| He_c_uniform = np.sqrt(3.0 * He_gain / He_n) | |
| He_uniform_init = tf.initializers.random_uniform(minval=-He_c_uniform, maxval=He_c_uniform) | |
| Xavier_gain = 1.0 | |
| Xavier_n = ((fan_in * (kernel_size ** 2)) + (fan_out * (kernel_size ** 2))) / 2.0 | |
| Xavier_c_trunc_norm = np.sqrt(1.3 * Xavier_gain / Xavier_n) | |
| Xavier_normal_init = tf.initializers.truncated_normal(stddev=Xavier_c_trunc_norm) | |
| # Tensorflow's default initialization method. | |
| Xavier_c_uniform = np.sqrt(3.0 * Xavier_gain / Xavier_n) | |
| Xavier_uniform_init = tf.initializers.random_uniform(minval=-Xavier_c_uniform, maxval=Xavier_c_uniform) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment