zomux’s gists

zomux / wmt14_en.cfg.top3.freq

Created January 2, 2020 11:09

	1796288 (ROOT (S NP VP .))
	232718 (ROOT (S PP , NP VP .))
	136106 (ROOT (S S , CC S .))
	112669 (ROOT (S ADVP , NP VP .))
	105347 (ROOT (S NP ADVP VP .))
	101963 (ROOT (S VP .))
	98094 (ROOT (S SBAR , NP VP .))
	97453 (ROOT (S PP NP VP .))
	96065 (ROOT (S S , NP VP .))
	81013 (ROOT (S S CC S .))

zomux / tree2code_codebits-8_dtok-wmt14_limit_datapoints-100000_limit_tree_depth-2_without_source.mapping

Created January 2, 2020 10:45

This file has been truncated, but you can view the full file.

(S NP VP .)	-0.29 -0.10 -0.16 0.53 2.37 0.43 0.85 0.25 1.27 -0.93 0.15 0.43 0.05 0.86 0.68 -0.05 0.44 0.05 -0.17 -1.16 -0.77 -0.28 0.75 0.51 0.89 1.05 -0.04 0.04 -0.40 -0.55 0.32 -0.08 -0.20 0.21 -0.69 0.32 0.33 1.86 -0.09 0.29 0.60 -0.58 0.52 0.19 0.22 0.10 -1.63 0.25 -0.27 -0.76 -0.91 -0.93 0.80 -0.22 0.61 -0.26 0.85 0.48 0.94 -0.14 0.01 -0.32 -0.29 -0.61 -0.80 0.11 0.40 -0.72 -0.12 0.04 -0.64 0.08 0.06 0.37 -0.50 0.71 0.04 0.89 -0.03 -0.24 1.03 -0.33 -2.35 -0.12 -0.49 1.35 0.32 -0.17 0.06 -0.96 -0.22 -1.02 0.18 -0.62 0.34 0.03 -0.06 -0.00 -0.07 0.38 -0.28 -0.15 0.43 0.16 0.11 1.15 0.05 -0.63 0.33 0.41 0.14 0.10 -0.49 -0.66 0.51 -0.25 0.43 1.17 -1.11 -0.93 -0.06 -0.06 -0.21 0.92 -0.04 0.81 -0.85 -0.98 0.37 -0.58 -0.42 -0.04 -0.35 0.76 0.27 0.89 -0.16 0.22 0.85 -0.32 0.60 -1.16 0.05 0.12 0.04 -1.70 0.18 0.41 -0.37 -0.85 0.83 -0.25 -0.51 -1.28 -0.12 0.60 -0.40 -0.82 -0.08 0.10 -0.63 -0.06 0.36 -0.01 -0.16 -1.07 -0.36 0.38 -0.10 0.17 -0.69 -0.23 0.51 -0.44 0.00 -0.61 -0.15 -0.07 -0.24 -0.01 -1.11 0.33 1.06 -0.

zomux / tree2code_codebits-8_dtok-wmt14_limit_datapoints-100000_limit_tree_depth-2_without_source.mapping

Created January 2, 2020 07:58

This file has been truncated, but you can view the full file.

(S NP VP .)	-0.29 -0.10 -0.16 0.53 2.37 0.43 0.85 0.25 1.27 -0.93 0.15 0.43 0.05 0.86 0.68 -0.05 0.44 0.05 -0.17 -1.16 -0.77 -0.28 0.75 0.51 0.89 1.05 -0.04 0.04 -0.40 -0.55 0.32 -0.08 -0.20 0.21 -0.69 0.32 0.33 1.86 -0.09 0.29 0.60 -0.58 0.52 0.19 0.22 0.10 -1.63 0.25 -0.27 -0.76 -0.91 -0.93 0.80 -0.22 0.61 -0.26 0.85 0.48 0.94 -0.14 0.01 -0.32 -0.29 -0.61 -0.80 0.11 0.40 -0.72 -0.12 0.04 -0.64 0.08 0.06 0.37 -0.50 0.71 0.04 0.89 -0.03 -0.24 1.03 -0.33 -2.35 -0.12 -0.49 1.35 0.32 -0.17 0.06 -0.96 -0.22 -1.02 0.18 -0.62 0.34 0.03 -0.06 -0.00 -0.07 0.38 -0.28 -0.15 0.43 0.16 0.11 1.15 0.05 -0.63 0.33 0.41 0.14 0.10 -0.49 -0.66 0.51 -0.25 0.43 1.17 -1.11 -0.93 -0.06 -0.06 -0.21 0.92 -0.04 0.81 -0.85 -0.98 0.37 -0.58 -0.42 -0.04 -0.35 0.76 0.27 0.89 -0.16 0.22 0.85 -0.32 0.60 -1.16 0.05 0.12 0.04 -1.70 0.18 0.41 -0.37 -0.85 0.83 -0.25 -0.51 -1.28 -0.12 0.60 -0.40 -0.82 -0.08 0.10 -0.63 -0.06 0.36 -0.01 -0.16 -1.07 -0.36 0.38 -0.10 0.17 -0.69 -0.23 0.51 -0.44 0.00 -0.61 -0.15 -0.07 -0.24 -0.01 -1.11 0.33 1.06 -0.

zomux / all_syntoks.txt

Created December 25, 2019 01:14

	(ROOT_(NP_NP_PP_.))
	(ROOT_(S_S_CC_S_.))
	(ROOT_(S_NP_VP_.))
	(ROOT_(FRAG_FRAG_:_FRAG_.))
	(ROOT_(S_VP_.))
	(ROOT_(S_SBAR_,_S_:_:_S_.))
	(ROOT_(S_``_NP_VP_.))
	(ROOT_(S_ADVP_,_NP_VP_.))
	(ROOT_(S_S_:_:_CC_S_.))
	(ROOT_(S_S_,_CC_S_.))

zomux / take_top3_levels.py

Created December 19, 2019 06:24

	#!/usr/bin/env python
	# -- coding: utf-8 --

	from __future__ import absolute_import
	from __future__ import division
	from __future__ import print_function
	from nltk.tree import ParentedTree
	from argparse import ArgumentParser
	import sys, os
	import re

zomux / stat_withoutroot.txt

Created December 19, 2019 06:16

This file has been truncated, but you can view the full file.

	13184 (S NP (VP VBP (SBAR S)) .)
	10334 (S NP (VP VBD (SBAR S)) .)
	9655 (S NP (VP MD (VP VB VP)) .)
	9278 (S (VP VB NP) .)
	9083 (S NP (VP MD (VP VB NP)) .)
	8748 (S NP (VP VBP (S VP)) .)
	7665 (S NP (VP VBZ (NP NP PP)) .)
	6077 (S (VP VB S) .)
	5938 (S NP (VP VBD (S VP)) .)
	5897 (S NP (VP VBD (PP IN NP)) .)

zomux / stat_withroot.txt

Created December 19, 2019 06:15

	488085 (ROOT (S NP VP .))
	68446 (ROOT (S VP .))
	42516 (ROOT (S S , NP VP .))
	37409 (ROOT (S ADVP , NP VP .))
	33285 (ROOT (S S , CC S .))
	25008 (ROOT (S NP ADVP VP .))
	22790 (ROOT (S CC NP VP .))
	22512 (ROOT (SBARQ WHNP SQ .))
	19231 (ROOT (S SBAR , NP VP .))
	19058 (ROOT (S S : S .))

zomux / take_top3_levels.py

Created December 19, 2019 06:08

	#!/usr/bin/env python
	# -- coding: utf-8 --

	from __future__ import absolute_import
	from __future__ import division
	from __future__ import print_function
	from nltk.tree import ParentedTree
	import sys, os
	import re

zomux / codenmt_dtok-paramt_5bits_gpus-8.result.diversity

Created December 18, 2019 06:23

	0 -1.07 <c13> <eoc> article 97 product substitutability 1 .
	0 -1.30 <c8> <eoc> article 97 sugar production of the product 1 .
	0 -2.90 <c7> <eoc> article 97 sugar production volume 1 .
	0 -4.21 <c17> <eoc> article 97 market quality substitute 1 .
	0 -4.21 <c10> <eoc> article 97 the sugar production of the goods 1 .
	1 -0.48 <c22> <eoc> containers must be covered by plastic with holes , or foil to prevent the drying of the substrate and be used for 14 days .
	1 -2.74 <c6> <eoc> containers are covered with plastic tubing , dishes or foil to prevent the drying of the substrate and continue to test conditions for 14 days .
	1 -2.95 <c8> <eoc> the containers cover the plastic sheets , or foil to prevent the drying of the substrate and keep in the test conditions for 14 days .
	1 -3.61 <c3> <eoc> the containers are covered with plastic sheets , or foil to prevent the drying of the substrate and be kept in the test conditions for 14 days .
	1 -3.76 <c4> <eoc> containers shall be covered by plastic with holes , or foil to pr

zomux / detokenizer.perl

Created December 12, 2019 23:39

	#!/usr/bin/env perl

	# $Id: detokenizer.perl 4134 2011-08-08 15:30:54Z bgottesman $
	# Sample De-Tokenizer
	# written by Josh Schroeder, based on code by Philipp Koehn
	# further modifications by Ondrej Bojar
	#
	# This file is part of moses. Its use is licensed under the GNU Lesser General
	# Public License version 2.1 or, at your option, any later version.

Raphael Shu zomux