Machel Reid machelreid

Abstractive Summarization (CNN-DM)

Here are things that I spent a lot of time on, so you don’t have to - especially with regard to preprocessing data for abstractive summarization. It will be pretty disorganized, but bear with me - there might be something useful in here.

Using Rouge

Important: Don’t use pure Python implementations of Rouge!! Use the following Python wrapper for the original Perl package: https://github.com/pltrdy/files2rouge

	#!/usr/bin/env python
	# coding=utf-8
	# Copyright The HuggingFace Team and The HuggingFace Inc. team. All rights reserved.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#

	import argparse

	def get_enc_params(embed_dim, ffn_dim):
	return embed_dim * embed_dim * 4 + embed_dim * ffn_dim * 2 + embed_dim * 5 + ffn_dim


	def get_dec_params(embed_dim, ffn_dim, encoder_embed_dim=None):
	return (
	embed_dim * embed_dim * 4
	+ embed_dim * ffn_dim * 2

	import sys
	import os
	import hashlib
	import struct
	import subprocess
	import collections


	dm_single_close_quote = u'\u2019' # unicode
	dm_double_close_quote = u'\u201d'

	#!/bin/bash

	TEXT=$1
	echo Finding $TEXT
	grep -iRl "$TEXT" ./

Machel Reid machelreid

Abstractive Summarization (CNN-DM)

Using Rouge

Preprocessing Data for Abstractive Summarization