Skip to content

Instantly share code, notes, and snippets.

@rjpower

rjpower/keyphrase.ipynb

Created July 12, 2016 16:31
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save rjpower/eacf87bafbf8a21b09a42bdab5ca2e23 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save rjpower/eacf87bafbf8a21b09a42bdab5ca2e23 to your computer and use it in GitHub Desktop.
Download ZIP
keyphrase classification
Display the source blob
Display the rendered blob
Raw
keyphrase.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Learning to rank using embeddings\n",
"\n",
"This notebook is a template for using keras with research paper data. In the real system, we'd load in our ranking data alongside and try to predict our ranker, but for now we'll predict keyphrases given titles as a proxy.\n",
"\n",
"## TODO\n",
"* Tokenizer should take a pre-built word_to_idx mapping\n",
"* Write tweak layer\n",
"* Load ranking data\n",
"* Try out convolution layer"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
}
],
"source": [
"import json\n",
"import sys\n",
"import os\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"import keras\n",
"from keras.layers import Dense, Activation, Dropout, Flatten, Merge, Layer, RepeatVector\n",
"from keras.layers.recurrent import LSTM\n",
"from keras.layers.embeddings import Embedding\n",
"from keras.models import Sequential\n",
"from keras.preprocessing import text as keras_text\n",
"import keras.preprocessing.sequence as keras_sequence\n",
"\n",
"import keras.backend as K\n",
"\n",
"import deeplearn\n",
"\n",
"from __future__ import print_function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Loading Data\n",
"\n",
"To start, we'll be using the pre-built word2vec vectors from Google; we should be able to swap in our own w2v vectors at a later date. We will be using the filtered paper data from our corpus (340k documents)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading.... /data/filtered-papers/joined-json/part-r-00000-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00001-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00002-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00003-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00004-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00005-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00006-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00007-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00008-4a18d38d-7ee1-4190-8c9a-f453040bea01\n",
"Loading.... /data/filtered-papers/joined-json/part-r-00009-4a18d38d-7ee1-4190-8c9a-f453040bea01\n"
]
}
],
"source": [
"import glob\n",
"files = sorted(glob.glob('/data/filtered-papers/joined-json/part-r-*'))\n",
"\n",
"def load_files():\n",
" frames = []\n",
" for file in files[:10]:\n",
" print('Loading....', file)\n",
" with open(file) as f:\n",
" frames.append(pd.DataFrame.from_records([\n",
" json.loads(line) for line in f\n",
" ]))\n",
" del frames[-1]['body_text']\n",
" \n",
" return pd.concat(frames).reset_index()\n",
" \n",
"paper_data = load_files()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"VOCAB_SIZE = 50000\n",
"BATCH_SIZE = 32\n",
"MAX_CONTEXT = 16\n",
"EMBEDDING_SIZE = 100"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Processing... 4.42MB."
]
}
],
"source": [
"from deeplearn import preprocessing\n",
"import importlib\n",
"importlib.reload(preprocessing)\n",
"tokenizer = preprocessing.Tokenizer(vocab_size=VOCAB_SIZE)\n",
"tokenizer.fit(paper_data.title)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Predicting Keyphrases\n",
"\n",
"We'll use a toy problem: predicting the top keyphrase of a paper given it's title to get started. We'll take the top `MAX_KEYPHRASES` keyphrases by count, and build a predictor for them."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r",
"Processing... 4.60MB."
]
}
],
"source": [
"MAX_KEYPHRASES = 1000\n",
"\n",
"import collections\n",
"kp_counts = collections.Counter()\n",
"for kp in paper_data.key_phrases:\n",
" kp_counts.update(kp)\n",
" \n",
"keyphrases = dict(sorted(kp_counts.items(), key=lambda kv: kv[1], reverse=True))\n",
"kp_to_idx = dict(zip(keyphrases.keys(), range(0, MAX_KEYPHRASES)))\n",
"idx_to_kp = { idx:kp for (kp, idx) in kp_to_idx.items() }"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Batching...\n"
]
},
{
"data": {
"text/plain": [
"(array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 809, 181, 6, 40, 414],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 1579, 3,\n",
" 295, 105, 199, 6, 446, 869, 161],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 2,\n",
" 1655, 728, 13, 4, 2, 24, 12],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 663, 30,\n",
" 0, 462, 553, 5703, 9, 582, 671],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 1, 2, 3],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 14, 1,\n",
" 49, 24542, 25, 6, 1484, 841, 2192],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 17, 22, 2],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 1, 10, 8],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 25, 1,\n",
" 664, 548, 715, 3, 40, 32, 11],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 1, 10, 738, 1083, 336, 30],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 109, 0, 1],\n",
" [ 0, 0, 0, 0, 0, 0, 121, 2369, 1,\n",
" 4739, 506, 6, 524, 405, 790, 3191],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 244, 88, 732, 0, 1304, 4219, 183],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 147, 200, 88, 9, 831, 216],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 4, 1, 3],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 117, 148,\n",
" 1084, 29, 2, 87, 0, 145, 41],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 298, 463, 854, 0, 273],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 1107, 3, 7, 59],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 1, 0, 3],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 8, 2, 3],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 87, 1, 2015, 110, 9],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 427,\n",
" 416, 19, 8, 1334, 4, 92, 888],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 2, 1, 0],\n",
" [ 0, 0, 0, 0, 0, 0, 1, 906, 201,\n",
" 7, 2169, 4079, 1, 111, 1213, 605],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 586, 27, 9, 275, 402, 55],\n",
" [ 0, 0, 0, 0, 0, 58, 3, 1473, 12,\n",
" 15271, 187, 420, 2182, 2, 1683, 1554],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 900, 7, 207, 186, 223, 3, 227],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 20156, 8, 25204, 245, 8646],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 1, 1, 4, 1, 2],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 251, 9,\n",
" 514, 1227, 2, 262, 37, 2621, 174],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 1, 2, 0],\n",
" [ 0, 0, 0, 0, 0, 0, 0, 0, 18,\n",
" 426, 2132, 2130, 80, 3385, 2, 30]], dtype=int32),\n",
" array([[ 0., 0., 0., ..., 0., 0., 0.],\n",
" [ 0., 0., 0., ..., 0., 0., 0.],\n",
" [ 0., 0., 0., ..., 0., 0., 0.],\n",
" ..., \n",
" [ 0., 0., 0., ..., 0., 0., 0.],\n",
" [ 0., 0., 0., ..., 0., 0., 0.],\n",
" [ 0., 0., 0., ..., 0., 0., 0.]], dtype=float32))"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def _terms(txt):\n",
" terms_ary = np.zeros((MAX_CONTEXT,), dtype=np.int32)\n",
" terms = np.asarray(list(tokenizer.terms_from_text(txt)))\n",
" terms = terms[:MAX_CONTEXT]\n",
" if len(terms) < 3:\n",
" return None\n",
" terms_ary[-len(terms):] = terms \n",
" return terms_ary\n",
" \n",
"\n",
"def training_data():\n",
" batch = []\n",
" for idx, paper in paper_data.iterrows():\n",
" kp_ary = np.zeros((MAX_KEYPHRASES,), dtype=np.float32)\n",
" kps = [kp_to_idx[k] for k in paper.key_phrases if k in kp_to_idx]\n",
" terms_ary = _terms(paper.title)\n",
" \n",
" if terms_ary is None or len(kps) == 0:\n",
" continue\n",
"\n",
" kp_ary[kps[0]] = 1.\n",
" yield terms_ary, kp_ary\n",
" \n",
"def batchify(gen, batch_size):\n",
" print('Batching...')\n",
" examples = []\n",
" labels = []\n",
" for e, l in gen:\n",
" examples.append(e)\n",
" labels.append(l)\n",
" if len(examples) >= batch_size:\n",
" yield np.asarray(examples), np.asarray(labels)\n",
" examples = []\n",
" labels = []\n",
"\n",
"next(batchify(preprocessing.forever(lambda: training_data()), 32))"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"____________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"====================================================================================================\n",
"bow_1 (BOW) (None, 100) 5000000 bow_input_1[0][0] \n",
"____________________________________________________________________________________________________\n",
"stripmask_1 (StripMask) (None, 100) 0 bow_1[0][0] \n",
"____________________________________________________________________________________________________\n",
"dense_1 (Dense) (None, 1000) 101000 stripmask_1[0][0] \n",
"____________________________________________________________________________________________________\n",
"activation_1 (Activation) (None, 1000) 0 dense_1[0][0] \n",
"====================================================================================================\n",
"Total params: 5101000\n",
"____________________________________________________________________________________________________\n"
]
}
],
"source": [
"import keras.backend as K\n",
"import tensorflow as tf\n",
"from keras.regularizers import l2\n",
"from deeplearn.layers import BOW, StripMask\n",
"\n",
"def model():\n",
" with tf.device('/gpu:0'):\n",
" words = Sequential()\n",
" words.add(BOW(\n",
" input_dim=VOCAB_SIZE, \n",
" output_dim=EMBEDDING_SIZE,\n",
" input_length=MAX_CONTEXT, \n",
" mask_zero=True,\n",
" weights=None\n",
" ))\n",
" words.add(StripMask())\n",
" words.add(Dense(MAX_KEYPHRASES))\n",
" words.add(Activation('sigmoid'))\n",
" words.compile(loss='categorical_crossentropy', optimizer='adadelta')\n",
" return words\n",
"\n",
"training_model = model()\n",
"training_model.summary()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Batching...Epoch 1/10\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/russellp/anaconda3/lib/python3.5/site-packages/Keras-1.0.4-py3.5.egg/keras/engine/training.py:1403: UserWarning: Epoch comprised more than `samples_per_epoch` samples, which might affect learning results. Set `samples_per_epoch` correctly to avoid this warning.\n",
" warnings.warn('Epoch comprised more than '\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"98s - loss: 6.3787\n",
"Epoch 2/10\n",
"97s - loss: 5.8631\n",
"Epoch 3/10\n",
"98s - loss: 5.7398\n",
"Epoch 4/10\n",
"97s - loss: 5.6721\n",
"Epoch 5/10\n",
"96s - loss: 5.5888\n",
"Epoch 6/10\n",
"96s - loss: 5.5046\n",
"Epoch 7/10\n",
"96s - loss: 5.3951\n",
"Epoch 8/10\n",
"96s - loss: 5.2350\n",
"Epoch 9/10\n",
"95s - loss: 5.0771\n",
"Epoch 10/10\n",
"97s - loss: 4.9227\n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x7f66b2f52160>"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"training_model.fit_generator(\n",
" batchify(preprocessing.forever(lambda: training_data()), 32),\n",
" samples_per_epoch=10000,\n",
" verbose=2,\n",
" nb_epoch=10)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exploring the role of individual differences in information visualization\n",
"0.177668 Design Space\n",
"0.228286 USER EXPERIENCE\n",
"0.277629 SUS\n",
"0.999475 False Positive\n",
"0.999949 Twitter\n",
"No terms!\n",
"No terms!\n",
"No terms!\n",
"Mahler measures, short walks and log-sine integrals\n",
"0.183263 SMART CARD\n",
"0.21358 Design Space\n",
"0.284401 Kullback-Leibler Divergence\n",
"0.997784 False Positive\n",
"0.999478 Twitter\n",
"No terms!\n",
"No terms!\n",
"No terms!\n",
"No terms!\n",
"An hybrid finite volume-finite element method for variable density incompressible flows\n",
"0.329046 SMART CARD\n",
"0.331887 Pattern Matching\n",
"0.343717 Kullback-Leibler Divergence\n",
"0.977185 False Positive\n",
"0.991388 Twitter\n",
"No terms!\n",
"Minimization of exclusive sum-of-products expressions for multiple-valued input, incompletely specified functions\n",
"0.366679 Design Space\n",
"0.373654 CONVERGENCE RATE\n",
"0.408311 Spanish\n",
"0.988152 False Positive\n",
"0.996225 Twitter\n",
"No terms!\n",
"No terms!\n",
"Body-centric interaction with mobile devices\n",
"0.224811 Design Space\n",
"0.253866 Kullback-Leibler Divergence\n",
"0.31476 Relevance Feedback\n",
"0.999368 False Positive\n",
"0.999842 Twitter\n",
"Implicit modeling using subdivision curves\n",
"0.284351 Pattern Matching\n",
"0.302852 Relevance Feedback\n",
"0.329516 Kullback-Leibler Divergence\n",
"0.999235 False Positive\n",
"0.999818 Twitter\n",
"Performance and Energy Benefits of Instruction Set Extensions in an FPGA Soft Core\n",
"0.0953808 Spanish\n",
"0.11628 Design Space\n",
"0.121704 SUS\n",
"0.99992 False Positive\n",
"0.999995 Twitter\n",
"No terms!\n",
"No terms!\n",
"Visual analytic roadblocks for novice investigators\n",
"0.252092 SMART CARD\n",
"0.348931 Kullback-Leibler Divergence\n",
"0.406167 Pattern Matching\n",
"0.998562 False Positive\n",
"0.99958 Twitter\n"
]
}
],
"source": [
"def evaluate(model, example):\n",
" terms = _terms(example)\n",
" if terms is None:\n",
" print('No terms!')\n",
" return\n",
" prediction = model.predict_on_batch(terms.reshape((1,16)))[0]\n",
" best_idx = np.argsort(prediction)[-5:]\n",
" print(example)\n",
" for i in best_idx:\n",
" print(prediction[i], idx_to_kp[i])\n",
" \n",
"for i in range(20):\n",
" evaluate(training_model, paper_data.loc[i].title)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>\n",
"<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
"<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>\n",
"<style>\n",
" @font-face {\n",
" font-family: \"Computer Modern\";\n",
" src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
" }\n",
" div.cell{\n",
" width:800px;\n",
" margin-left:16% !important;\n",
" margin-right:auto;\n",
" }\n",
" h1 {\n",
" font-family: 'Alegreya Sans', sans-serif;\n",
" }\n",
" h2 {\n",
" font-family: 'Fenix', serif;\n",
" }\n",
" h3{\n",
"\t\tfont-family: 'Fenix', serif;\n",
" margin-top:12px;\n",
" margin-bottom: 3px;\n",
" }\n",
"\th4{\n",
"\t\tfont-family: 'Fenix', serif;\n",
" }\n",
" h5 {\n",
" font-family: 'Alegreya Sans', sans-serif;\n",
" }\t \n",
" div.text_cell_render{\n",
" font-family: 'Alegreya Sans',Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
" line-height: 135%;\n",
" font-size: 120%;\n",
" width:600px;\n",
" margin-left:auto;\n",
" margin-right:auto;\n",
" }\n",
" .CodeMirror{\n",
" font-family: \"Source Code Pro\";\n",
"\t\t\tfont-size: 90%;\n",
" }\n",
"/* .prompt{\n",
" display: None;\n",
" }*/\n",
" .text_cell_render h1 {\n",
" font-weight: 200;\n",
" font-size: 50pt;\n",
"\t\tline-height: 100%;\n",
" color:#CD2305;\n",
" margin-bottom: 0.5em;\n",
" margin-top: 0.5em;\n",
" display: block;\n",
" }\t\n",
" .text_cell_render h5 {\n",
" font-weight: 300;\n",
" font-size: 16pt;\n",
" color: #CD2305;\n",
" font-style: italic;\n",
" margin-bottom: .5em;\n",
" margin-top: 0.5em;\n",
" display: block;\n",
" }\n",
" \n",
" .warning{\n",
" color: rgb( 240, 20, 20 )\n",
" } \n",
"</style>\n",
"<script>\n",
" MathJax.Hub.Config({\n",
" TeX: {\n",
" extensions: [\"AMSmath.js\"]\n",
" },\n",
" tex2jax: {\n",
" inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
" displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
" },\n",
" displayAlign: 'center', // Change this to 'center' to center equations.\n",
" \"HTML-CSS\": {\n",
" styles: {'.MathJax_Display': {\"margin\": 4}}\n",
" }\n",
" });\n",
"</script>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.core.display import HTML\n",
"import requests\n",
"\n",
"def css_styling():\n",
" styles = requests.get('https://raw.githubusercontent.com/barbagroup/CFDPython/master/styles/custom.css').text\n",
" return HTML(styles)\n",
"css_styling()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.1+"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
@tanthml

tanthml commented Apr 26, 2018

Copy link
Copy Markdown

Awesome,
Could you please give me the details of reference papers you are following to implement this one ?
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment