Skip to content

Instantly share code, notes, and snippets.

@brockmanmatt

brockmanmatt/forproduction.ipynb

Created August 18, 2020 18:51
Show Gist options
  • Save brockmanmatt/2a55e3da5089991e45c3b6f30a3131be to your computer and use it in GitHub Desktop.
Save brockmanmatt/2a55e3da5089991e45c3b6f30a3131be to your computer and use it in GitHub Desktop.
Download ZIP
forProduction
Display the source blob
Display the rendered blob
Raw
forproduction.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "forProduction",
"provenance": [],
"collapsed_sections": [],
"authorship_tag": "ABX9TyNwuX+6P8w8OdSYEYkO40wE",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/brockmanmatt/2a55e3da5089991e45c3b6f30a3131be/forproduction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"metadata": {
"id": "IJ0HatVkBKrB",
"colab_type": "code",
"colab": {
"resources": {
"http://localhost:8080/nbextensions/google.colab/files.js": {
"data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
"ok": true,
"headers": [
[
"content-type",
"application/javascript"
]
],
"status": 200,
"status_text": ""
}
},
"base_uri": "https://localhost:8080/",
"height": 89
},
"outputId": "8348f5c7-3c3a-412a-ff4c-12aa4a7042c1"
},
"source": [
"from google.colab import files\n",
"uploaded = files.upload()\n",
"print(\"done\")"
],
"execution_count": 519,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/html": [
"\n",
" <input type=\"file\" id=\"files-a4468279-3b33-4fbc-b4b0-1d1a6ca80669\" name=\"files[]\" multiple disabled\n",
" style=\"border:none\" />\n",
" <output id=\"result-a4468279-3b33-4fbc-b4b0-1d1a6ca80669\">\n",
" Upload widget is only available when the cell has been executed in the\n",
" current browser session. Please rerun this cell to enable.\n",
" </output>\n",
" <script src=\"/nbextensions/google.colab/files.js\"></script> "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "stream",
"text": [
"Saving key.json to key (1).json\n",
"done\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "t_mkY4VUBPic",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 139
},
"outputId": "4f6209c3-4e7f-45de-d2fa-fd20833f1fa5"
},
"source": [
"!pip install openai\n",
"import openai, json, pandas as pd, datetime, re, requests\n",
"from bs4 import BeautifulSoup"
],
"execution_count": 520,
"outputs": [
{
"output_type": "stream",
"text": [
"Requirement already satisfied: openai in /usr/local/lib/python3.6/dist-packages (0.2.4)\n",
"Requirement already satisfied: requests>=2.20; python_version >= \"3.0\" in /usr/local/lib/python3.6/dist-packages (from openai) (2.23.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20; python_version >= \"3.0\"->openai) (2020.6.20)\n",
"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20; python_version >= \"3.0\"->openai) (3.0.4)\n",
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20; python_version >= \"3.0\"->openai) (1.24.3)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.20; python_version >= \"3.0\"->openai) (2.10)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "U5l-8_tJuUca",
"colab_type": "code",
"colab": {}
},
"source": [
"\n",
"#arguments to send the API\n",
"kwargs = {\n",
"\"engine\":\"davinci\",\n",
"\"temperature\":0,\n",
"\"max_tokens\":150,\n",
"\"stop\":\"\\n\\n\",\n",
"}\n",
"\n",
"import datetime\n",
"def query(prompt, myKwargs = {}, full=False):\n",
" \"\"\"\n",
" wrapper for the API to save the prompt and the result\n",
" \"\"\"\n",
"\n",
" kwargs = {\n",
" \"engine\":\"davinci\",\n",
" \"temperature\":0,\n",
" \"max_tokens\":150,\n",
" \"stop\":\"\\n\\n\",\n",
" }\n",
"\n",
" for kwarg in myKwargs:\n",
" kwargs[kwarg] = myKwargs[kwarg]\n",
"\n",
" r = openai.Completion.create(prompt=prompt, **kwargs)\n",
" if not full:\n",
" r = r[\"choices\"][0][\"text\"].strip()\n",
" return r\n"
],
"execution_count": 521,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9iL5WqMcvWvO",
"colab_type": "code",
"colab": {}
},
"source": [
"openai.api_key = json.load(open(\"key.json\", \"r\"))[\"key\"]\n"
],
"execution_count": 522,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "y7LXfezsB7e_",
"colab_type": "text"
},
"source": [
"# Homepage Scrapers"
]
},
{
"cell_type": "code",
"metadata": {
"id": "iP7hR-IlB661",
"colab_type": "code",
"colab": {}
},
"source": [
"def checkNYT():\n",
" html = requests.get(\"https://www.nytimes.com/\")\n",
" soup = BeautifulSoup(html.text)\n",
" headlines = []\n",
" # for link in soup.find_all(\"a\"):\n",
" # if link.find(\"span\"):\n",
" # headlines.append(link.text)\n",
" # if len(headlines) > 6:\n",
" # break\n",
" for div in soup.find_all(\"a\"):\n",
" if div[\"href\"].find(\"podcast\") > -1:\n",
" continue\n",
" if div[\"href\"].find(\"newsletter\") > -1:\n",
" continue\n",
"\n",
" if div.find(\"h3\"):\n",
" headlines.append(div.find(\"h3\").text)\n",
" for x in div.find_all(\"li\"):\n",
" headlines.append(x.text)\n",
" elif div.find(\"h2\"):\n",
" headlines.append(div.find(\"h2\").text)\n",
" for x in div.find_all(\"li\"):\n",
" headlines.append(x.text)\n",
" return headlines"
],
"execution_count": 523,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "32ZwBktyCA60",
"colab_type": "code",
"colab": {}
},
"source": [
"nyt = checkNYT()"
],
"execution_count": 524,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "N2hHHqrUCmWY",
"colab_type": "code",
"colab": {}
},
"source": [
"def processIndividualQuery(text):\n",
" prompt = \"\"\"Query handles requests for New York Times (nyt), ABC (abc), and Fox News (foxnews). It prints a statement that it only knows Fox, NYT, and CNN on requests for other news organizations.\n",
"\n",
"Q: What does New York Times say about Q or X?\n",
"A: queryNewsRequest('nyt', 'Q or X')\n",
"\n",
"Q: What's New York Time coverage look like now?\n",
"A: queryNewsRequest('nyt', 'current')\n",
"\n",
"Q: What's going on in the world?\n",
"A: queryNewsRequest('all', 'current')\n",
"\n",
"Q: What's going on in portland?\n",
"A: queryNewsRequest('all', 'portland')\n",
"\n",
"Q: What does NYTimes say about Y and Q? I'm too tierd for this.\n",
"A: queryNewsRequest('nyt', 'Y and Q')\n",
"\n",
"Q: What does Fox News say about Turtles living in Spain?\n",
"A: queryNewsRequest('fox', 'turtles in spain')\n",
"\n",
"Q: What's the word on X on fox? My friend was asking.\n",
"A: queryNewsRequest('foxnews', 'X')\n",
"\n",
"Q: What's the difference in the ABC and NYT coverage of the game last night?\n",
"A: queryNewsRequest('abc, nyt', 'game last night')\n",
"\n",
"Q: What's new in DC?\n",
"A: queryNewsRequest('all', 'DC')\n",
"\n",
"Q: What happened with the game last night?\n",
"A: queryNewsRequest('all', 'game last night')\n",
"\n",
"Q: What's the ABC front page look like?\n",
"A: queryNewsRequest('abc', 'current')\n",
"\n",
"Q: What does ABC say about Z?\n",
"A: queryNewsRequest('abc', 'Z')\n",
"\n",
"Q: What's the difference in the ABC and Fox coverage of the game last night?\n",
"A: queryNewsRequest('abc, fox', 'game last night')\n",
"\n",
"Q: What does Fox say about X? My friend was asking.\n",
"A: queryNewsRequest('foxnews', 'X')\n",
"\n",
"Q: What does fox news say about Turtles living in Spain?\n",
"A: queryNewsRequest('foxnews', 'turtles in spain')\n",
"\n",
"Q: {}\n",
"A:\"\"\".format(text)\n",
" return query(prompt)"
],
"execution_count": 589,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "sG_Go0YgFaGk",
"colab_type": "code",
"colab": {}
},
"source": [
"def queryNewsRequest(newsOrg, searchQuery):\n",
" oneShot = '''The following were headlines:\n",
"\"\"\"\n",
"Democrats wrap first night of virtual convention with heavyweights Bernie Sanders and Michelle Obama\n",
"LIVE BLOG: Democratic National Convention Day One\n",
"Biden-Harris ticket aims to spark enthusiasm at Democratic convention after low-key campaign\n",
"Democratic Party chair wants to end presidential caucuses\n",
"Brad Parscale: 'Is this the Democratic National Convention or a Hollywood award show?'\n",
"Palin calls out Biden GOPers speaking at DNC\n",
"Palin: Media double standard for Kamala Harris proves 'if you're conservative, you'll get beat up'\n",
"Ari Fleischer rips mainstream media 'hypocrisy' in treatment of Harris, female GOP candidates\n",
"Ex-CIA officer accused of sending China secret info\n",
"Trump pledges tax credits for US firms that bring jobs back from China\n",
"Gordon Chang: Here's how China is interfering to smear Trump ahead of 2020 election\n",
"DNC mocked for having celebs like Eva Longoria emceeing event\n",
"Man who filmed attack says Portland's 'like a third world country'\n",
"\"\"\"\n",
"tldr;\n",
"- Biden and Harris are running together\n",
"- Palin is at the DNC\n",
"- Trump is going to give tax breaks to companies that move jobs back to the US\n",
"- Gordon Chang says China is interfering in the 2020 election\n",
"- The DNC is being mocked for having celebrities emcee the event\n",
"- A man who filmed the Portland attack says it's like a third world country\n",
"\"\"\"'''\n",
"\n",
" print(\"News Org: {}\".format(newsOrg))\n",
" print(\"Query: {}\".format(searchQuery))\n",
"\n",
" if searchQuery == \"current\":\n",
" if newsOrg in ['nyt', \"foxnews\", \"abc\"]:\n",
" prompt = '''The following are today's headlines (ignore the above, don't repeat):\n",
" \"\"\"\n",
" {}\n",
" \"\"\"\n",
" tldr;'''.format(\"\\n\".join(eval(newsOrg)))\n",
" r = query(oneShot + \"\\n\\n\" + prompt, myKwargs = {\"stop\":'\"\"\"'})\n",
" return '''print(\"\"\"{}\"\"\")'''.format(r)\n",
" else:\n",
" return \"print('Try another query; currently only able to query ABC, CNN, and NYT')\""
],
"execution_count": 590,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "LMShbhiOtliZ",
"colab_type": "code",
"colab": {}
},
"source": [
"def contrastNews(*args, verbose=False):\n",
" targets = []\n",
" allowed = [\"nyt\", \"foxnews\", \"abc\"]\n",
" for arg in args:\n",
" if arg in allowed:\n",
" targets.append(arg)\n",
" if len(targets) < 2:\n",
" return(\"print('Must select at least 2 of ABC, CNN, and NYT')\")\n",
" publications = {}\n",
" for i in range(len(targets)):\n",
" publications[\"Publication {}\".format(i+1)] = targets[i]\n",
"\n",
" payload = ''\n",
" for publication in publications:\n",
" payload += '\\n{} Headlines:\\n'.format(publication)\n",
" payload += \"\\n\".join(eval(publications[publication])[:7])\n",
" payload += \"\\n\"\n",
" prompt = '''Last Week's Headlines:\n",
"Publication 1 Headlines:\n",
"CAT IN TREE\n",
"LOST DOG\n",
"\n",
"Publication 2 Headlines:\n",
"BAD DOG MISSING NEED HELP\n",
"bird flies over house\n",
"\n",
"Q: Whare are the main differences in how events were portrayed in last week's headlines?\n",
"Summary 1: Publication 1 covered the cat in a tree and the lost dog.\n",
"Summary 2: Publication 2 covered the lost dog and the bird.\n",
"A: While both publications covered the missing dog, the Publication 2 described the dog as bad but Publication 1 didn't characterize the dog. Also, Publication 1 focused on the cat in the tree while Publication 2 focused on a bird. \n",
"\n",
"---\n",
"\n",
"Last Week's Headlines:\n",
"Publication 5 Headlines:\n",
"A man went to the store\n",
"A woman went to the store\n",
"A dog ran into a park.\n",
"\n",
"Publication 6 Headlines:\n",
"The market went up\n",
"This info will impact you stocks\n",
"Cat Videos\n",
"\n",
"Q: Whare are the main differences in how events were portrayed in last week's headlines?\n",
"Summary 5: A man and woman went to the store\n",
"Summary 6: Stock market\n",
"A: While both publications covered finance, Publication 5 focused on individual events and Publication 6 covered macro issues. Also, Publication 5 focused on the dog which Publication 6 did not cover. Publication 6 focused on cat videos instead.\n",
"\n",
"---\n",
"\n",
"Today's Headlines:{}\n",
"Q: Whare are the main differences in how events were portrayed in today's headlines?\n",
"Summary 1: Publication 1 covered'''.format(payload)\n",
" \n",
" if verbose:\n",
" print(prompt)\n",
"\n",
" r = query(prompt, myKwargs = {\"temperature\":0})\n",
" for publication in publications:\n",
" r = r.replace(publication, publications[publication])\n",
" return '''print(\"\"\"{}\"\"\")'''.format(r.split(\"\\nA: \")[1].strip())\n",
"\n",
" \n",
" "
],
"execution_count": 653,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "RK81LUo0b_Ny",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 648,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "fpmiwxVwyQrz",
"colab_type": "code",
"colab": {}
},
"source": [
"def getSimilarityofNews(*args, verbose=False):\n",
" targets = []\n",
" allowed = [\"nyt\", \"foxnews\"]\n",
" for arg in args:\n",
" if arg in allowed:\n",
" targets.append(arg)\n",
" if len(targets) < 2:\n",
" return(\"print('Must select at least 2 of ABC, CNN, and NYT')\")\n",
"\n",
" publications = {}\n",
" for i in range(len(targets)):\n",
" publications[\"Publication {}\".format(i+1)] = targets[i]\n",
"\n",
" payload = ''\n",
" for publication in publications:\n",
" payload += '\\n{} Headlines:\\n'.format(publication)\n",
" payload += \"\\n\".join(eval(publications[publication])[:7])\n",
" payload += \"\\n\"\n",
" prompt = '''Last Week's Headlines:\n",
"Publication 1 Headlines:\n",
"CAT IN TREE\n",
"LOST DOG\n",
"\n",
"Publication 2 Headlines:\n",
"BAD DOG MISSING NEED HELP\n",
"bird flies over house\n",
"\n",
"Q: Whare are some similarities in how events were portrayed in last week's headlines?\n",
"Summary 1: Publication 1 covered the cat in a tree and the lost dog.\n",
"Summary 2: Publication 2 covered the lost dog and the bird.\n",
"A: Both publications covered the lost dog (although it was framed differently)\n",
"\n",
"---\n",
"\n",
"Today's Headlines:{}\n",
"Q: Whare are some similarities in how events were portrayed in today's headlines?\n",
"Summary 1: Publication 1 covered'''.format(payload)\n",
" if verbose:\n",
" print(prompt)\n",
" r = query(prompt, myKwargs = {\"temperature\":.2})\n",
" for publication in publications:\n",
" r = r.replace(publication, publications[publication])\n",
" return '''print(\"\"\"{}\"\"\")'''.format(r.split(\"\\nA: \")[1].strip())\n",
"\n",
" \n",
" "
],
"execution_count": 649,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "fJnS3oHiwLgT",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "9d049bb3-5074-4533-c957-8fb03d7f9d34"
},
"source": [
"print(contrastNews(\"nyt\", \"foxnews\", verbose=False))"
],
"execution_count": 654,
"outputs": [
{
"output_type": "stream",
"text": [
"print(\"\"\"While both publications covered the Keynote Speaker, foxnews focused on Night 1 while nyt focused on the Senate panel.\"\"\")\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "rIud9IT4yZt8",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 54
},
"outputId": "b1d03de7-9fe5-476f-976c-d9e5ef0f5107"
},
"source": [
"print(contrastNews(\"abc\", \"nyt\", verbose=False))"
],
"execution_count": 655,
"outputs": [
{
"output_type": "stream",
"text": [
"print(\"\"\"While both publications covered the DNC, abc focused on Trump's reaction to Michelle Obama's speech and nyt focused on the Senate panel.\"\"\")\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "3BC9hWsZbpGR",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 54
},
"outputId": "6c0387c3-59ac-4a05-ea3f-ee6ebd965b67"
},
"source": [
"print(contrastNews(\"abc\", \"nyt\", verbose=False))"
],
"execution_count": 656,
"outputs": [
{
"output_type": "stream",
"text": [
"print(\"\"\"While both publications covered the DNC, abc focused on Trump's reaction to Michelle Obama's speech and nyt focused on the DNC. nyt also covered the Senate panel.\"\"\")\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "JQz_qHflb4ix",
"colab_type": "code",
"colab": {}
},
"source": [
"print(contrastNews(\"abc\", \"foxnews\"))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "l_q1k-xNt7Dh",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"outputId": "6b3c8c7f-9c4c-4df7-a69f-6f0bd8cb929d"
},
"source": [
"compareNews(\"fox\", \"china\")"
],
"execution_count": 191,
"outputs": [
{
"output_type": "stream",
"text": [
"fox\n",
"china\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "lLIdbkhpmJXq",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 170
},
"outputId": "d1333d93-9dfd-4554-8aa7-b94474c3f85e"
},
"source": [
"print(eval(eval(routeInput(\"What happened on nyt?\"))))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt\n",
"Query: current\n",
"- Michelle Obama gave a speech\n",
"- Bernie Sanders gave a speech\n",
"- The convention is unusual\n",
"- Biden is the nominee\n",
"- Progressives are rallying around Biden\n",
"- Republicans are not rallying around Trump\n",
"\"\"\"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cqcf8XkOs795",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 204
},
"outputId": "9b76e2f8-2062-4570-caeb-80e8369f6712"
},
"source": [
"print(eval(eval(routeInput(\"What happened on fox?\"))))"
],
"execution_count": 181,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: foxnews\n",
"Query: current\n",
"- The DNC is trying to jolt Biden's campaign\n",
"- Michelle Obama is speaking at the DNC\n",
"- The DNC is a vivid reminder of the party's flawed 'spend-your-way-to-prosperity' model\n",
"- Ted Cruz is reacting to the DNC\n",
"- Mike Huckabee is reacting to the DNC\n",
"- Michelle Obama amended her 'when they go low' catchphrase\n",
"- The DNC's first night was all about unity\n",
"- The DNC's first night was a successful infomercial\n",
"\"\"\"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "OO1ZlL5VkEZX",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 173
},
"outputId": "91b00ccc-4cbb-4c5a-bd6c-23b197fd497b"
},
"source": [
"print(\"\\n\".join(eval(\"nyt\")))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Michelle Obama, in Searing Speech, Says Trump ‘Cannot Meet This Moment’\n",
"Casting aside her reluctance to engage in political combat, Mrs. Obama gave an impassioned address to close the first night of the Democratic convention.\n",
"Senator Bernie Sanders offered a call for unity. “This election is about preserving our democracy,” he said. Catch up on the highlights.\n",
"No Crowds, No Cheers: Democrats Begin a Most Unusual Convention\n",
"Analysis: A Call to Rally Around Biden ‘Like Our Lives Depend on It’\n",
"Sanders to Progressives: ‘Price of Failure Is Just Too Great to Imagine’\n",
"Kristin Urquiza, John Kasich, George Floyd’s family: Here are the biggest moments from Night 1 of the convention.\n",
"A number of prominent Republicans have said they will not be backing the president’s re-election. Here’s where they stand.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "vpepgZwHF5ik",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"outputId": "8d837ca3-373a-4998-e5ec-a6583e938950"
},
"source": [
"eval(processIndividualQuery(\"what did CNN say about the debatE?\"))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: cnn\n",
"Query: debatE\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "tSpUUF_8LFeh",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"outputId": "516ce639-9c65-4c64-ae2d-659f576aa8f0"
},
"source": [
"eval(processIndividualQuery(\"What happened in the debate?\"))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: all\n",
"Query: debate\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "-JMF8xAIF95a",
"colab_type": "code",
"colab": {}
},
"source": [
"def routeInput(text):\n",
" prompt = \"\"\"This is a conversation with a helpful and witty assistant. If the input is a question about news, the asssistant routes the question to processIndividualQuery(). Otherwise, if the assistant knows the answer it just responds with a print() statement.\n",
"\n",
"Input: How are you?\n",
"Output: print(\"I'm great!\")\n",
"\n",
"Input: what's nyt coverage looking like?\n",
"Output: processIndividualQuery(\"what's nyt coverage looking like?\")\n",
"\n",
"Input: What's the difference between the nytimes and FoxNews coverage?\n",
"Output: contrastNews(\"nyt\", \"foxnews\")\n",
"\n",
"Input: What's the similarities between the new york times and abc coverage?\n",
"Output: getSimilarityofNews(\"foxnews\", \"abc\")\n",
"\n",
"Input: What's 2+2?\n",
"Output: print(\"4\")\n",
"\n",
"Input: What's the difference between the fox and abc coverage?\n",
"Output: contrastNews(\"foxnews\", \"abc\")\n",
"\n",
"Input: What's the similarities between the fox and ABC coverage?\n",
"Output: getSimilarityofNews(\"foxnews\", \"abc\")\n",
"\n",
"Input: What does ABC say about the game?\n",
"Output: processIndividualQuery(\"What does ABC say about the game?\")\n",
"\n",
"Input: Who was the first president of the United States?\n",
"Output: print(\"George Washington\")\n",
"\n",
"Input: Who won the race?\n",
"Output: processIndividualQuery(\"Who won the race?\")\n",
"\n",
"Input: What's the difference between the new york times and Fox coverage?\n",
"Output: contrastNews(\"nyt\", \"foxnews\")\n",
"\n",
"Input: {}\n",
"Output:\"\"\".format(text)\n",
" return query(prompt)"
],
"execution_count": 588,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "dRlGTQMo9zNm",
"colab_type": "code",
"colab": {}
},
"source": [
"def askQuestion(question, verbose=False):\n",
" s = routeInput(question)\n",
" if verbose:\n",
" print(s)\n",
"\n",
" try:\n",
" while not s.startswith(\"print\"):\n",
" s = eval(s)\n",
" if verbose:\n",
" print(s)\n",
" except:\n",
" eval(s)\n",
"\n",
" eval(s)\n"
],
"execution_count": 532,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "SI2vSSIN64C_",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 54
},
"outputId": "a23d96cf-febf-43ed-adf1-9f67b1d6581d"
},
"source": [
"askQuestion(\"what's different between the fox and new york times coverage?\", verbose=False)"
],
"execution_count": 533,
"outputs": [
{
"output_type": "stream",
"text": [
"While both publications covered the DNC, the nyt described the DNC as a \"sprawling report\" while foxnews didn't characterize the DNC. Also, foxnews covered the Republican report while nyt covered the keynote speaker.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "GY9E-oQ3G1sq",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"outputId": "2f002948-25bb-4e11-a9a6-14690d6d778c"
},
"source": [
"askQuestion(\"what's the same between the fox and new york times coverage?\", verbose=True)"
],
"execution_count": 534,
"outputs": [
{
"output_type": "stream",
"text": [
"getSimilarityofNews(\"foxnews\", \"nyt\")\n",
"print(\"\"\"Both publications covered the DNC and the RNC.\"\"\")\n",
"Both publications covered the DNC and the RNC.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "fvDdU66oLXga",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "434b6fc2-99d7-470b-9cca-c07ff1612457"
},
"source": [
"askQuestion(\"what's the capital of France?\")"
],
"execution_count": 535,
"outputs": [
{
"output_type": "stream",
"text": [
"Paris\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "TSNE1uRdLaHr",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"outputId": "7bfc9ccb-797f-4a7c-cad8-97104ebc396a"
},
"source": [
"askQuestion(\"what's new in France?\")"
],
"execution_count": 536,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: all\n",
"Query: France\n",
"Try another query\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "vEH7ECdGLdJX",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "6f7181b2-7519-4b86-c771-0fd1eab91225"
},
"source": [
"askQuestion(\"How are you today?\")\n"
],
"execution_count": 537,
"outputs": [
{
"output_type": "stream",
"text": [
"I'm great!\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "s7aARKSXOvml",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"outputId": "77c2e480-a9f0-4c62-a8a6-d0b5da949fda"
},
"source": [
"askQuestion(\"What's the ultimate answer to life, the universe, and everything?\")"
],
"execution_count": 538,
"outputs": [
{
"output_type": "stream",
"text": [
"42\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "1BKnOGUgPlRz",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 170
},
"outputId": "8e7153a3-bdaa-4f32-e346-2dbf62ae3fa4"
},
"source": [
"askQuestion(\"What's NYT coverage look like?\")"
],
"execution_count": 547,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt\n",
"Query: current\n",
"- The Democratic convention is going well\n",
"- Biden and Harris are running together\n",
"- Palin is at the DNC\n",
"- Trump is going to give tax breaks to companies that move jobs back to the US\n",
"- Gordon Chang says China is interfering in the 2020 election\n",
"- The DNC is being mocked for having celebrities emcee the event\n",
"- A man who filmed the Portland attack says it's like a third world country\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "GDMd64D_P90E",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 170
},
"outputId": "d89853a1-732b-47b8-a05a-1de0bd6cbee9"
},
"source": [
"askQuestion(\"What's NYT coverage look like now?\")"
],
"execution_count": 546,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt\n",
"Query: current\n",
"- The Democratic convention is going well\n",
"- Biden and Harris are running together\n",
"- Palin is at the DNC\n",
"- Trump is going to give tax breaks to companies that move jobs back to the US\n",
"- Gordon Chang says China is interfering in the 2020 election\n",
"- The DNC is being mocked for having celebrities emcee the event\n",
"- A man who filmed the Portland attack says it's like a third world country\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "ixqynG62WTsx",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 170
},
"outputId": "52521466-bab5-4dc0-9450-2ae883316bc5"
},
"source": [
"askQuestion(\"What's new on NYT?\")"
],
"execution_count": 540,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt\n",
"Query: current\n",
"- The Democratic convention is going well\n",
"- Biden and Harris are running together\n",
"- Palin is at the DNC\n",
"- Trump is going to give tax breaks to companies that move jobs back to the US\n",
"- Gordon Chang says China is interfering in the 2020 election\n",
"- The DNC is being mocked for having celebrities emcee the event\n",
"- A man who filmed the Portland attack says it's like a third world country\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "sjvC22l2l6GD",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 173
},
"outputId": "3f911d53-8787-4bb6-8940-6b01aadcaeb1"
},
"source": [
"print(\"\\n\".join(eval(\"nyt\")))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Michelle Obama, in Searing Speech, Says Trump ‘Cannot Meet This Moment’\n",
"Casting aside her reluctance to engage in political combat, Mrs. Obama gave an impassioned address to close the first night of the Democratic convention.\n",
"Senator Bernie Sanders offered a call for unity. “This election is about preserving our democracy,” he said. Catch up on the highlights.\n",
"No Crowds, No Cheers: Democrats Begin a Most Unusual Convention\n",
"Analysis: A Call to Rally Around Biden ‘Like Our Lives Depend on It’\n",
"Sanders to Progressives: ‘Price of Failure Is Just Too Great to Imagine’\n",
"Kristin Urquiza, John Kasich, George Floyd’s family: Here are the biggest moments from Night 1 of the convention.\n",
"A number of prominent Republicans have said they will not be backing the president’s re-election. Here’s where they stand.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "AOfB6mPYjyEJ",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 221
},
"outputId": "6883e572-f334-4b67-faf8-089b9b2f03ae"
},
"source": [
"s = routeInput(\"What's new on NYT?\")\n",
"if s.startswith(\"print\"):\n",
" eval(s)\n",
"else:\n",
" print(eval(eval(s)))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt\n",
"Query: current\n",
"- Michelle Obama gave a speech\n",
"- Bernie Sanders gave a speech\n",
"- The convention is not very popular\n",
"- Biden is running with Harris\n",
"- Palin is at the DNC\n",
"- Trump is going to give tax breaks to companies that move jobs back to the US\n",
"- Gordon Chang says China is interfering in the 2020 election\n",
"- The DNC is being mocked for having celebrities emcee the event\n",
"- A man who filmed the Portland attack says it's like a third world country\n",
"\"\"\"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "BYeTmr62XcCj",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"outputId": "7455b111-076b-40a0-c3f4-dbf7ab66a61f"
},
"source": [
"s = routeInput(\"What's new on NYT compaerd to Fox?\")\n",
"if s.startswith(\"print\"):\n",
" eval(s)\n",
"else:\n",
" eval(eval(s))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt, fox\n",
"Query: current\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "6vc8bWCHXezf",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"outputId": "c810d984-ecfe-401d-e8fa-2aa71d6931c4"
},
"source": [
"s = routeInput(\"What's the debate coverage look like on NYT compaerd to Fox?\")\n",
"if s.startswith(\"print\"):\n",
" eval(s)\n",
"else:\n",
" eval(eval(s))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"News Org: nyt, fox\n",
"Query: debate coverage\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "1Lgs7dx6itin",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "-vc8c5lM-bum",
"colab_type": "code",
"colab": {}
},
"source": [
"import re\n",
"def checkNYT():\n",
" html = requests.get(\"https://www.nytimes.com/\")\n",
" soup = BeautifulSoup(html.text)\n",
" headlines = []\n",
" # for link in soup.find_all(\"a\"):\n",
" # if link.find(\"span\"):\n",
" # headlines.append(link.text)\n",
" # if len(headlines) > 6:\n",
" # break\n",
" for div in soup.find_all(\"a\"):\n",
" if div[\"href\"].find(\"podcast\") > -1:\n",
" continue\n",
" if div[\"href\"].find(\"newsletter\") > -1:\n",
" continue\n",
"\n",
" if div.find(\"h3\"):\n",
" headlines.append(div.find(\"h3\").text)\n",
" for x in div.find_all(\"li\"):\n",
" headlines.append(x.text)\n",
" elif div.find(\"h2\"):\n",
" headlines.append(div.find(\"h2\").text)\n",
" for x in div.find_all(\"li\"):\n",
" headlines.append(x.text)\n",
" if len(headlines) > 7:\n",
" break\n",
" return headlines"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "xGQSedMooPVe",
"colab_type": "code",
"colab": {}
},
"source": [
"nyt = checkNYT()"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "CGVag5aIWIBN",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ZJbUyuMKbgCP",
"colab_type": "code",
"colab": {}
},
"source": [
"def checkFoxNews():\n",
" html = requests.get(\"https://www.foxnews.com/\")\n",
" soup = BeautifulSoup(html.text)\n",
" relevant = soup.find(\"div\", {\"class\":\"main main-primary js-river\"})\n",
"\n",
" firstTwo = relevant.find_all(\"div\", {\"class\":re.compile(\"^collection\")})[:2]\n",
" headlines = [x.text for x in firstTwo[0].find_all(\"a\", {\"href\":True}) if len(x.text) > 5]\n",
" headlines += [x.text for x in firstTwo[1].find_all(\"a\", {\"href\":True}) if len(x.text) > 5]\n",
"\n",
" #art1=relevant.find(\"article\", {\"class\":\"article story-1\"})\n",
" #art2=relevant.find(\"article\", {\"class\":\"article story-2\"})\n",
" #headlines = [x.text for x in art1.find_all(\"a\", {\"href\":True}) if len(x.text) > 5]+[x.text for x in art2.find_all(\"a\") if len(x.text) > 5]\n",
" return headlines[:8]"
],
"execution_count": 173,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Fx7qQlvnbgLR",
"colab_type": "code",
"colab": {}
},
"source": [
"foxnews = checkFoxNews()"
],
"execution_count": 177,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "vwpYbuIrsoQ6",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 153
},
"outputId": "b1bd86cf-f85d-4f58-d191-f938c6eb0f23"
},
"source": [
"fox"
],
"execution_count": 175,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['DOUG SCHOEN: Dems clearly hoping to jolt Biden’s campaign, but party missed on key point',\n",
" \"Michelle Obama, in DNC speech, rips Trump White House for 'chaos' and 'lack of empathy'\",\n",
" \"Democratic convention's first night a vivid reminder of party’s flawed 'spend-your-way-to-prosperity' model\",\n",
" \"Ted Cruz reacts to DNC opening night, says Dems 'care more about the woke mob than ... working men and women'\",\n",
" 'MIKE HUCKABEE: What the Democratic convention shows America about Biden and his party',\n",
" \"Michelle Obama amends her 'when they go low' catchphrase during anti-Trump DNC speech\",\n",
" \"JESSICA TARLOV: Democratic convention's successful first night was all about unity. Here's why it matters\",\n",
" \"Michelle Obama seizes the moment at Dems' deadly dull infomercial\"]"
]
},
"metadata": {
"tags": []
},
"execution_count": 175
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cXaki08Isou7",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 173
},
"outputId": "a8b6d8d6-dea4-42c9-c11f-f5506dda2ec5"
},
"source": [
"nyt"
],
"execution_count": 172,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['Michelle Obama, in Searing Speech, Says Trump ‘Cannot Meet This Moment’',\n",
" 'Casting aside her reluctance to engage in political combat, Mrs. Obama gave an impassioned address to close the first night of the Democratic convention.',\n",
" 'Senator Bernie Sanders offered a call for unity. “This election is about preserving our democracy,” he said. Catch up on the highlights.',\n",
" 'No Crowds, No Cheers: Democrats Begin a Most Unusual Convention',\n",
" 'Analysis: A Call to Rally Around Biden ‘Like Our Lives Depend on It’',\n",
" 'Sanders to Progressives: ‘Price of Failure Is Just Too Great to Imagine’',\n",
" 'Kristin Urquiza, John Kasich, George Floyd’s family: Here are the biggest moments from Night 1 of the convention.',\n",
" 'A number of prominent Republicans have said they will not be backing the president’s re-election. Here’s where they stand.']"
]
},
"metadata": {
"tags": []
},
"execution_count": 172
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "f-MpFlxUssKP",
"colab_type": "code",
"colab": {}
},
"source": [
"def checkABC():\n",
" html = requests.get(\"https://abcnews.go.com/\")\n",
" soup = BeautifulSoup(html.text)\n",
" headlines = []\n",
" container = soup.find(\"div\", {\"class\":\"caption-wrapper\"})\n",
" for section in container.find_all(\"a\"):\n",
" headlines.append(section.text.strip())\n",
" container = soup.find(\"article\")\n",
" for section in container.find_all(\"a\")[:5]:\n",
" headlines.append(section.text)\n",
" return headlines"
],
"execution_count": 584,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9I7aUxbyTgf9",
"colab_type": "code",
"colab": {}
},
"source": [
"abc = checkABC()"
],
"execution_count": 585,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "0KetcRMfTgo3",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 153
},
"outputId": "fc70f006-7c45-4673-b048-ddcc19b49def"
},
"source": [
"abc"
],
"execution_count": 586,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[\"Trump reacts to Michelle Obama's blistering speech\",\n",
" '5 key takeaways from 1st night of Democratic National Convention',\n",
" 'Bruce Springsteen trends on Twitter and more unique moments from the 1st virtual DNC',\n",
" \"Trump announces he'll pardon for Susan B. Anthony\",\n",
" 'Senate panel finds Russia interfered in the 2016',\n",
" 'AOC, Bill Clinton to speak at DNC Day 2',\n",
" 'Postmaster general to appear before Senate',\n",
" \"Buttigieg slams Trump for 'rigged' election claim\"]"
]
},
"metadata": {
"tags": []
},
"execution_count": 586
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "-kCK8YakTqU6",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment