Skip to content

Instantly share code, notes, and snippets.

@kkdai

kkdai/github-isssue-rag-langchain.ipynb

Last active February 21, 2024 03:36
Show Gist options
  • Save kkdai/dee9005f76dee41195a49165fe0b876d to your computer and use it in GitHub Desktop.
Save kkdai/dee9005f76dee41195a49165fe0b876d to your computer and use it in GitHub Desktop.
Download ZIP
Github Issues LangChain RAG Q&A 的 Colab
Display the source blob
Display the rendered blob
Raw
github-isssue-rag-langchain.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"authorship_tag": "ABX9TyMzAuZJbUWznco36ptpzT43",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/kkdai/dee9005f76dee41195a49165fe0b876d/github-isssue-rag-langchain.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "afECGc6ooY_8",
"outputId": "4a96e476-1fb5-4c55-b301-b488d1073e48"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Python 3.10.12\n"
]
}
],
"source": [
"!python --version\n",
"#Python 3.10.12, but langchain need workaround for pydantic==1.10.8 to fix import issue. change to 3.11"
]
},
{
"cell_type": "code",
"source": [
"!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11\n",
"#Install Python3.11 (no need for final version, will use latest one)"
],
"metadata": {
"id": "aodmVwTcrBjh",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "a9878d4f-be4d-4a63-95cc-71a95a9a3859"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"update-alternatives: --install needs <link> <name> <path> <priority>\n",
"\n",
"Use 'update-alternatives --help' for program usage information.\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!sudo apt-get update -y\n",
"!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11"
],
"metadata": {
"id": "hpMQDB_Yp3cY",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "0734f47f-7ab6-4a1d-ed5f-c13dffe1fcff"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\r0% [Working]\r \rHit:1 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
"\r0% [Waiting for headers] [Connected to cloud.r-project.org (108.157.162.103)] [\r \rGet:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]\n",
"\r \rGet:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]\n",
"\r0% [2 InRelease 27.2 kB/119 kB 23%] [3 InRelease 48.9 kB/110 kB 44%] [Waiting f\r \rHit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease\n",
"\r0% [2 InRelease 47.5 kB/119 kB 40%] [3 InRelease 77.9 kB/110 kB 71%] [Waiting f\r \rGet:5 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease [3,626 B]\n",
"Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
"Hit:7 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease\n",
"Get:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease [18.1 kB]\n",
"Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
"Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
"Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,796 kB]\n",
"Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,070 kB]\n",
"Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,463 kB]\n",
"Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [50.4 kB]\n",
"Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,742 kB]\n",
"Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,834 kB]\n",
"Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,343 kB]\n",
"Get:18 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy/main amd64 Packages [28.2 kB]\n",
"Fetched 9,578 kB in 5s (2,128 kB/s)\n",
"Reading package lists... Done\n",
"update-alternatives: --install needs <link> <name> <path> <priority>\n",
"\n",
"Use 'update-alternatives --help' for program usage information.\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!sudo apt update\n",
"!sudo apt install python3-pip"
],
"metadata": {
"id": "aV6qcBo6slLD",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "261f4376-a22d-457d-ae0c-1a6ec3662a43"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\u001b[33m\r0% [Working]\u001b[0m\r \rHit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease\n",
"\u001b[33m\r0% [Waiting for headers] [Connected to cloud.r-project.org (108.157.162.103)] [\u001b[0m\r \rHit:2 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
"\u001b[33m\r0% [Waiting for headers] [Connected to cloud.r-project.org (108.157.162.103)] [\u001b[0m\r \rHit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n",
"\u001b[33m\r0% [Waiting for headers] [Waiting for headers] [Connecting to ppa.launchpadcont\u001b[0m\r \rHit:4 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease\n",
"\u001b[33m\r0% [Waiting for headers] [Connected to ppa.launchpadcontent.net (185.125.190.80\u001b[0m\r \rHit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
"Hit:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease\n",
"Hit:7 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease\n",
"Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n",
"Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
"Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
"Reading package lists... Done\n",
"Building dependency tree... Done\n",
"Reading state information... Done\n",
"43 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
"Reading package lists... Done\n",
"Building dependency tree... Done\n",
"Reading state information... Done\n",
"The following additional packages will be installed:\n",
" python3-setuptools python3-wheel\n",
"Suggested packages:\n",
" python-setuptools-doc\n",
"The following NEW packages will be installed:\n",
" python3-pip python3-setuptools python3-wheel\n",
"0 upgraded, 3 newly installed, 0 to remove and 43 not upgraded.\n",
"Need to get 1,677 kB of archives.\n",
"After this operation, 8,967 kB of additional disk space will be used.\n",
"Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-setuptools all 59.6.0-1.2ubuntu0.22.04.1 [339 kB]\n",
"Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-wheel all 0.37.1-2ubuntu0.22.04.1 [32.0 kB]\n",
"Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-pip all 22.0.2+dfsg-1ubuntu0.4 [1,305 kB]\n",
"Fetched 1,677 kB in 0s (5,154 kB/s)\n",
"debconf: unable to initialize frontend: Dialog\n",
"debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 3.)\n",
"debconf: falling back to frontend: Readline\n",
"debconf: unable to initialize frontend: Readline\n",
"debconf: (This frontend requires a controlling tty.)\n",
"debconf: falling back to frontend: Teletype\n",
"dpkg-preconfigure: unable to re-open stdin: \n",
"Selecting previously unselected package python3-setuptools.\n",
"(Reading database ... 121749 files and directories currently installed.)\n",
"Preparing to unpack .../python3-setuptools_59.6.0-1.2ubuntu0.22.04.1_all.deb ...\n",
"Unpacking python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...\n",
"Selecting previously unselected package python3-wheel.\n",
"Preparing to unpack .../python3-wheel_0.37.1-2ubuntu0.22.04.1_all.deb ...\n",
"Unpacking python3-wheel (0.37.1-2ubuntu0.22.04.1) ...\n",
"Selecting previously unselected package python3-pip.\n",
"Preparing to unpack .../python3-pip_22.0.2+dfsg-1ubuntu0.4_all.deb ...\n",
"Unpacking python3-pip (22.0.2+dfsg-1ubuntu0.4) ...\n",
"Setting up python3-setuptools (59.6.0-1.2ubuntu0.22.04.1) ...\n",
"Setting up python3-wheel (0.37.1-2ubuntu0.22.04.1) ...\n",
"Setting up python3-pip (22.0.2+dfsg-1ubuntu0.4) ...\n",
"Processing triggers for man-db (2.10.2-1) ...\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!python --version\n",
"# Make sure it changes to 3.11"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "L2xTRH62mzAb",
"outputId": "e0317516-2310-4b96-c818-e70abb2085de"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Python 3.10.12\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!pip install install langchain_community\n",
"!pip install langchain_openai\n",
"!pip install langchain_core\n",
"!pip install faiss-cpu"
],
"metadata": {
"id": "L9t8K0iPqn28",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "7f6286a3-323a-40fb-a5b0-58402c08bb29"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: install in /usr/local/lib/python3.10/dist-packages (1.3.5)\n",
"Collecting langchain_community\n",
" Downloading langchain_community-0.0.20-py3-none-any.whl (1.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (6.0.1)\n",
"Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (2.0.27)\n",
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (3.9.3)\n",
"Collecting dataclasses-json<0.7,>=0.5.7 (from langchain_community)\n",
" Downloading dataclasses_json-0.6.4-py3-none-any.whl (28 kB)\n",
"Collecting langchain-core<0.2,>=0.1.21 (from langchain_community)\n",
" Downloading langchain_core-0.1.23-py3-none-any.whl (241 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m241.2/241.2 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting langsmith<0.1,>=0.0.83 (from langchain_community)\n",
" Downloading langsmith-0.0.92-py3-none-any.whl (56 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (1.25.2)\n",
"Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (2.31.0)\n",
"Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain_community) (8.2.3)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (23.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.4.1)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (6.0.5)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (1.9.4)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain_community) (4.0.3)\n",
"Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain_community)\n",
" Downloading marshmallow-3.20.2-py3-none-any.whl (49 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain_community)\n",
" Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
"Requirement already satisfied: anyio<5,>=3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.21->langchain_community) (3.7.1)\n",
"Collecting jsonpatch<2.0,>=1.33 (from langchain-core<0.2,>=0.1.21->langchain_community)\n",
" Downloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
"Collecting langsmith<0.1,>=0.0.83 (from langchain_community)\n",
" Downloading langsmith-0.0.87-py3-none-any.whl (55 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.4/55.4 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: packaging<24.0,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.21->langchain_community) (23.2)\n",
"Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.21->langchain_community) (2.6.1)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (3.3.2)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (3.6)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_community) (2024.2.2)\n",
"Requirement already satisfied: typing-extensions>=4.6.0 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain_community) (4.9.0)\n",
"Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain_community) (3.0.3)\n",
"Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.21->langchain_community) (1.3.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.21->langchain_community) (1.2.0)\n",
"Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain-core<0.2,>=0.1.21->langchain_community)\n",
" Downloading jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.2,>=0.1.21->langchain_community) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.2,>=0.1.21->langchain_community) (2.16.2)\n",
"Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain_community)\n",
" Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
"Installing collected packages: mypy-extensions, marshmallow, jsonpointer, typing-inspect, jsonpatch, langsmith, dataclasses-json, langchain-core, langchain_community\n",
"Successfully installed dataclasses-json-0.6.4 jsonpatch-1.33 jsonpointer-2.4 langchain-core-0.1.23 langchain_community-0.0.20 langsmith-0.0.87 marshmallow-3.20.2 mypy-extensions-1.0.0 typing-inspect-0.9.0\n",
"Collecting langchain_openai\n",
" Downloading langchain_openai-0.0.6-py3-none-any.whl (29 kB)\n",
"Requirement already satisfied: langchain-core<0.2,>=0.1.16 in /usr/local/lib/python3.10/dist-packages (from langchain_openai) (0.1.23)\n",
"Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain_openai) (1.25.2)\n",
"Collecting openai<2.0.0,>=1.10.0 (from langchain_openai)\n",
" Downloading openai-1.12.0-py3-none-any.whl (226 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tiktoken<1,>=0.5.2 (from langchain_openai)\n",
" Downloading tiktoken-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m42.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (6.0.1)\n",
"Requirement already satisfied: anyio<5,>=3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (3.7.1)\n",
"Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (1.33)\n",
"Requirement already satisfied: langsmith<0.0.88,>=0.0.87 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (0.0.87)\n",
"Requirement already satisfied: packaging<24.0,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (23.2)\n",
"Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (2.6.1)\n",
"Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (2.31.0)\n",
"Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.16->langchain_openai) (8.2.3)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai<2.0.0,>=1.10.0->langchain_openai) (1.7.0)\n",
"Collecting httpx<1,>=0.23.0 (from openai<2.0.0,>=1.10.0->langchain_openai)\n",
" Downloading httpx-0.26.0-py3-none-any.whl (75 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.9/75.9 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.10.0->langchain_openai) (1.3.0)\n",
"Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.10.0->langchain_openai) (4.66.2)\n",
"Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from openai<2.0.0,>=1.10.0->langchain_openai) (4.9.0)\n",
"Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken<1,>=0.5.2->langchain_openai) (2023.12.25)\n",
"Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.16->langchain_openai) (3.6)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.16->langchain_openai) (1.2.0)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai<2.0.0,>=1.10.0->langchain_openai) (2024.2.2)\n",
"Collecting httpcore==1.* (from httpx<1,>=0.23.0->openai<2.0.0,>=1.10.0->langchain_openai)\n",
" Downloading httpcore-1.0.3-py3-none-any.whl (77 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.0/77.0 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore==1.*->httpx<1,>=0.23.0->openai<2.0.0,>=1.10.0->langchain_openai)\n",
" Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.2,>=0.1.16->langchain_openai) (2.4)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.2,>=0.1.16->langchain_openai) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain-core<0.2,>=0.1.16->langchain_openai) (2.16.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-core<0.2,>=0.1.16->langchain_openai) (3.3.2)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain-core<0.2,>=0.1.16->langchain_openai) (2.0.7)\n",
"Installing collected packages: h11, tiktoken, httpcore, httpx, openai, langchain_openai\n",
"\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
"llmx 0.0.15a0 requires cohere, which is not installed.\u001b[0m\u001b[31m\n",
"\u001b[0mSuccessfully installed h11-0.14.0 httpcore-1.0.3 httpx-0.26.0 langchain_openai-0.0.6 openai-1.12.0 tiktoken-0.6.0\n",
"Requirement already satisfied: langchain_core in /usr/local/lib/python3.10/dist-packages (0.1.23)\n",
"Requirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (6.0.1)\n",
"Requirement already satisfied: anyio<5,>=3 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (3.7.1)\n",
"Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (1.33)\n",
"Requirement already satisfied: langsmith<0.0.88,>=0.0.87 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (0.0.87)\n",
"Requirement already satisfied: packaging<24.0,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (23.2)\n",
"Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (2.6.1)\n",
"Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (2.31.0)\n",
"Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain_core) (8.2.3)\n",
"Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain_core) (3.6)\n",
"Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain_core) (1.3.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain_core) (1.2.0)\n",
"Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain_core) (2.4)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain_core) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain_core) (2.16.2)\n",
"Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain_core) (4.9.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_core) (3.3.2)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_core) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain_core) (2024.2.2)\n",
"Collecting faiss-cpu\n",
" Downloading faiss_cpu-1.7.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.6 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.6/17.6 MB\u001b[0m \u001b[31m33.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hInstalling collected packages: faiss-cpu\n",
"Successfully installed faiss-cpu-1.7.4\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"from langchain_community.document_loaders import GitHubIssuesLoader"
],
"metadata": {
"id": "2l2dcXG9ohG4"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from google.colab import userdata\n",
"import os\n",
"\n",
"github_token = userdata.get('GITHUB_TOKEN')"
],
"metadata": {
"id": "uZ9CoMq8neOB"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"loader = GitHubIssuesLoader(\n",
" repo=\"kkdai/bookmarks\",\n",
" access_token=github_token, # delete/comment out this argument if you've set the access token as an env var.\n",
" creator=\"kkdai\",\n",
" include_prs=False,\n",
")"
],
"metadata": {
"id": "RKWvkQ6_o18t"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"docs = loader.load()"
],
"metadata": {
"id": "QM7Vb6FWpR1m"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"len(docs)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZMgiXHVQ_Rj3",
"outputId": "dec8ff06-82cc-478e-96a5-f35bd94229ec"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1371"
]
},
"metadata": {},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"source": [
"from operator import itemgetter\n",
"\n",
"from langchain_community.vectorstores import FAISS\n",
"from langchain_core.output_parsers import StrOutputParser\n",
"from langchain_core.prompts import ChatPromptTemplate\n",
"from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n",
"from langchain_openai import ChatOpenAI, OpenAIEmbeddings"
],
"metadata": {
"id": "bxjHTedw_VaJ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from google.colab import userdata\n",
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')"
],
"metadata": {
"id": "8kH4LQo7_7_n"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!pip install faiss-cpu"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MzsUYkBSB65I",
"outputId": "72e6abe5-f0d3-4890-d831-732898b5449f"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: faiss-cpu in /usr/local/lib/python3.10/dist-packages (1.7.4)\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!pip install langchain"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "TUYHCw7oqkY_",
"outputId": "d7e77560-5bb7-46be-94c5-a349843effb3"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Collecting langchain\n",
" Downloading langchain-0.1.7-py3-none-any.whl (815 kB)\n",
"\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/815.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m174.1/815.9 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━\u001b[0m \u001b[32m583.7/815.9 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m815.9/815.9 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: PyYAML>=5.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (6.0.1)\n",
"Requirement already satisfied: SQLAlchemy<3,>=1.4 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.0.27)\n",
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.10/dist-packages (from langchain) (3.9.3)\n",
"Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (4.0.3)\n",
"Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.6.4)\n",
"Requirement already satisfied: jsonpatch<2.0,>=1.33 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.33)\n",
"Requirement already satisfied: langchain-community<0.1,>=0.0.20 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.20)\n",
"Requirement already satisfied: langchain-core<0.2,>=0.1.22 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.1.23)\n",
"Requirement already satisfied: langsmith<0.1,>=0.0.83 in /usr/local/lib/python3.10/dist-packages (from langchain) (0.0.87)\n",
"Requirement already satisfied: numpy<2,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (1.25.2)\n",
"Requirement already satisfied: pydantic<3,>=1 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.6.1)\n",
"Requirement already satisfied: requests<3,>=2 in /usr/local/lib/python3.10/dist-packages (from langchain) (2.31.0)\n",
"Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from langchain) (8.2.3)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.5)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n",
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (3.20.2)\n",
"Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (0.9.0)\n",
"Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.10/dist-packages (from jsonpatch<2.0,>=1.33->langchain) (2.4)\n",
"Requirement already satisfied: anyio<5,>=3 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.22->langchain) (3.7.1)\n",
"Requirement already satisfied: packaging<24.0,>=23.2 in /usr/local/lib/python3.10/dist-packages (from langchain-core<0.2,>=0.1.22->langchain) (23.2)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.16.2 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (2.16.2)\n",
"Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1->langchain) (4.9.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.3.2)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (3.6)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2->langchain) (2024.2.2)\n",
"Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\n",
"Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.22->langchain) (1.3.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.22->langchain) (1.2.0)\n",
"Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain) (1.0.0)\n",
"Installing collected packages: langchain\n",
"Successfully installed langchain-0.1.7\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
"text_splitter = RecursiveCharacterTextSplitter(\n",
" chunk_size = 500,\n",
" chunk_overlap = 50,\n",
" length_function = len\n",
")\n",
"\n",
"ecicoder_chunks = text_splitter.transform_documents(docs)"
],
"metadata": {
"id": "CgDNp73OqGJq"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"ecicoder_chunks[0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "i3z9XuDqqqH3",
"outputId": "5ec1d3d0-cc82-4cbf-91c9-cd6ec0a5a3be"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"Document(page_content='{\\n \"summary\": \"分享一组我在手机上经常使用的大模型应用：\\\\n- ChatGPT\\\\n- Perplexity\\\\n- Hugging Chat\\\\n- Gemini\\\\n#twitter\",\\n \"full_content\": \"分享一组我在手机上经常使用的大模型应用：\\\\n- ChatGPT\\\\n- Perplexity\\\\n- Hugging Chat\\\\n- Gemini\\\\n#twitter\",\\n \"url\": \"https://t.co/MVmqv8Q6AH\",\\n \"tags\": [\"#twitter\", \"自然语言处理\", \"大模型\", \"手机应用\"]\\n}', metadata={'url': 'https://github.com/kkdai/bookmarks/issues/1394', 'title': '分享一组我在手机上经常使用的大模型应用：\\n- ChatGPT\\n- Perplexity\\n- Hugging Chat\\n- Gemini\\n#twitter', 'creator': 'kkdai', 'created_at': '2024-02-19T00:32:00Z', 'comments': 0, 'state': 'open', 'labels': ['#twitter', '自然语言处理', '大模型', '手机应用'], 'assignee': None, 'milestone': None, 'locked': False, 'number': 1394, 'is_pull_request': False})"
]
},
"metadata": {},
"execution_count": 17
}
]
},
{
"cell_type": "code",
"source": [
"from langchain.llms.openai import OpenAIChat\n",
"from langchain.chains import RetrievalQA\n",
"from langchain.callbacks import StdOutCallbackHandler"
],
"metadata": {
"id": "aL_0AEEsrSTF"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"embeddings = OpenAIEmbeddings()\n",
"vectorstore = FAISS.from_documents(docs, embeddings)\n",
"vectorstore.add_documents(ecicoder_chunks)\n",
"\n",
"retriever = vectorstore.as_retriever()\n",
"handler = StdOutCallbackHandler()\n",
"llm = ChatOpenAI()\n",
"\n",
"qa_with_sources_chain = RetrievalQA.from_chain_type(\n",
" llm=llm,\n",
" retriever=retriever,\n",
" callbacks=[handler],\n",
" return_source_documents=True\n",
")"
],
"metadata": {
"id": "QWU9zCwD_Wx8"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# This is the entire augment system!\n",
"response = qa_with_sources_chain({\"query\":\"關於多模態大語言模型的相關資料?\"})"
],
"metadata": {
"id": "gJQ5U3Sy_Zli",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "dd11af8c-6d0d-4156-b99c-8f42b0c56e73"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"print(response['source_documents'])\n"
],
"metadata": {
"id": "PWdKpZRq_bTo",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "9c9a103f-b919-43bc-8ad0-b6118df9818d"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[Document(page_content='[link](https://t.co/8MZKBmWsMZ) [link](https://t.co/Ju92uWWVoQ)\\n本文對近年來多模態大語言模型（MM-LLMs）的發展進行了全面的調查，介紹了26種現有的MM-LLMs，並探討了它們在主流基準上的表現和訓練技巧。', metadata={'url': 'https://github.com/kkdai/bookmarks/issues/1341', 'title': '多模態大語言模型的最新進展 ', 'creator': 'kkdai', 'created_at': '2024-01-25T09:06:34Z', 'comments': 0, 'state': 'open', 'labels': ['paper', 'Twitter', 'LMM'], 'assignee': None, 'milestone': None, 'locked': False, 'number': 1341, 'is_pull_request': False}), Document(page_content=' [link](https://t.co/8MZKBmWsMZ) [link](https://t.co/Ju92uWWVoQ)\\n本文對近年來多模態大語言模型（MM-LLMs）的發展進行了全面的調查，介紹了26種現有的MM-LLMs，並探討了它們在主流基準上的表現和訓練技巧。', metadata={'url': 'https://github.com/kkdai/bookmarks/issues/1341', 'title': '多模態大語言模型的最新進展 ', 'creator': 'kkdai', 'created_at': '2024-01-25T09:06:34Z', 'comments': 0, 'state': 'open', 'labels': ['paper', 'Twitter', 'LMM'], 'assignee': None, 'milestone': None, 'locked': False, 'number': 1341, 'is_pull_request': False}), Document(page_content='[link](https://t.co/EjnZbe9eGb)\\n這是一個能夠處理多模態和長文本的混合專家模型，能夠從數百萬個字元的文本中回憶和推理出細節信息，包括多個長文檔和數小時的視頻和音頻。', metadata={'url': 'https://github.com/kkdai/bookmarks/issues/1385', 'title': 'https://t.co/EjnZbe9eGb\\n\\nGemini 1.5 Pro - 解鎖1M 個 token 的多模態理解 ', 'creator': 'kkdai', 'created_at': '2024-02-16T03:56:44Z', 'comments': 0, 'state': 'open', 'labels': ['Twitter'], 'assignee': None, 'milestone': None, 'locked': False, 'number': 1385, 'is_pull_request': False}), Document(page_content='[link](https://t.co/NWLKLbyrcZ)\\n- 現今程式設計師沒理由不去使用 輔助程式設計- 正確地向大模型提問是一項關鍵技能，學會向 提問也有利於提升程式設計師的溝通能力\\n- 把 不能完全替代文件，畢竟有幻覺。\\n\\n有人翻譯成中文(有廣告):', metadata={'url': 'https://github.com/kkdai/bookmarks/issues/1298', 'title': '關於 ChatGPT 寫 Code 來自 Antirez 的建議 \\n', 'creator': 'kkdai', 'created_at': '2024-01-03T07:56:09Z', 'comments': 0, 'state': 'open', 'labels': ['chatGPT', 'LLM'], 'assignee': None, 'milestone': None, 'locked': False, 'number': 1298, 'is_pull_request': False})]\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"print(response['result'])\n",
"for doc in response['source_documents']:\n",
" url = doc.metadata['url']\n",
" title = doc.metadata['title']\n",
" print(url, title)\n"
],
"metadata": {
"id": "CCdASAb-sR1Y",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "ffe4fa8f-2025-4a04-f59d-51f8cdc78c25"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"根據提供的資訊，這篇文章探討了近年來多模態大語言模型（MM-LLMs）的發展，介紹了26種現有的MM-LLMs，並討論了它們在主流基準上的表現和訓練技巧。此外，還有一個混合專家模型，可以處理多模態和長文本，從數百萬個字元的文本中回憶和推理出細節信息，包括多個長文檔和數小時的視頻和音頻。如果您想深入了解多模態大語言模型，您可以查看提供的連結以獲取更多資訊。\n",
"https://github.com/kkdai/bookmarks/issues/1341 多模態大語言模型的最新進展 \n",
"https://github.com/kkdai/bookmarks/issues/1341 多模態大語言模型的最新進展 \n",
"https://github.com/kkdai/bookmarks/issues/1385 https://t.co/EjnZbe9eGb\n",
"\n",
"Gemini 1.5 Pro - 解鎖1M 個 token 的多模態理解 \n",
"https://github.com/kkdai/bookmarks/issues/1298 關於 ChatGPT 寫 Code 來自 Antirez 的建議 \n",
"\n"
]
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment