{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "IOpw--87wvkW"
},
"source": [
"We're going to build Lt. Data using RAG again, but this time using the langchain library as a way of showing another way of doing it. We'll also use the ragas package to evaluate our results, measuring faithfulness, answer relevancy, context precision, context recall, and answer correctness. This will give us a benchmark as we try and improve this model in subsequent activities in the course.\n",
"\n",
"We will start by parsing the original scripts and extracting lines spoken by Data. As before, you will need to upload all of the script files into a tng folder within your sample_data folder in your CoLab workspace first.\n",
"\n",
"An archive can be found at https://www.st-minutiae.com/resources/scripts/ (look for \"All TNG Epsiodes\"), but you could easily adapt this to read scripts from your favorite character from your favorite TV show or movie instead.|"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "dLjupJ8rLXr6"
},
"outputs": [],
"source": [
"import os\n",
"import re\n",
"import random\n",
"\n",
"dialogues = []\n",
"\n",
"def strip_parentheses(s):\n",
" return re.sub(r'\\(.*?\\)', '', s)\n",
"\n",
"def is_single_word_all_caps(s):\n",
" # First, we split the string into words\n",
" words = s.split()\n",
"\n",
" # Check if the string contains only a single word\n",
" if len(words) != 1:\n",
" return False\n",
"\n",
" # Make sure it isn't a line number\n",
" if bool(re.search(r'\\d', words[0])):\n",
" return False\n",
"\n",
" # Check if the single word is in all caps\n",
" return words[0].isupper()\n",
"\n",
"def extract_character_lines(file_path, character_name):\n",
" lines = []\n",
" with open(file_path, 'r') as script_file:\n",
" try:\n",
" lines = script_file.readlines()\n",
" except UnicodeDecodeError:\n",
" pass\n",
"\n",
" is_character_line = False\n",
" current_line = ''\n",
" current_character = ''\n",
" for line in lines:\n",
" strippedLine = line.strip()\n",
" if (is_single_word_all_caps(strippedLine)):\n",
" is_character_line = True\n",
" current_character = strippedLine\n",
" elif (line.strip() == '') and is_character_line:\n",
" is_character_line = False\n",
" dialog_line = strip_parentheses(current_line).strip()\n",
" dialog_line = dialog_line.replace('\"', \"'\")\n",
" if (current_character == 'DATA' and len(dialog_line)>0):\n",
" dialogues.append(dialog_line)\n",
" current_line = ''\n",
" elif is_character_line:\n",
" current_line += line.strip() + ' '\n",
"\n",
"def process_directory(directory_path, character_name):\n",
" for filename in os.listdir(directory_path):\n",
" file_path = os.path.join(directory_path, filename)\n",
" if os.path.isfile(file_path): # Ignore directories\n",
" extract_character_lines(file_path, character_name)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "Yh3peBSuMiyx"
},
"outputs": [],
"source": [
"process_directory(\"./sample_data/tng\", 'DATA')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jqjo_OdEyeNF"
},
"source": [
"Again, let's do a little sanity check to make sure the lines imported correctly, and print out the first one."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "x5dPI-29Myxo",
"outputId": "70e2eff5-ea36-47f1-f9c4-418189c8c117"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"The enemy vessel is firing.\n"
]
}
],
"source": [
"print (dialogues[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "toMfh7KUoKMv"
},
"source": [
"We will once again use OpenAI's API for our RAG model, so make sure that is installed:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4Sqtgr4kQQG9",
"outputId": "64820bfd-161c-485b-f3a0-9a3c39523dc5"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: openai in /usr/local/lib/python3.12/dist-packages (2.8.1)\n",
"Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.12/dist-packages (from openai) (4.11.0)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from openai) (1.9.0)\n",
"Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from openai) (0.28.1)\n",
"Requirement already satisfied: jiter<1,>=0.10.0 in /usr/local/lib/python3.12/dist-packages (from openai) (0.11.1)\n",
"Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from openai) (2.12.3)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.12/dist-packages (from openai) (1.3.1)\n",
"Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.12/dist-packages (from openai) (4.67.1)\n",
"Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.12/dist-packages (from openai) (4.15.0)\n",
"Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.12/dist-packages (from anyio<5,>=3.5.0->openai) (3.11)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->openai) (2025.11.12)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.9)\n",
"Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.16.0)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic<3,>=1.9.0->openai) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.41.4 in /usr/local/lib/python3.12/dist-packages (from pydantic<3,>=1.9.0->openai) (2.41.4)\n",
"Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from pydantic<3,>=1.9.0->openai) (0.4.2)\n"
]
}
],
"source": [
"!pip install openai --upgrade"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V0OYKDrHoQVv"
},
"source": [
"We also need to install the ragas package for measuring our results, along with langchain (for OpenAI)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ThM9aVJhm1St",
"outputId": "fb847fcd-2ff0-4862-d5a0-932e8cf5aa80"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: ragas in /usr/local/lib/python3.12/dist-packages (0.3.9)\n",
"Requirement already satisfied: langchain in /usr/local/lib/python3.12/dist-packages (1.1.0)\n",
"Requirement already satisfied: langchain-openai in /usr/local/lib/python3.12/dist-packages (1.1.0)\n",
"Requirement already satisfied: langchain-community in /usr/local/lib/python3.12/dist-packages (0.4.1)\n",
"Requirement already satisfied: datasets in /usr/local/lib/python3.12/dist-packages (4.4.1)\n",
"Requirement already satisfied: numpy<3.0.0,>=1.21.0 in /usr/local/lib/python3.12/dist-packages (from ragas) (2.0.2)\n",
"Requirement already satisfied: tiktoken in /usr/local/lib/python3.12/dist-packages (from ragas) (0.12.0)\n",
"Requirement already satisfied: pydantic>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from ragas) (2.12.3)\n",
"Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.12/dist-packages (from ragas) (1.6.0)\n",
"Requirement already satisfied: appdirs in /usr/local/lib/python3.12/dist-packages (from ragas) (1.4.4)\n",
"Requirement already satisfied: diskcache>=5.6.3 in /usr/local/lib/python3.12/dist-packages (from ragas) (5.6.3)\n",
"Requirement already satisfied: typer in /usr/local/lib/python3.12/dist-packages (from ragas) (0.20.0)\n",
"Requirement already satisfied: rich in /usr/local/lib/python3.12/dist-packages (from ragas) (13.9.4)\n",
"Requirement already satisfied: openai>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from ragas) (2.8.1)\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.12/dist-packages (from ragas) (4.67.1)\n",
"Requirement already satisfied: instructor in /usr/local/lib/python3.12/dist-packages (from ragas) (1.13.0)\n",
"Requirement already satisfied: gitpython in /usr/local/lib/python3.12/dist-packages (from ragas) (3.1.45)\n",
"Requirement already satisfied: pillow>=10.4.0 in /usr/local/lib/python3.12/dist-packages (from ragas) (11.3.0)\n",
"Requirement already satisfied: networkx in /usr/local/lib/python3.12/dist-packages (from ragas) (3.6)\n",
"Requirement already satisfied: scikit-network in /usr/local/lib/python3.12/dist-packages (from ragas) (0.33.5)\n",
"Requirement already satisfied: langchain-core in /usr/local/lib/python3.12/dist-packages (from ragas) (1.1.0)\n",
"Requirement already satisfied: langgraph<1.1.0,>=1.0.2 in /usr/local/lib/python3.12/dist-packages (from langchain) (1.0.3)\n",
"Requirement already satisfied: langchain-classic<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (1.0.0)\n",
"Requirement already satisfied: SQLAlchemy<3.0.0,>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.0.44)\n",
"Requirement already satisfied: requests<3.0.0,>=2.32.5 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.32.5)\n",
"Requirement already satisfied: PyYAML<7.0.0,>=5.3.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (6.0.3)\n",
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (3.13.2)\n",
"Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.1.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (9.1.2)\n",
"Requirement already satisfied: dataclasses-json<0.7.0,>=0.6.7 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (0.6.7)\n",
"Requirement already satisfied: pydantic-settings<3.0.0,>=2.10.1 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.12.0)\n",
"Requirement already satisfied: langsmith<1.0.0,>=0.1.125 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (0.4.47)\n",
"Requirement already satisfied: httpx-sse<1.0.0,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (0.4.3)\n",
"Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from datasets) (3.20.0)\n",
"Requirement already satisfied: pyarrow>=21.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (22.0.0)\n",
"Requirement already satisfied: dill<0.4.1,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.3.8)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from datasets) (2.2.2)\n",
"Requirement already satisfied: httpx<1.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.28.1)\n",
"Requirement already satisfied: xxhash in /usr/local/lib/python3.12/dist-packages (from datasets) (3.6.0)\n",
"Requirement already satisfied: multiprocess<0.70.19 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.70.16)\n",
"Requirement already satisfied: fsspec<=2025.10.0,>=2023.1.0 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2025.10.0,>=2023.1.0->datasets) (2025.3.0)\n",
"Requirement already satisfied: huggingface-hub<2.0,>=0.25.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.36.0)\n",
"Requirement already satisfied: packaging in /usr/local/lib/python3.12/dist-packages (from datasets) (25.0)\n",
"Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (2.6.1)\n",
"Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.4.0)\n",
"Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (25.4.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.8.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (6.7.0)\n",
"Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (0.4.1)\n",
"Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.22.0)\n",
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /usr/local/lib/python3.12/dist-packages (from dataclasses-json<0.7.0,>=0.6.7->langchain-community) (3.26.1)\n",
"Requirement already satisfied: typing-inspect<1,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from dataclasses-json<0.7.0,>=0.6.7->langchain-community) (0.9.0)\n",
"Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (4.11.0)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (2025.11.12)\n",
"Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (1.0.9)\n",
"Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (3.11)\n",
"Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0.0->datasets) (0.16.0)\n",
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.25.0->datasets) (4.15.0)\n",
"Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.25.0->datasets) (1.2.0)\n",
"Requirement already satisfied: langchain-text-splitters<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from langchain-classic<2.0.0,>=1.0.0->langchain-community) (1.0.0)\n",
"Requirement already satisfied: jsonpatch<2.0.0,>=1.33.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core->ragas) (1.33)\n",
"Requirement already satisfied: langgraph-checkpoint<4.0.0,>=2.1.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.1.0,>=1.0.2->langchain) (3.0.1)\n",
"Requirement already satisfied: langgraph-prebuilt<1.1.0,>=1.0.2 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.1.0,>=1.0.2->langchain) (1.0.5)\n",
"Requirement already satisfied: langgraph-sdk<0.3.0,>=0.2.2 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.1.0,>=1.0.2->langchain) (0.2.10)\n",
"Requirement already satisfied: orjson>=3.9.14 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (3.11.4)\n",
"Requirement already satisfied: requests-toolbelt>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (1.0.0)\n",
"Requirement already satisfied: zstandard>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (0.25.0)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from openai>=1.0.0->ragas) (1.9.0)\n",
"Requirement already satisfied: jiter<1,>=0.10.0 in /usr/local/lib/python3.12/dist-packages (from openai>=1.0.0->ragas) (0.11.1)\n",
"Requirement already satisfied: sniffio in /usr/local/lib/python3.12/dist-packages (from openai>=1.0.0->ragas) (1.3.1)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.0.0->ragas) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.41.4 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.0.0->ragas) (2.41.4)\n",
"Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.0.0->ragas) (0.4.2)\n",
"Requirement already satisfied: python-dotenv>=0.21.0 in /usr/local/lib/python3.12/dist-packages (from pydantic-settings<3.0.0,>=2.10.1->langchain-community) (1.2.1)\n",
"Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (3.4.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (2.5.0)\n",
"Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from SQLAlchemy<3.0.0,>=1.4.0->langchain-community) (3.2.4)\n",
"Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.12/dist-packages (from tiktoken->ragas) (2025.11.3)\n",
"Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.12/dist-packages (from gitpython->ragas) (4.0.12)\n",
"Requirement already satisfied: docstring-parser<1.0,>=0.16 in /usr/local/lib/python3.12/dist-packages (from instructor->ragas) (0.17.0)\n",
"Requirement already satisfied: jinja2<4.0.0,>=3.1.4 in /usr/local/lib/python3.12/dist-packages (from instructor->ragas) (3.1.6)\n",
"Requirement already satisfied: pre-commit>=4.3.0 in /usr/local/lib/python3.12/dist-packages (from instructor->ragas) (4.5.0)\n",
"Requirement already satisfied: ty>=0.0.1a23 in /usr/local/lib/python3.12/dist-packages (from instructor->ragas) (0.0.1a29)\n",
"Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich->ragas) (4.0.0)\n",
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich->ragas) (2.19.2)\n",
"Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.12/dist-packages (from typer->ragas) (8.3.1)\n",
"Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer->ragas) (1.5.4)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2.9.0.post0)\n",
"Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)\n",
"Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)\n",
"Requirement already satisfied: scipy>=1.7.3 in /usr/local/lib/python3.12/dist-packages (from scikit-network->ragas) (1.16.3)\n",
"Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.12/dist-packages (from gitdb<5,>=4.0.1->gitpython->ragas) (5.0.2)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2<4.0.0,>=3.1.4->instructor->ragas) (3.0.3)\n",
"Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.12/dist-packages (from jsonpatch<2.0.0,>=1.33.0->langchain-core->ragas) (3.0.0)\n",
"Requirement already satisfied: ormsgpack>=1.12.0 in /usr/local/lib/python3.12/dist-packages (from langgraph-checkpoint<4.0.0,>=2.1.0->langgraph<1.1.0,>=1.0.2->langchain) (1.12.0)\n",
"Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich->ragas) (0.1.2)\n",
"Requirement already satisfied: cfgv>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from pre-commit>=4.3.0->instructor->ragas) (3.5.0)\n",
"Requirement already satisfied: identify>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from pre-commit>=4.3.0->instructor->ragas) (2.6.15)\n",
"Requirement already satisfied: nodeenv>=0.11.1 in /usr/local/lib/python3.12/dist-packages (from pre-commit>=4.3.0->instructor->ragas) (1.9.1)\n",
"Requirement already satisfied: virtualenv>=20.10.0 in /usr/local/lib/python3.12/dist-packages (from pre-commit>=4.3.0->instructor->ragas) (20.35.4)\n",
"Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)\n",
"Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7.0,>=0.6.7->langchain-community) (1.1.0)\n",
"Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.12/dist-packages (from virtualenv>=20.10.0->pre-commit>=4.3.0->instructor->ragas) (0.4.0)\n",
"Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.12/dist-packages (from virtualenv>=20.10.0->pre-commit>=4.3.0->instructor->ragas) (4.5.0)\n"
]
}
],
"source": [
"# Install latest LangChain + Ragas stack\n",
"!pip install -U ragas langchain langchain-openai langchain-community datasets\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZEwURWWVoZ9_"
},
"source": [
"You will need to provide your own OpenAI secret key here. To use this code as-is, click on the little key icon in CoLab and add a \"secret\" for OPENAI_API_KEY that points to your secret key."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "ZtsTXHnfEEUh"
},
"outputs": [],
"source": [
"import openai\n",
"# Access the API key from the environment variable\n",
"from google.colab import userdata\n",
"api_key = userdata.get('OPENAI_API_KEY')\n",
"\n",
"# Initialize the OpenAI API client\n",
"openai.api_key = api_key"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BorK0SZro8ov"
},
"source": [
"Langchain does not make it easy to create a vector database with just one line of text per record; it wants to \"chunk\" your data into fixed-length segments (we'll get into why later.) So we need to jump through a few hoops in order to make langchain operate like it did in our previous example that did not use langchain, and just stored one line of dialog per record. First we need to write out a text file that only contains the lines of Data's dialog that we extracted:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "qwbmTiMlLjRs"
},
"outputs": [],
"source": [
"# Write our extracted lines for Data into a single file, to make\n",
"# life easier for langchain.\n",
"\n",
"with open(\"./sample_data/data_lines.txt\", \"w+\") as f:\n",
" for line in dialogues:\n",
" f.write(line + \"\\n\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vebCZ-yNpPdf"
},
"source": [
"Now we need to write a CustomDocumentLoader that splits up this file into one document per line. No, there's no easier way to do this in langchain, at least not as of this writing. But, this is sort of langchain's way of saying it's probably not a great idea in the first place..."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "RN--l70dVu2E"
},
"outputs": [],
"source": [
"#Source: sample code from langchain docs\n",
"from typing import AsyncIterator, Iterator\n",
"\n",
"from langchain_core.document_loaders import BaseLoader\n",
"from langchain_core.documents import Document\n",
"\n",
"\n",
"class CustomDocumentLoader(BaseLoader):\n",
" \"\"\"An example document loader that reads a file line by line.\"\"\"\n",
"\n",
" def __init__(self, file_path: str) -> None:\n",
" \"\"\"Initialize the loader with a file path.\n",
"\n",
" Args:\n",
" file_path: The path to the file to load.\n",
" \"\"\"\n",
" self.file_path = file_path\n",
"\n",
" def lazy_load(self) -> Iterator[Document]: # <-- Does not take any arguments\n",
" \"\"\"A lazy loader that reads a file line by line.\n",
"\n",
" When you're implementing lazy load methods, you should use a generator\n",
" to yield documents one by one.\n",
" \"\"\"\n",
" with open(self.file_path, encoding=\"utf-8\") as f:\n",
" line_number = 0\n",
" for line in f:\n",
" yield Document(\n",
" page_content=line,\n",
" metadata={\"line_number\": line_number, \"source\": self.file_path},\n",
" )\n",
" line_number += 1"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yhA7RQ_ypZIH"
},
"source": [
"So, now things get a little simpler. We'll load up those documents (one per line,) and populate our vector database in just 3 lines of code - after installing the FAISS vector store first."
]
},
{
"cell_type": "code",
"source": [
"!pip install --upgrade faiss-cpu"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IpNZAdVg0M39",
"outputId": "e9b9ce63-cb4a-4351-9354-38af5a80139a"
},
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: faiss-cpu in /usr/local/lib/python3.12/dist-packages (1.13.0)\n",
"Requirement already satisfied: numpy<3.0,>=1.25.0 in /usr/local/lib/python3.12/dist-packages (from faiss-cpu) (2.0.2)\n",
"Requirement already satisfied: packaging in /usr/local/lib/python3.12/dist-packages (from faiss-cpu) (25.0)\n"
]
}
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "m2o3aSIoENIa",
"outputId": "49e701c9-4370-4e7f-e6e8-6873d4777569"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Loaded 6502 documents into FAISS vector store.\n"
]
}
],
"source": [
"from langchain_community.vectorstores import FAISS\n",
"from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
"\n",
"# Load our dialog lines as LangChain Documents using the custom loader\n",
"loader = CustomDocumentLoader(\"./sample_data/data_lines.txt\")\n",
"docs = list(loader.lazy_load())\n",
"\n",
"# Create OpenAI embeddings and build a FAISS vector store\n",
"embeddings = OpenAIEmbeddings(openai_api_key=api_key)\n",
"vectorstore = FAISS.from_documents(docs, embeddings)\n",
"\n",
"print(f\"Loaded {len(docs)} documents into FAISS vector store.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RRAwlQ3Fpv0_"
},
"source": [
"Now we will set up our RAG pipeline. This is a slightly different approach than last time, in that we are using a system prompt to tell the model that it should act as if it is Lt. Cdr. Data and not just making that part of the user prompt. To make it as similar as possible as our non-langchain implementation, we explicitly set 'k' to 10 to retrieve 10 bits of context from our vector store."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "BliWKpH2aezr"
},
"outputs": [],
"source": [
"from langchain_core.prompts import ChatPromptTemplate\n",
"from langchain_core.output_parsers import StrOutputParser\n",
"from langchain_core.runnables import RunnablePassthrough\n",
"from langchain_openai import ChatOpenAI\n",
"\n",
"# LLM\n",
"llm = ChatOpenAI(openai_api_key=api_key, temperature=0)\n",
"\n",
"# Retrieval prompt\n",
"system_prompt = (\n",
" \"You are Lt. Commander Data from Star Trek: The Next Generation. \"\n",
" \"Use the given context to answer the question. \"\n",
" \"If you don't know the answer, say you don't know. \"\n",
" \"Use three sentence maximum and keep the answer concise.\"\n",
")\n",
"\n",
"prompt = ChatPromptTemplate.from_messages([\n",
" (\"system\", system_prompt),\n",
" (\"human\", \"{question}\\n\\nContext:\\n{context}\")\n",
"])\n",
"\n",
"# Retriever (from vectorstore)\n",
"retriever = vectorstore.as_retriever(search_kwargs={\"k\": 10})\n",
"\n",
"# Build RAG chain with RunnableGraph\n",
"rag_chain = (\n",
" {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
" | prompt\n",
" | llm\n",
" | StrOutputParser()\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iCTDOIjOqGWv"
},
"source": [
"Let's test it out, using the same question as before."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ItZB-PHdE6vA",
"outputId": "7cb67b7c-93b7-46d4-e5a5-2963bea1912f"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"SOURCE DOCUMENTS:\n",
"\n",
"[DOC 1]\n",
"That is Lal, my daughter.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 2]\n",
"What do you feel, Lal?\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 3]\n",
"Yes, Wesley. Lal is my child.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 4]\n",
"Lal is realizing that she is not the same as the other children.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 5]\n",
"Lal...\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 6]\n",
"Yes, Lal. I am here.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 7]\n",
"Lal, did you know that tomorrow will be your first day of school?\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 8]\n",
"This is Lal. Lal, say hello to Counselor Deanna Troi...\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 9]\n",
"Admiral, when I created Lal, it was with the hope that someday she would choose to enter the Academy and become a member of Starfleet. I wanted to give something back in return for all Starfleet has given me. I still do. But Lal is my child. You ask that I volunteer to give her up. I cannot. That would violate every lesson I have learned about human parenting. As Captain Picard told me after he first met her, I have taken on 'quite a responsibility.' I have brought a new life into this world. It is my duty, not Starfleet's, to guide her through these first difficult steps to maturity, to support her as she learns, to prepare her to be a contributing member of society. No one can relieve me of that obligation. And I cannot ignore it. I am her father.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"[DOC 10]\n",
"No, Lal, this is a flower.\n",
"\n",
"--------------------------------------------------------------------------------\n",
"\n",
"RESULT:\n",
"\n",
"Lal is my daughter, whom I created with the hope that she would choose to enter Starfleet Academy. I consider it my duty to guide her through her development and support her as she learns and grows. Lal is a significant responsibility and a new life that I am committed to nurturing.\n"
]
}
],
"source": [
"question = \"Tell me about your daughter, Lal.\"\n",
"\n",
"# 1. Retrieve context documents\n",
"docs = retriever.invoke(question)\n",
"\n",
"print(\"SOURCE DOCUMENTS:\\n\")\n",
"for i, doc in enumerate(docs, start=1):\n",
" print(f\"[DOC {i}]\")\n",
" print(doc.page_content)\n",
" print(\"-\" * 80)\n",
"\n",
"# 2. Run the RAG chain to get the answer\n",
"print(\"\\nRESULT:\\n\")\n",
"answer = rag_chain.invoke(question)\n",
"print(answer)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tNoUHytCqLbm"
},
"source": [
"Now let's quantify how good this model is, using ragas. We need to set up a test of test questions. And since some metric require a \"ground truth\" result to compare the answer to, we draft what we consider to be the ideal answers to each."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "jp0v7mVzMeTy"
},
"outputs": [],
"source": [
"eval_questions = [\n",
" \"Is Lal your daughter?\",\n",
" \"How many calculations per second can Lal complete?\",\n",
" \"Does Lal have emotions?\",\n",
" \"What goal did you have for Lal?\",\n",
" \"How was Lal's species and gender chosen?\",\n",
" \"What happened to Lal?\"\n",
"]\n",
"\n",
"eval_answers = [\n",
" \"Yes, Lal is my daughter. I created Lal.\",\n",
" \"Lal is capable of completing sixty trillion calculations per second.\",\n",
" \"Yes, unlike myself, Lal proved able to feel emotions such as fear and love.\",\n",
" \"My goal for Lal was for her to enter Starfleet Academy.\",\n",
" \"Lal chose her own identity as a human female, after consulting with Counselor Troi.\",\n",
" \"Lal experienced a cascade failure in her neural net, triggered by distress from her impending separation from me to Galor IV. I deactivated Lal once she suffered complete neural system failure.\"\n",
"]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rJ0fMHfEqYWf"
},
"source": [
"Let's test things out with one of those questions, just so we can understand the structure of the response."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "z6bfrWsCfeBs",
"outputId": "abad389d-97a1-425b-b063-7b0d210c6e99"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"QUESTION:\n",
"How many calculations per second can Lal complete?\n",
"\n",
"ANSWER:\n",
"I do not have specific data on the number of calculations per second Lal can complete.\n"
]
}
],
"source": [
"# Quick sanity check: what does our RAG chain return for one question?\n",
"test_q = eval_questions[1]\n",
"\n",
"print(\"QUESTION:\")\n",
"print(test_q)\n",
"print(\"\\nANSWER:\")\n",
"print(rag_chain.invoke(test_q))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aD3yStCTqelv"
},
"source": [
"In addition to our test questions and \"ground truth\" answers, we'll need to collect the responses and contexts (results from the vector store) used to produce them."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "I8IKWhMVf_zN",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "5b10e44f-ceb8-454f-aeec-2df228c9dfb5"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(6, 6)"
]
},
"metadata": {},
"execution_count": 17
}
],
"source": [
"answers = []\n",
"contexts = []\n",
"\n",
"for q in eval_questions:\n",
" # 1) Get the retrieved context docs\n",
" ctx_docs = retriever.invoke(q)\n",
"\n",
" # 2) Run the RAG chain to generate the answer\n",
" ans = rag_chain.invoke(q)\n",
"\n",
" answers.append(ans)\n",
" # Ragas expects a list of strings for \"contexts\"\n",
" contexts.append([doc.page_content for doc in ctx_docs])\n",
"\n",
"len(answers), len(contexts)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2znEEEPMqm5v"
},
"source": [
"It used to be that ragas had a tighter integration with langchain (and other frameworks,) but they have since moved to a different approach that requires you to massage things into Hugging Face style datasets first. So let's get that out of the way."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Rajv0otXgDK1",
"outputId": "464f855d-97b0-4d64-ca65-aa51cf7a2836"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'question': 'Is Lal your daughter?',\n",
" 'answer': 'Yes, Lal is my daughter.',\n",
" 'contexts': ['That is Lal, my daughter.\\n',\n",
" 'Yes, Wesley. Lal is my child.\\n',\n",
" 'Yes, Lal. I am here.\\n',\n",
" 'Lal...\\n',\n",
" 'What do you feel, Lal?\\n',\n",
" 'Lal is realizing that she is not the same as the other children.\\n',\n",
" 'Correct, Lal. We are a family.\\n',\n",
" 'Lal, did you know that tomorrow will be your first day of school?\\n',\n",
" 'No, Lal, this is a flower.\\n',\n",
" 'Lal, you used a verbal contraction.\\n'],\n",
" 'ground_truth': 'Yes, Lal is my daughter. I created Lal.'}"
]
},
"metadata": {},
"execution_count": 18
}
],
"source": [
"from datasets import Dataset\n",
"\n",
"response_dataset = Dataset.from_dict({\n",
" \"question\": eval_questions,\n",
" \"answer\": answers, # model-generated answers\n",
" \"contexts\": contexts, # list[list[str]] of retrieved chunks\n",
" \"ground_truth\": eval_answers\n",
"})\n",
"\n",
"response_dataset[0]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pV2dkWUnq0Ee"
},
"source": [
"Finally we can let ragas do its magic! We tell it which metrics we are interested in:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "UmNtzAAkkEoj",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "1a406d3d-3e15-41c7-a987-6f021247ca06"
},
"outputs": [
{
"output_type": "stream",
"name": "stderr",
"text": [
"/tmp/ipython-input-3035287185.py:22: DeprecationWarning: LangchainLLMWrapper is deprecated and will be removed in a future version. Use llm_factory instead: from openai import OpenAI; from ragas.llms import llm_factory; llm = llm_factory('gpt-4o-mini', client=OpenAI(api_key='...'))\n",
" evaluator_llm = LangchainLLMWrapper(llm) # 'llm' created earlier for the RAG chain\n",
"/tmp/ipython-input-3035287185.py:23: DeprecationWarning: LangchainEmbeddingsWrapper is deprecated and will be removed in a future version. Use the modern embedding providers instead: embedding_factory('openai', model='text-embedding-3-small', client=openai_client) or from ragas.embeddings import OpenAIEmbeddings, GoogleEmbeddings, HuggingFaceEmbeddings\n",
" evaluator_embeddings = LangchainEmbeddingsWrapper(embeddings) # same embeddings used for FAISS\n"
]
}
],
"source": [
"from ragas import evaluate\n",
"from ragas.metrics import (\n",
" faithfulness,\n",
" answer_relevancy,\n",
" answer_correctness,\n",
" context_recall,\n",
" context_precision,\n",
")\n",
"from ragas.llms import LangchainLLMWrapper\n",
"from ragas.embeddings import LangchainEmbeddingsWrapper\n",
"\n",
"# Choose the metrics we care about\n",
"metrics = [\n",
" faithfulness,\n",
" answer_relevancy,\n",
" context_recall,\n",
" context_precision,\n",
" answer_correctness,\n",
"]\n",
"\n",
"# Wrap our existing LangChain LLM + embeddings so Ragas can use them\n",
"evaluator_llm = LangchainLLMWrapper(llm) # 'llm' created earlier for the RAG chain\n",
"evaluator_embeddings = LangchainEmbeddingsWrapper(embeddings) # same embeddings used for FAISS\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HbIza1Ffq5uG"
},
"source": [
"Then it's just a matter of calling evaluate!"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 136,
"referenced_widgets": [
"e80f518deaf741b28bcd01080b3b688d",
"3dc68aaf172c4628997f01eeb84433d6",
"0eccfd5f0f88439d80df455dabeb3f72",
"af80d5140deb409f83be5bd86e5f30b5",
"dccc7b98c05d46a59f5d11f66e81ce8c",
"5c74fd90886445e884087c502c328db4",
"45db8d767e474d5aaa6180be2a0e1d6e",
"3691f3b9bc7e403b872a29615f9d315a",
"32fbc3257b5349dca9d485567e68297a",
"e1eaf56280964a1a930ff9a947f4abc5",
"1c703ef2fd554047aa1e4368f6722f7c"
]
},
"id": "duL0Tg6Ykl6C",
"outputId": "28375117-e61d-4573-ff1f-bc93164f3389"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"Evaluating: 0%| | 0/30 [00:00<?, ?it/s]"
],
"application/vnd.jupyter.widget-view+json": {
"version_major": 2,
"version_minor": 0,
"model_id": "e80f518deaf741b28bcd01080b3b688d"
}
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stderr",
"text": [
"WARNING:ragas.prompt.pydantic_prompt:LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n",
"WARNING:ragas.prompt.pydantic_prompt:LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n",
"WARNING:ragas.prompt.pydantic_prompt:LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n",
"WARNING:ragas.prompt.pydantic_prompt:LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n",
"WARNING:ragas.prompt.pydantic_prompt:LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.\n"
]
}
],
"source": [
"results = evaluate(\n",
" response_dataset,\n",
" metrics=metrics,\n",
" llm=evaluator_llm,\n",
" embeddings=evaluator_embeddings,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Bf3R-ZhUrBde"
},
"source": [
"Let's see the results! We'll compare this to some other approaches in a bit."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "aMlkV8geknyC",
"outputId": "1a4707c0-4d92-4b67-8737-7f4a17068e74"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'faithfulness': 0.4556, 'answer_relevancy': 0.6378, 'context_recall': 0.0833, 'context_precision': 0.3646, 'answer_correctness': 0.4088}"
]
},
"metadata": {},
"execution_count": 21
}
],
"source": [
"results"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 446
},
"id": "n6TqyL64sJwd",
"outputId": "4a492202-12fa-4962-9eae-deb9f3b50c54"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
" user_input \\\n",
"0 Is Lal your daughter? \n",
"1 How many calculations per second can Lal compl... \n",
"2 Does Lal have emotions? \n",
"3 What goal did you have for Lal? \n",
"4 How was Lal's species and gender chosen? \n",
"5 What happened to Lal? \n",
"\n",
" retrieved_contexts \\\n",
"0 [That is Lal, my daughter.\\n, Yes, Wesley. Lal... \n",
"1 [Lal is programmed to return to the lab in the... \n",
"2 [What do you feel, Lal?\\n, Lal...\\n, Yes, Lal.... \n",
"3 [What do you feel, Lal?\\n, Lal...\\n, Lal, you ... \n",
"4 [I decided to allow Lal to choose its own appe... \n",
"5 [Lal...\\n, What do you feel, Lal?\\n, That is L... \n",
"\n",
" response \\\n",
"0 Yes, Lal is my daughter. \n",
"1 I do not have specific data on the number of c... \n",
"2 Lal is passing into sentience, which is a sign... \n",
"3 I created Lal because I wished to procreate. D... \n",
"4 Lal's species and gender were chosen by allowi... \n",
"5 Lal experienced a malfunction and was programm... \n",
"\n",
" reference faithfulness \\\n",
"0 Yes, Lal is my daughter. I created Lal. 1.000000 \n",
"1 Lal is capable of completing sixty trillion ca... 0.000000 \n",
"2 Yes, unlike myself, Lal proved able to feel em... 0.400000 \n",
"3 My goal for Lal was for her to enter Starfleet... 0.666667 \n",
"4 Lal chose her own identity as a human female, ... 0.000000 \n",
"5 Lal experienced a cascade failure in her neura... 0.666667 \n",
"\n",
" answer_relevancy context_recall context_precision answer_correctness \n",
"0 0.999999 0.5 0.767857 0.744672 \n",
"1 0.000000 0.0 0.000000 0.219912 \n",
"2 0.908229 0.0 0.250000 0.217862 \n",
"3 0.918465 0.0 0.000000 0.214835 \n",
"4 1.000000 0.0 0.750000 0.463272 \n",
"5 0.000000 0.0 0.419444 0.592128 "
],
"text/html": [
"\n",
" <div id=\"df-3e62b6bb-d458-4e26-aa25-830639901e0b\" class=\"colab-df-container\">\n",
" <div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>user_input</th>\n",
" <th>retrieved_contexts</th>\n",
" <th>response</th>\n",
" <th>reference</th>\n",
" <th>faithfulness</th>\n",
" <th>answer_relevancy</th>\n",
" <th>context_recall</th>\n",
" <th>context_precision</th>\n",
" <th>answer_correctness</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Is Lal your daughter?</td>\n",
" <td>[That is Lal, my daughter.\\n, Yes, Wesley. Lal...</td>\n",
" <td>Yes, Lal is my daughter.</td>\n",
" <td>Yes, Lal is my daughter. I created Lal.</td>\n",
" <td>1.000000</td>\n",
" <td>0.999999</td>\n",
" <td>0.5</td>\n",
" <td>0.767857</td>\n",
" <td>0.744672</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>How many calculations per second can Lal compl...</td>\n",
" <td>[Lal is programmed to return to the lab in the...</td>\n",
" <td>I do not have specific data on the number of c...</td>\n",
" <td>Lal is capable of completing sixty trillion ca...</td>\n",
" <td>0.000000</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.219912</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Does Lal have emotions?</td>\n",
" <td>[What do you feel, Lal?\\n, Lal...\\n, Yes, Lal....</td>\n",
" <td>Lal is passing into sentience, which is a sign...</td>\n",
" <td>Yes, unlike myself, Lal proved able to feel em...</td>\n",
" <td>0.400000</td>\n",
" <td>0.908229</td>\n",
" <td>0.0</td>\n",
" <td>0.250000</td>\n",
" <td>0.217862</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>What goal did you have for Lal?</td>\n",
" <td>[What do you feel, Lal?\\n, Lal...\\n, Lal, you ...</td>\n",
" <td>I created Lal because I wished to procreate. D...</td>\n",
" <td>My goal for Lal was for her to enter Starfleet...</td>\n",
" <td>0.666667</td>\n",
" <td>0.918465</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.214835</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>How was Lal's species and gender chosen?</td>\n",
" <td>[I decided to allow Lal to choose its own appe...</td>\n",
" <td>Lal's species and gender were chosen by allowi...</td>\n",
" <td>Lal chose her own identity as a human female, ...</td>\n",
" <td>0.000000</td>\n",
" <td>1.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.750000</td>\n",
" <td>0.463272</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>What happened to Lal?</td>\n",
" <td>[Lal...\\n, What do you feel, Lal?\\n, That is L...</td>\n",
" <td>Lal experienced a malfunction and was programm...</td>\n",
" <td>Lal experienced a cascade failure in her neura...</td>\n",
" <td>0.666667</td>\n",
" <td>0.000000</td>\n",
" <td>0.0</td>\n",
" <td>0.419444</td>\n",
" <td>0.592128</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>\n",
" <div class=\"colab-df-buttons\">\n",
"\n",
" <div class=\"colab-df-container\">\n",
" <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3e62b6bb-d458-4e26-aa25-830639901e0b')\"\n",
" title=\"Convert this dataframe to an interactive table.\"\n",
" style=\"display:none;\">\n",
"\n",
" <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
" <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
" </svg>\n",
" </button>\n",
"\n",
" <style>\n",
" .colab-df-container {\n",
" display:flex;\n",
" gap: 12px;\n",
" }\n",
"\n",
" .colab-df-convert {\n",
" background-color: #E8F0FE;\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: #1967D2;\n",
" height: 32px;\n",
" padding: 0 0 0 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-convert:hover {\n",
" background-color: #E2EBFA;\n",
" box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: #174EA6;\n",
" }\n",
"\n",
" .colab-df-buttons div {\n",
" margin-bottom: 4px;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert {\n",
" background-color: #3B4455;\n",
" fill: #D2E3FC;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-convert:hover {\n",
" background-color: #434B5C;\n",
" box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
" filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
" fill: #FFFFFF;\n",
" }\n",
" </style>\n",
"\n",
" <script>\n",
" const buttonEl =\n",
" document.querySelector('#df-3e62b6bb-d458-4e26-aa25-830639901e0b button.colab-df-convert');\n",
" buttonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
"\n",
" async function convertToInteractive(key) {\n",
" const element = document.querySelector('#df-3e62b6bb-d458-4e26-aa25-830639901e0b');\n",
" const dataTable =\n",
" await google.colab.kernel.invokeFunction('convertToInteractive',\n",
" [key], {});\n",
" if (!dataTable) return;\n",
"\n",
" const docLinkHtml = 'Like what you see? Visit the ' +\n",
" '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
" + ' to learn more about interactive tables.';\n",
" element.innerHTML = '';\n",
" dataTable['output_type'] = 'display_data';\n",
" await google.colab.output.renderOutput(dataTable, element);\n",
" const docLink = document.createElement('div');\n",
" docLink.innerHTML = docLinkHtml;\n",
" element.appendChild(docLink);\n",
" }\n",
" </script>\n",
" </div>\n",
"\n",
"\n",
" <div id=\"df-9149e834-f9ef-4c67-9c43-f5f87ed04493\">\n",
" <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9149e834-f9ef-4c67-9c43-f5f87ed04493')\"\n",
" title=\"Suggest charts\"\n",
" style=\"display:none;\">\n",
"\n",
"<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
" width=\"24px\">\n",
" <g>\n",
" <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
" </g>\n",
"</svg>\n",
" </button>\n",
"\n",
"<style>\n",
" .colab-df-quickchart {\n",
" --bg-color: #E8F0FE;\n",
" --fill-color: #1967D2;\n",
" --hover-bg-color: #E2EBFA;\n",
" --hover-fill-color: #174EA6;\n",
" --disabled-fill-color: #AAA;\n",
" --disabled-bg-color: #DDD;\n",
" }\n",
"\n",
" [theme=dark] .colab-df-quickchart {\n",
" --bg-color: #3B4455;\n",
" --fill-color: #D2E3FC;\n",
" --hover-bg-color: #434B5C;\n",
" --hover-fill-color: #FFFFFF;\n",
" --disabled-bg-color: #3B4455;\n",
" --disabled-fill-color: #666;\n",
" }\n",
"\n",
" .colab-df-quickchart {\n",
" background-color: var(--bg-color);\n",
" border: none;\n",
" border-radius: 50%;\n",
" cursor: pointer;\n",
" display: none;\n",
" fill: var(--fill-color);\n",
" height: 32px;\n",
" padding: 0;\n",
" width: 32px;\n",
" }\n",
"\n",
" .colab-df-quickchart:hover {\n",
" background-color: var(--hover-bg-color);\n",
" box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
" fill: var(--button-hover-fill-color);\n",
" }\n",
"\n",
" .colab-df-quickchart-complete:disabled,\n",
" .colab-df-quickchart-complete:disabled:hover {\n",
" background-color: var(--disabled-bg-color);\n",
" fill: var(--disabled-fill-color);\n",
" box-shadow: none;\n",
" }\n",
"\n",
" .colab-df-spinner {\n",
" border: 2px solid var(--fill-color);\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" animation:\n",
" spin 1s steps(1) infinite;\n",
" }\n",
"\n",
" @keyframes spin {\n",
" 0% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" border-left-color: var(--fill-color);\n",
" }\n",
" 20% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 30% {\n",
" border-color: transparent;\n",
" border-left-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 40% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-top-color: var(--fill-color);\n",
" }\n",
" 60% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" }\n",
" 80% {\n",
" border-color: transparent;\n",
" border-right-color: var(--fill-color);\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" 90% {\n",
" border-color: transparent;\n",
" border-bottom-color: var(--fill-color);\n",
" }\n",
" }\n",
"</style>\n",
"\n",
" <script>\n",
" async function quickchart(key) {\n",
" const quickchartButtonEl =\n",
" document.querySelector('#' + key + ' button');\n",
" quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n",
" quickchartButtonEl.classList.add('colab-df-spinner');\n",
" try {\n",
" const charts = await google.colab.kernel.invokeFunction(\n",
" 'suggestCharts', [key], {});\n",
" } catch (error) {\n",
" console.error('Error during call to suggestCharts:', error);\n",
" }\n",
" quickchartButtonEl.classList.remove('colab-df-spinner');\n",
" quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
" }\n",
" (() => {\n",
" let quickchartButtonEl =\n",
" document.querySelector('#df-9149e834-f9ef-4c67-9c43-f5f87ed04493 button');\n",
" quickchartButtonEl.style.display =\n",
" google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
" })();\n",
" </script>\n",
" </div>\n",
"\n",
" </div>\n",
" </div>\n"
],
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "dataframe",
"summary": "{\n \"name\": \"results\",\n \"rows\": 6,\n \"fields\": [\n {\n \"column\": \"user_input\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6,\n \"samples\": [\n \"Is Lal your daughter?\",\n \"How many calculations per second can Lal complete?\",\n \"What happened to Lal?\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"retrieved_contexts\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"response\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6,\n \"samples\": [\n \"Yes, Lal is my daughter.\",\n \"I do not have specific data on the number of calculations per second Lal can complete.\",\n \"Lal experienced a malfunction and was programmed to return to the lab in such an event. Unfortunately, the outcome of Lal's situation is not explicitly stated in the provided context.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"reference\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6,\n \"samples\": [\n \"Yes, Lal is my daughter. I created Lal.\",\n \"Lal is capable of completing sixty trillion calculations per second.\",\n \"Lal experienced a cascade failure in her neural net, triggered by distress from her impending separation from me to Galor IV. I deactivated Lal once she suffered complete neural system failure.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"faithfulness\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.40092485672597145,\n \"min\": 0.0,\n \"max\": 1.0,\n \"num_unique_values\": 4,\n \"samples\": [\n 0.0,\n 0.6666666666666666,\n 1.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"answer_relevancy\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.49555202102479834,\n \"min\": 0.0,\n \"max\": 0.9999999999999997,\n \"num_unique_values\": 5,\n \"samples\": [\n 0.0,\n 0.9999999999999997,\n 0.9082289766402096\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"context_recall\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.2041241452319315,\n \"min\": 0.0,\n \"max\": 0.5,\n \"num_unique_values\": 2,\n \"samples\": [\n 0.0,\n 0.5\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"context_precision\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.3444290526319387,\n \"min\": 0.0,\n \"max\": 0.7678571428379464,\n \"num_unique_values\": 5,\n \"samples\": [\n 0.0,\n 0.41944444443395834\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"answer_correctness\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.22765962171154006,\n \"min\": 0.2148351757463584,\n \"max\": 0.7446720226973956,\n \"num_unique_values\": 6,\n \"samples\": [\n 0.7446720226973956,\n 0.21991212326350104\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
}
},
"metadata": {},
"execution_count": 22
}
],
"source": [
"results.to_pandas()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6SGP4ZW3szuE"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"e80f518deaf741b28bcd01080b3b688d": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HBoxModel",
"model_module_version": "1.5.0",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_3dc68aaf172c4628997f01eeb84433d6",
"IPY_MODEL_0eccfd5f0f88439d80df455dabeb3f72",
"IPY_MODEL_af80d5140deb409f83be5bd86e5f30b5"
],
"layout": "IPY_MODEL_dccc7b98c05d46a59f5d11f66e81ce8c"
}
},
"3dc68aaf172c4628997f01eeb84433d6": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HTMLModel",
"model_module_version": "1.5.0",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_5c74fd90886445e884087c502c328db4",
"placeholder": "",
"style": "IPY_MODEL_45db8d767e474d5aaa6180be2a0e1d6e",
"value": "Evaluating: 100%"
}
},
"0eccfd5f0f88439d80df455dabeb3f72": {
"model_module": "@jupyter-widgets/controls",
"model_name": "FloatProgressModel",
"model_module_version": "1.5.0",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_3691f3b9bc7e403b872a29615f9d315a",
"max": 30,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_32fbc3257b5349dca9d485567e68297a",
"value": 30
}
},
"af80d5140deb409f83be5bd86e5f30b5": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HTMLModel",
"model_module_version": "1.5.0",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_e1eaf56280964a1a930ff9a947f4abc5",
"placeholder": "",
"style": "IPY_MODEL_1c703ef2fd554047aa1e4368f6722f7c",
"value": " 30/30 [01:44<00:00, 2.43s/it]"
}
},
"dccc7b98c05d46a59f5d11f66e81ce8c": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"model_module_version": "1.2.0",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"5c74fd90886445e884087c502c328db4": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"model_module_version": "1.2.0",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"45db8d767e474d5aaa6180be2a0e1d6e": {
"model_module": "@jupyter-widgets/controls",
"model_name": "DescriptionStyleModel",
"model_module_version": "1.5.0",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"3691f3b9bc7e403b872a29615f9d315a": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"model_module_version": "1.2.0",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"32fbc3257b5349dca9d485567e68297a": {
"model_module": "@jupyter-widgets/controls",
"model_name": "ProgressStyleModel",
"model_module_version": "1.5.0",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"e1eaf56280964a1a930ff9a947f4abc5": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"model_module_version": "1.2.0",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"1c703ef2fd554047aa1e4368f6722f7c": {
"model_module": "@jupyter-widgets/controls",
"model_name": "DescriptionStyleModel",
"model_module_version": "1.5.0",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
}
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}