{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Item-Based Collaborative Filtering"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As before, we'll start by importing the MovieLens 100K data set into a pandas DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>movie_id</th>\n",
" <th>title</th>\n",
" <th>user_id</th>\n",
" <th>rating</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>Toy Story (1995)</td>\n",
" <td>308</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>Toy Story (1995)</td>\n",
" <td>287</td>\n",
" <td>5</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1</td>\n",
" <td>Toy Story (1995)</td>\n",
" <td>148</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1</td>\n",
" <td>Toy Story (1995)</td>\n",
" <td>280</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1</td>\n",
" <td>Toy Story (1995)</td>\n",
" <td>66</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" movie_id title user_id rating\n",
"0 1 Toy Story (1995) 308 4\n",
"1 1 Toy Story (1995) 287 5\n",
"2 1 Toy Story (1995) 148 4\n",
"3 1 Toy Story (1995) 280 4\n",
"4 1 Toy Story (1995) 66 3"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"r_cols = ['user_id', 'movie_id', 'rating']\n",
"ratings = pd.read_csv('ml-100k/u.data', sep='\\t', names=r_cols, usecols=range(3), encoding=\"ISO-8859-1\")\n",
"\n",
"m_cols = ['movie_id', 'title']\n",
"movies = pd.read_csv('ml-100k/u.item', sep='|', names=m_cols, usecols=range(2), encoding=\"ISO-8859-1\")\n",
"\n",
"ratings = pd.merge(movies, ratings)\n",
"\n",
"ratings.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we'll pivot this table to construct a nice matrix of users and the movies they rated. NaN indicates missing data, or movies that a given user did not watch:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>title</th>\n",
" <th>'Til There Was You (1997)</th>\n",
" <th>1-900 (1994)</th>\n",
" <th>101 Dalmatians (1996)</th>\n",
" <th>12 Angry Men (1957)</th>\n",
" <th>187 (1997)</th>\n",
" <th>2 Days in the Valley (1996)</th>\n",
" <th>20,000 Leagues Under the Sea (1954)</th>\n",
" <th>2001: A Space Odyssey (1968)</th>\n",
" <th>3 Ninjas: High Noon At Mega Mountain (1998)</th>\n",
" <th>39 Steps, The (1935)</th>\n",
" <th>...</th>\n",
" <th>Yankee Zulu (1994)</th>\n",
" <th>Year of the Horse (1997)</th>\n",
" <th>You So Crazy (1994)</th>\n",
" <th>Young Frankenstein (1974)</th>\n",
" <th>Young Guns (1988)</th>\n",
" <th>Young Guns II (1990)</th>\n",
" <th>Young Poisoner's Handbook, The (1995)</th>\n",
" <th>Zeus and Roxanne (1997)</th>\n",
" <th>unknown</th>\n",
" <th>Á köldum klaka (Cold Fever) (1994)</th>\n",
" </tr>\n",
" <tr>\n",
" <th>user_id</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2.0</td>\n",
" <td>5.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>3.0</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>5.0</td>\n",
" <td>3.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>4.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 1664 columns</p>\n",
"</div>"
],
"text/plain": [
"title 'Til There Was You (1997) 1-900 (1994) 101 Dalmatians (1996) \\\n",
"user_id \n",
"0 NaN NaN NaN \n",
"1 NaN NaN 2.0 \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"\n",
"title 12 Angry Men (1957) 187 (1997) 2 Days in the Valley (1996) \\\n",
"user_id \n",
"0 NaN NaN NaN \n",
"1 5.0 NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN 2.0 NaN \n",
"4 NaN NaN NaN \n",
"\n",
"title 20,000 Leagues Under the Sea (1954) 2001: A Space Odyssey (1968) \\\n",
"user_id \n",
"0 NaN NaN \n",
"1 3.0 4.0 \n",
"2 NaN NaN \n",
"3 NaN NaN \n",
"4 NaN NaN \n",
"\n",
"title 3 Ninjas: High Noon At Mega Mountain (1998) 39 Steps, The (1935) \\\n",
"user_id \n",
"0 NaN NaN \n",
"1 NaN NaN \n",
"2 1.0 NaN \n",
"3 NaN NaN \n",
"4 NaN NaN \n",
"\n",
"title ... Yankee Zulu (1994) Year of the Horse (1997) \\\n",
"user_id ... \n",
"0 ... NaN NaN \n",
"1 ... NaN NaN \n",
"2 ... NaN NaN \n",
"3 ... NaN NaN \n",
"4 ... NaN NaN \n",
"\n",
"title You So Crazy (1994) Young Frankenstein (1974) Young Guns (1988) \\\n",
"user_id \n",
"0 NaN NaN NaN \n",
"1 NaN 5.0 3.0 \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"\n",
"title Young Guns II (1990) Young Poisoner's Handbook, The (1995) \\\n",
"user_id \n",
"0 NaN NaN \n",
"1 NaN NaN \n",
"2 NaN NaN \n",
"3 NaN NaN \n",
"4 NaN NaN \n",
"\n",
"title Zeus and Roxanne (1997) unknown Á köldum klaka (Cold Fever) (1994) \n",
"user_id \n",
"0 NaN NaN NaN \n",
"1 NaN 4.0 NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"\n",
"[5 rows x 1664 columns]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"userRatings = ratings.pivot_table(index=['user_id'],columns=['title'],values='rating')\n",
"userRatings.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the magic happens - pandas has a built-in corr() method that will compute a correlation score for every column pair in the matrix! This gives us a correlation score between every pair of movies (where at least one user rated both movies - otherwise NaN's will show up.) That's amazing!"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>title</th>\n",
" <th>'Til There Was You (1997)</th>\n",
" <th>1-900 (1994)</th>\n",
" <th>101 Dalmatians (1996)</th>\n",
" <th>12 Angry Men (1957)</th>\n",
" <th>187 (1997)</th>\n",
" <th>2 Days in the Valley (1996)</th>\n",
" <th>20,000 Leagues Under the Sea (1954)</th>\n",
" <th>2001: A Space Odyssey (1968)</th>\n",
" <th>3 Ninjas: High Noon At Mega Mountain (1998)</th>\n",
" <th>39 Steps, The (1935)</th>\n",
" <th>...</th>\n",
" <th>Yankee Zulu (1994)</th>\n",
" <th>Year of the Horse (1997)</th>\n",
" <th>You So Crazy (1994)</th>\n",
" <th>Young Frankenstein (1974)</th>\n",
" <th>Young Guns (1988)</th>\n",
" <th>Young Guns II (1990)</th>\n",
" <th>Young Poisoner's Handbook, The (1995)</th>\n",
" <th>Zeus and Roxanne (1997)</th>\n",
" <th>unknown</th>\n",
" <th>Á köldum klaka (Cold Fever) (1994)</th>\n",
" </tr>\n",
" <tr>\n",
" <th>title</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>'Til There Was You (1997)</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>-1.000000</td>\n",
" <td>-0.500000</td>\n",
" <td>-0.500000</td>\n",
" <td>0.522233</td>\n",
" <td>NaN</td>\n",
" <td>-0.426401</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1-900 (1994)</th>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>-0.981981</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>-0.944911</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>101 Dalmatians (1996)</th>\n",
" <td>-1.0</td>\n",
" <td>NaN</td>\n",
" <td>1.000000</td>\n",
" <td>-0.049890</td>\n",
" <td>0.269191</td>\n",
" <td>0.048973</td>\n",
" <td>0.266928</td>\n",
" <td>-0.043407</td>\n",
" <td>NaN</td>\n",
" <td>0.111111</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>-1.000000</td>\n",
" <td>NaN</td>\n",
" <td>0.158840</td>\n",
" <td>0.119234</td>\n",
" <td>0.680414</td>\n",
" <td>-4.875600e-17</td>\n",
" <td>0.707107</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12 Angry Men (1957)</th>\n",
" <td>-0.5</td>\n",
" <td>NaN</td>\n",
" <td>-0.049890</td>\n",
" <td>1.000000</td>\n",
" <td>0.666667</td>\n",
" <td>0.256625</td>\n",
" <td>0.274772</td>\n",
" <td>0.178848</td>\n",
" <td>NaN</td>\n",
" <td>0.457176</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>0.096546</td>\n",
" <td>0.068944</td>\n",
" <td>-0.361961</td>\n",
" <td>1.443376e-01</td>\n",
" <td>1.000000</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>187 (1997)</th>\n",
" <td>-0.5</td>\n",
" <td>NaN</td>\n",
" <td>0.269191</td>\n",
" <td>0.666667</td>\n",
" <td>1.000000</td>\n",
" <td>0.596644</td>\n",
" <td>NaN</td>\n",
" <td>-0.554700</td>\n",
" <td>NaN</td>\n",
" <td>1.000000</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0.866025</td>\n",
" <td>NaN</td>\n",
" <td>0.455233</td>\n",
" <td>-0.500000</td>\n",
" <td>0.500000</td>\n",
" <td>4.753271e-01</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 1664 columns</p>\n",
"</div>"
],
"text/plain": [
"title 'Til There Was You (1997) 1-900 (1994) \\\n",
"title \n",
"'Til There Was You (1997) 1.0 NaN \n",
"1-900 (1994) NaN 1.0 \n",
"101 Dalmatians (1996) -1.0 NaN \n",
"12 Angry Men (1957) -0.5 NaN \n",
"187 (1997) -0.5 NaN \n",
"\n",
"title 101 Dalmatians (1996) 12 Angry Men (1957) \\\n",
"title \n",
"'Til There Was You (1997) -1.000000 -0.500000 \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) 1.000000 -0.049890 \n",
"12 Angry Men (1957) -0.049890 1.000000 \n",
"187 (1997) 0.269191 0.666667 \n",
"\n",
"title 187 (1997) 2 Days in the Valley (1996) \\\n",
"title \n",
"'Til There Was You (1997) -0.500000 0.522233 \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) 0.269191 0.048973 \n",
"12 Angry Men (1957) 0.666667 0.256625 \n",
"187 (1997) 1.000000 0.596644 \n",
"\n",
"title 20,000 Leagues Under the Sea (1954) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) 0.266928 \n",
"12 Angry Men (1957) 0.274772 \n",
"187 (1997) NaN \n",
"\n",
"title 2001: A Space Odyssey (1968) \\\n",
"title \n",
"'Til There Was You (1997) -0.426401 \n",
"1-900 (1994) -0.981981 \n",
"101 Dalmatians (1996) -0.043407 \n",
"12 Angry Men (1957) 0.178848 \n",
"187 (1997) -0.554700 \n",
"\n",
"title 3 Ninjas: High Noon At Mega Mountain (1998) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title 39 Steps, The (1935) ... Yankee Zulu (1994) \\\n",
"title ... \n",
"'Til There Was You (1997) NaN ... NaN \n",
"1-900 (1994) NaN ... NaN \n",
"101 Dalmatians (1996) 0.111111 ... NaN \n",
"12 Angry Men (1957) 0.457176 ... NaN \n",
"187 (1997) 1.000000 ... NaN \n",
"\n",
"title Year of the Horse (1997) You So Crazy (1994) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) -1.000000 NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) 0.866025 NaN \n",
"\n",
"title Young Frankenstein (1974) Young Guns (1988) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) -0.944911 NaN \n",
"101 Dalmatians (1996) 0.158840 0.119234 \n",
"12 Angry Men (1957) 0.096546 0.068944 \n",
"187 (1997) 0.455233 -0.500000 \n",
"\n",
"title Young Guns II (1990) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) 0.680414 \n",
"12 Angry Men (1957) -0.361961 \n",
"187 (1997) 0.500000 \n",
"\n",
"title Young Poisoner's Handbook, The (1995) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) -4.875600e-17 \n",
"12 Angry Men (1957) 1.443376e-01 \n",
"187 (1997) 4.753271e-01 \n",
"\n",
"title Zeus and Roxanne (1997) unknown \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) 0.707107 NaN \n",
"12 Angry Men (1957) 1.000000 1.0 \n",
"187 (1997) NaN NaN \n",
"\n",
"title Á köldum klaka (Cold Fever) (1994) \n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"[5 rows x 1664 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"corrMatrix = userRatings.corr()\n",
"corrMatrix.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, we want to avoid spurious results that happened from just a handful of users that happened to rate the same pair of movies. In order to restrict our results to movies that lots of people rated together - and also give us more popular results that are more easily recongnizable - we'll use the min_periods argument to throw out results where fewer than 100 users rated a given movie pair:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>title</th>\n",
" <th>'Til There Was You (1997)</th>\n",
" <th>1-900 (1994)</th>\n",
" <th>101 Dalmatians (1996)</th>\n",
" <th>12 Angry Men (1957)</th>\n",
" <th>187 (1997)</th>\n",
" <th>2 Days in the Valley (1996)</th>\n",
" <th>20,000 Leagues Under the Sea (1954)</th>\n",
" <th>2001: A Space Odyssey (1968)</th>\n",
" <th>3 Ninjas: High Noon At Mega Mountain (1998)</th>\n",
" <th>39 Steps, The (1935)</th>\n",
" <th>...</th>\n",
" <th>Yankee Zulu (1994)</th>\n",
" <th>Year of the Horse (1997)</th>\n",
" <th>You So Crazy (1994)</th>\n",
" <th>Young Frankenstein (1974)</th>\n",
" <th>Young Guns (1988)</th>\n",
" <th>Young Guns II (1990)</th>\n",
" <th>Young Poisoner's Handbook, The (1995)</th>\n",
" <th>Zeus and Roxanne (1997)</th>\n",
" <th>unknown</th>\n",
" <th>Á köldum klaka (Cold Fever) (1994)</th>\n",
" </tr>\n",
" <tr>\n",
" <th>title</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>'Til There Was You (1997)</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1-900 (1994)</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>101 Dalmatians (1996)</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12 Angry Men (1957)</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>187 (1997)</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 1664 columns</p>\n",
"</div>"
],
"text/plain": [
"title 'Til There Was You (1997) 1-900 (1994) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) NaN NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) NaN NaN \n",
"\n",
"title 101 Dalmatians (1996) 12 Angry Men (1957) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) 1.0 NaN \n",
"12 Angry Men (1957) NaN 1.0 \n",
"187 (1997) NaN NaN \n",
"\n",
"title 187 (1997) 2 Days in the Valley (1996) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) NaN NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) NaN NaN \n",
"\n",
"title 20,000 Leagues Under the Sea (1954) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title 2001: A Space Odyssey (1968) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title 3 Ninjas: High Noon At Mega Mountain (1998) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title 39 Steps, The (1935) ... Yankee Zulu (1994) \\\n",
"title ... \n",
"'Til There Was You (1997) NaN ... NaN \n",
"1-900 (1994) NaN ... NaN \n",
"101 Dalmatians (1996) NaN ... NaN \n",
"12 Angry Men (1957) NaN ... NaN \n",
"187 (1997) NaN ... NaN \n",
"\n",
"title Year of the Horse (1997) You So Crazy (1994) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) NaN NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) NaN NaN \n",
"\n",
"title Young Frankenstein (1974) Young Guns (1988) \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) NaN NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) NaN NaN \n",
"\n",
"title Young Guns II (1990) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title Young Poisoner's Handbook, The (1995) \\\n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"title Zeus and Roxanne (1997) unknown \\\n",
"title \n",
"'Til There Was You (1997) NaN NaN \n",
"1-900 (1994) NaN NaN \n",
"101 Dalmatians (1996) NaN NaN \n",
"12 Angry Men (1957) NaN NaN \n",
"187 (1997) NaN NaN \n",
"\n",
"title Á köldum klaka (Cold Fever) (1994) \n",
"title \n",
"'Til There Was You (1997) NaN \n",
"1-900 (1994) NaN \n",
"101 Dalmatians (1996) NaN \n",
"12 Angry Men (1957) NaN \n",
"187 (1997) NaN \n",
"\n",
"[5 rows x 1664 columns]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"corrMatrix = userRatings.corr(method='pearson', min_periods=100)\n",
"corrMatrix.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's produce some movie recommendations for user ID 0, who I manually added to the data set as a test case. This guy really likes Star Wars and The Empire Strikes Back, but hated Gone with the Wind. I'll extract his ratings from the userRatings DataFrame, and use dropna() to get rid of missing data (leaving me only with a Series of the movies I actually rated:)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"title\n",
"Empire Strikes Back, The (1980) 5.0\n",
"Gone with the Wind (1939) 1.0\n",
"Star Wars (1977) 5.0\n",
"Name: 0, dtype: float64"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"myRatings = userRatings.loc[0].dropna()\n",
"myRatings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's go through each movie I rated one at a time, and build up a list of possible recommendations based on the movies similar to the ones I rated.\n",
"\n",
"So for each movie I rated, I'll retrieve the list of similar movies from our correlation matrix. I'll then scale those correlation scores by how well I rated the movie they are similar to, so movies similar to ones I liked count more than movies similar to ones I hated:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Adding sims for Empire Strikes Back, The (1980)...\n",
"Adding sims for Gone with the Wind (1939)...\n",
"Adding sims for Star Wars (1977)...\n",
"sorting...\n",
"Empire Strikes Back, The (1980) 5.000000\n",
"Star Wars (1977) 5.000000\n",
"Empire Strikes Back, The (1980) 3.741763\n",
"Star Wars (1977) 3.741763\n",
"Return of the Jedi (1983) 3.606146\n",
"Return of the Jedi (1983) 3.362779\n",
"Raiders of the Lost Ark (1981) 2.693297\n",
"Raiders of the Lost Ark (1981) 2.680586\n",
"Austin Powers: International Man of Mystery (1997) 1.887164\n",
"Sting, The (1973) 1.837692\n",
"dtype: float64\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\Frank\\AppData\\Local\\Temp\\ipykernel_15080\\3684523232.py:7: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`\n",
" sims = sims.map(lambda x: x * myRatings[i])\n",
"C:\\Users\\Frank\\AppData\\Local\\Temp\\ipykernel_15080\\3684523232.py:9: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.\n",
" simCandidates = pd.concat([simCandidates, sims])\n",
"C:\\Users\\Frank\\AppData\\Local\\Temp\\ipykernel_15080\\3684523232.py:7: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`\n",
" sims = sims.map(lambda x: x * myRatings[i])\n",
"C:\\Users\\Frank\\AppData\\Local\\Temp\\ipykernel_15080\\3684523232.py:7: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`\n",
" sims = sims.map(lambda x: x * myRatings[i])\n"
]
}
],
"source": [
"simCandidates = pd.Series()\n",
"for i in range(0, len(myRatings.index)):\n",
" print (\"Adding sims for \" + myRatings.index[i] + \"...\")\n",
" # Retrieve similar movies to this one that I rated\n",
" sims = corrMatrix[myRatings.index[i]].dropna()\n",
" # Now scale its similarity by how well I rated this movie\n",
" sims = sims.map(lambda x: x * myRatings[i])\n",
" # Add the score to the list of similarity candidates\n",
" simCandidates = pd.concat([simCandidates, sims])\n",
" \n",
"#Glance at our results so far:\n",
"print (\"sorting...\")\n",
"simCandidates.sort_values(inplace = True, ascending = False)\n",
"print (simCandidates.head(10))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is starting to look like something useful! Note that some of the same movies came up more than once, because they were similar to more than one movie I rated. We'll use groupby() to add together the scores from movies that show up more than once, so they'll count more:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"simCandidates = simCandidates.groupby(simCandidates.index).sum()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Empire Strikes Back, The (1980) 8.877450\n",
"Star Wars (1977) 8.870971\n",
"Return of the Jedi (1983) 7.178172\n",
"Raiders of the Lost Ark (1981) 5.519700\n",
"Indiana Jones and the Last Crusade (1989) 3.488028\n",
"Bridge on the River Kwai, The (1957) 3.366616\n",
"Back to the Future (1985) 3.357941\n",
"Sting, The (1973) 3.329843\n",
"Cinderella (1950) 3.245412\n",
"Field of Dreams (1989) 3.222311\n",
"dtype: float64"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"simCandidates.sort_values(inplace = True, ascending = False)\n",
"simCandidates.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The last thing we have to do is filter out movies I've already rated, as recommending a movie I've already watched isn't helpful:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Return of the Jedi (1983) 7.178172\n",
"Raiders of the Lost Ark (1981) 5.519700\n",
"Indiana Jones and the Last Crusade (1989) 3.488028\n",
"Bridge on the River Kwai, The (1957) 3.366616\n",
"Back to the Future (1985) 3.357941\n",
"Sting, The (1973) 3.329843\n",
"Cinderella (1950) 3.245412\n",
"Field of Dreams (1989) 3.222311\n",
"Wizard of Oz, The (1939) 3.200268\n",
"Dumbo (1941) 2.981645\n",
"dtype: float64"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"filteredSims = simCandidates.drop(myRatings.index)\n",
"filteredSims.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There we have it!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Can you improve on these results? Perhaps a different method or min_periods value on the correlation computation would produce more interesting results.\n",
"\n",
"Also, it looks like some movies similar to Gone with the Wind - which I hated - made it through to the final list of recommendations. Perhaps movies similar to ones the user rated poorly should actually be penalized, instead of just scaled down?\n",
"\n",
"There are also probably some outliers in the user rating data set - some users may have rated a huge amount of movies and have a disporportionate effect on the results. Go back to earlier lectures to learn how to identify these outliers, and see if removing them improves things.\n",
"\n",
"For an even bigger project: we're evaluating the result qualitatively here, but we could actually apply train/test and measure our ability to predict user ratings for movies they've already watched. Whether that's actually a measure of a \"good\" recommendation is debatable, though!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}