1_Initial setting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b3dadcea",
   "metadata": {},
   "source": [
    "# 1. Initial setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "70c053be-de24-4735-856f-ce01ff25e7d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import datetime as dt\n",
    "import json\n",
    "import textwrap\n",
    "import logging\n",
    "\n",
    "from IPython.display import Markdown\n",
    "import polars as pl\n",
    "from praw import Reddit\n",
    "from pydantic import BaseModel, TypeAdapter, AwareDatetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ab0ecd-2557-48c2-83d8-b4cd02554006",
   "metadata": {},
   "source": [
    "# Extract data from Reddit\n",
    "\n",
    "We use PRAW https://pypi.org/project/praw/, \"Python Reddit API Wrapper\".\n",
    "\n",
    "Login to Reddit and create an app at https://www.reddit.com/prefs/apps\n",
    "\n",
    "![Reddit app](img/reddit-app.png)\n",
    "\n",
    "(Image from https://www.jcchouinard.com/get-reddit-api-credentials-with-praw/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ac5ca095",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a1333255",
   "metadata": {},
   "outputs": [],
   "source": [
    "%dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "735ec6c3-6f69-441d-99de-9846347935f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "reddit_client_id = os.environ[\"REDDIT_CLIENT_ID\"]\n",
    "reddit_client_secret = os.environ[\"REDDIT_SECRET\"]\n",
    "reddit_username = os.environ[\"REDDIT_USERNAME\"]  # Just for assembling the user agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4356ee1a-0cc6-4e26-b44f-15c296d76834",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, True)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read-only Reddit connection https://praw.readthedocs.io/en/stable/getting_started/quick_start.html#read-only-reddit-instances\n",
    "reddit = Reddit(\n",
    "    client_id=reddit_client_id,\n",
    "    client_secret=reddit_client_secret,\n",
    "    user_agent=f\"a-hole predictor by u/{reddit_username}\",\n",
    ")\n",
    "reddit.user.me(), reddit.read_only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50b4f677-8fd4-44e9-af66-759d673708fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_subreddit_name = \"r/AmItheAsshole\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2dc00270-2404-4120-99c9-d6275be6bd49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Am I the Asshole? \n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "#Welcome to r/AmITheAsshole!\r\n",
       "\r\n",
       "A catharsis for the frustrated moral philosopher in all of us, and a place to finally find out if you were wrong in a real-world argument that's been bothering you. Tell us about any non-violent conflict you have experienced; give us both sides of the story, and find out if you're right, or you're the asshole.\r\n",
       "\r\n",
       "This is the sub to lay out your  actions and conflicts and get impartial judgment rendered against you.  Were you the asshole in that situation or not? Post should reflect real situations, and abide by the rules below.\r\n",
       "\r\n",
       "After 18 hours, your post will be given a flair representing the final judgment on your matter.  This flair is determined by the subscribers who have both rendered judgment and voted on which judgment is best.  ***The power of the crowd will judge you***.  If your top level comment has the highest number of upvotes in a thread, you will get a flair point. More details are listed in [our FAQ](https://www.reddit.com/r/AmItheAssho..."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subreddit = reddit.subreddit(target_subreddit_name.removeprefix(\"r/\"))\n",
    "print(subreddit.title)\n",
    "Markdown(subreddit.description[:1_000] + \"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cbfe70bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://praw.readthedocs.io/en/v7.7.1/getting_started/logging.html\n",
    "handler = logging.StreamHandler()\n",
    "handler.setLevel(logging.DEBUG)\n",
    "for logger_name in (\"praw\", \"prawcore\"):\n",
    "    logger = logging.getLogger(logger_name)\n",
    "    logger.setLevel(logging.DEBUG)\n",
    "    logger.addHandler(handler)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1acb70ec",
   "metadata": {},
   "source": [
    "> \"To page through a listing, start by fetching the first page without specifying values for after and count.\n",
    "> The response will contain an after value which you can pass in the next request.\n",
    "> It is a good idea, but not required, to send an updated value for count which should be the number of items already fetched.\n",
    "\n",
    "https://old.reddit.com/dev/api#listings\n",
    "\n",
    "> We enforce rate limits for those eligible for free access usage of our Data API. The limit is:\n",
    "> - 100 queries per minute (QPM) per OAuth client id\n",
    "> QPM limits will be an average over a time window (currently 10 minutes) to support bursting requests.\n",
    "\n",
    "https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6f1474f8-54ed-462d-88ea-d333f8df4060",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395504.189213\n",
      "Data: None\n",
      "Params: {'after': 't3_1dwignu', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (114184 bytes) (rst-95:rem-981.0:used-19 ratelimit) at 1720395505.4814022\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395505.5022268\n",
      "Data: None\n",
      "Params: {'after': 't3_1dvv09v', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (109113 bytes) (rst-94:rem-980.0:used-20 ratelimit) at 1720395507.268327\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395507.283094\n",
      "Data: None\n",
      "Params: {'after': 't3_1dv8tz6', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (112015 bytes) (rst-92:rem-979.0:used-21 ratelimit) at 1720395508.988001\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395509.003242\n",
      "Data: None\n",
      "Params: {'after': 't3_1duoubx', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (108062 bytes) (rst-90:rem-978.0:used-22 ratelimit) at 1720395510.574493\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395510.593164\n",
      "Data: None\n",
      "Params: {'after': 't3_1du5csj', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (113577 bytes) (rst-89:rem-977.0:used-23 ratelimit) at 1720395511.978988\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395511.987009\n",
      "Data: None\n",
      "Params: {'after': 't3_1dtmvav', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (117148 bytes) (rst-87:rem-976.0:used-24 ratelimit) at 1720395513.7725499\n",
      "Fetching: GET https://oauth.reddit.com/r/AmItheAsshole/new at 1720395513.788673\n",
      "Data: None\n",
      "Params: {'after': 't3_1dt4xma', 'limit': 1000, 'raw_json': 1}\n",
      "Response: 200 (78273 bytes) (rst-86:rem-975.0:used-25 ratelimit) at 1720395514.635891\n"
     ]
    }
   ],
   "source": [
    "submissions = []\n",
    "for submission in subreddit.new(limit=1_000, params=dict(after=\"t3_1dwignu\")):\n",
    "    submissions.append(\n",
    "        {\n",
    "            \"title\": submission.title,\n",
    "            \"author_name\": submission.author.name if submission.author else None,\n",
    "            \"creation_datetime\": dt.datetime.fromtimestamp(\n",
    "                submission.created_utc, tz=dt.timezone.utc,\n",
    "            ).isoformat(),\n",
    "            \"subreddit_name\": submission.subreddit_name_prefixed,\n",
    "            \"num_comments\": submission.num_comments,\n",
    "            \"sfw\": not submission.over_18,\n",
    "            \"score\": submission.score,\n",
    "            \"upvote_ratio\": submission.upvote_ratio,\n",
    "            \"is_self\": submission.is_self,\n",
    "            \"permalink\": submission.permalink,\n",
    "            \"selftext\": submission.selftext,\n",
    "            \"flair_text\": submission.link_flair_text,  # Target\n",
    "        }\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6077c721",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "668"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(submissions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "89812861",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"title\": \"AITA for kicking guests out of a theme park restroom at\n",
      "12:40 AM\", \"author_name\": \"TheHylind\", \"creation_datetime\":\n",
      "\"2024-07-06T05:36:33+00:00\", \"subreddit_name\": \"r/AmItheAsshole\",\n",
      "\"num_comments\": 7, \"sfw\": true, \"score\": 1, \"upvote_ratio\": 1.0,\n",
      "\"is_self\": true, \"permalink\": \"/r/AmItheAsshole/comments/1dwif5p/aita_\n",
      "for_kicking_guests_out_of_a_theme_park/\", \"selftext\": \"For context, I\n",
      "work at a local theme park that, for reasons beyond my comprehension,\n",
      "closes at midnight. Obviously I can't disclose the name of the park or\n",
      "anything like that, but it's notably smaller than Disney or even most\n",
      "Six Flags parks. Most of our guests are either locals that come here\n",
      "every day, or tourists who came here for another reason and just\n",
      "wanted to check out what was in the area.\\n\\nBecause of this, we often\n",
      "have a lot of stragglers even when we're already closed, which makes\n",
      "it really frustrating when I'm assigned to clean the bathrooms,\n",
      "especially because my bathrooms are literally right ...\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(textwrap.wrap((json.dumps(submissions)[:1_000] + \"...\"), width=70)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417238f6-3028-40d9-9057-77a7192f9083",
   "metadata": {},
   "source": [
    "Notice the `creation_datetime` is `str`, for easier serialization!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "499e6b75-8f55-4393-981a-1c6b47af7803",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2024-07-01T13:15:44+00:00'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submissions[-1][\"creation_datetime\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf9cf92b-8c4a-47d1-91f0-1300d1cade92",
   "metadata": {},
   "source": [
    "# Model the data\n",
    "\n",
    "We use Pydantic https://pypi.org/project/pydantic/, a popular Python library for data validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e467c5d0-96b2-4572-b1aa-3735fe05ccaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RedditSubmission(BaseModel):\n",
    "    title: str\n",
    "    author_name: str | None\n",
    "    creation_datetime: AwareDatetime\n",
    "    subreddit_name: str\n",
    "    num_comments: int\n",
    "    sfw: bool\n",
    "    score: int\n",
    "    upvote_ratio: float\n",
    "    is_self: bool\n",
    "    permalink: str\n",
    "    selftext: str | None\n",
    "    flair_text: str | None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ca0049fc-69e3-47d1-8661-41de5f571214",
   "metadata": {},
   "outputs": [],
   "source": [
    "adapter = TypeAdapter(list[RedditSubmission])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "00c492f5-231d-4638-80b9-d7bafaff291e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RedditSubmission(title='AITA for kicking guests out of a theme park restroom at 12:40 AM', author_name='TheHylind', creation_datetime=datetime.datetime(2024, 7, 6, 5, 36, 33, tzinfo=TzInfo(UTC)), subreddit_name='r/AmItheAsshole', num_comments=7, sfw=True, score=1, upvote_ratio=1.0, is_self=True, permalink='/r/AmItheAsshole/comments/1dwif5p/aita_for_kicking_guests_out_of_a_theme_park/', selftext='For context, I work at a local theme park that, for reasons beyond my comprehension, closes at midnight. Obviously I can\\'t disclose the name of the park or anything like that, but it\\'s notably smaller than Disney or even most Six Flags parks. Most of our guests are either locals that come here every day, or tourists who came here for another reason and just wanted to check out what was in the area.\\n\\nBecause of this, we often have a lot of stragglers even when we\\'re already closed, which makes it really frustrating when I\\'m assigned to clean the bathrooms, especially because my bathrooms are literally right next to the park exit. I\\'ll be practically done with everything and a drunk 50-something will stroll in and be like \"dude I gotta pee\"\\n\\nI used to clean everything but the urinals just in case, but something broke in me tonight. I completely blocked off the door until the place was mopped, forced everyone out, saying \"The park has been closed for 30 minutes, you had all night.\"\\n\\nPart of me feels proud for escaping my usual doormat demeanor, but surely these guys had struggles too, right? Maybe a long drive, or something? I pride myself on making guests feel welcome, so it makes me feel kinda icky turning people in need away. That being said, I don\\'t want to get home at 1:30-2:00 AM because someone decided that they needed to use the restroom now as opposed to, you know, when we were open.\\n\\nSide note: I\\'m not a custodian, I\\'m a ride operator, but we get assigned closing tasks and I\\'m almost always cleaning the restrooms.', flair_text='Not enough info'),\n",
       " RedditSubmission(title=\"AITA for reporting coworker's unprofessional behavior to my boss?\", author_name='Allethiia', creation_datetime=datetime.datetime(2024, 7, 6, 5, 21, 10, tzinfo=TzInfo(UTC)), subreddit_name='r/AmItheAsshole', num_comments=2, sfw=True, score=3, upvote_ratio=0.6, is_self=True, permalink='/r/AmItheAsshole/comments/1dwi6cw/aita_for_reporting_coworkers_unprofessional/', selftext=\"I (28) have been working at my company for almost a year. I have a coworker, Loopy (31) who has worked at the company with me for the same time - we are part of the same team and no one has a higher position than the other one. We previously worked together at another company, where I was a project manager and had to fix their mistakes and the owner would come to me to ask me to redo what they had done. Currently, they tends to micromanage my work, even though they lack expertise in my field. Despite attempts to resolve issues directly, Loopy's unprofessional conduct continues.\\n\\nThe problems include delays in workflow, where their involvement in every detail of projects leads to unnecessary delays. Projects that take me little time to do are held up for days due to their slow feedback. A lot of things had to be rushed or were not launched at all because they took too long. This is a consistent issue, happening with nearly every project and minor detail since they want to be involved in everything.\\n\\nAdditionally, Loopy often claims credit for work I’ve done, telling others they contributed significantly when they only provided negative feedback. For example, she has told our boss that they talked to clients or created documents that were entirely my work. They misrepresented accomplishments in a letter to our boss claiming they achieved things I did. In meetings, they frequently diminishe my individual contributions by presenting them as collaborative efforts, while taking full credit for their own work and not even mention my big contributions to them. I feel they just want to review everything to put their stamp on it and make it seem they contributed more than they did.\\n\\nLoopy's conduct include gossiping about colleagues and speaking negatively about our boss. They have called another coworker “bitchy” and “bossy,” and has said our boss doesn't know what he is talking about and that the services the company offer are imposible to sell because the services are bad. They have also created a hostile environment by being dismissive of my feedback and making me feel discouraged from voicing professional opinions. For instance, when I provide feedback or disagree with their ideas, they responds with hostility and does whatever they want without taking my experienced opinion, making collaboration difficult.\\n\\nFurthermore, they have deliberately excluded me from participating in events or special out of office activities. These exclusions are deliberate and not a simple oversight and I know it because they broadcasted the invitation to everyone except me. Their behavior has made it challenging to maintain a positive work environment.\\n\\nThe last drop was when they called me unprofessional. Something exploded in me after that and I have now reported all of these issues to my boss in a detailed letter. Nothing came from it really, but I feel that is because the boss don't have time. At the end of the day, he has a company to run; however, I think Loopy was told something because afterwards, they confronted me angrily, accusing me of trying to undermine them and minimize their contributions. I sincerely think they are afraid I take their job or that I stall their growth in the company which is outrageous because I don't want to do that and I don't really care about anything else but my own growth and contributions. Since then the process has changed a little and now I do not wait for Loopy's revision or feedback before I go to the final decision maker directly. They definitely are not okay with this and they have stopped speaking to me. This also includes not replying to the emails I send to them AND the boss for feedback or approval. Also another thing that happened recently is that I was trying to start a new project and asked them if a meeting with them and the boss ti what they replied that they didn't care about my opinions and why would I ask them anything if I am still going to do whatever I want.\\n\\nAt the end of the day is not that I dont want feedback, is that I don't think there should be these many layers between me and the final approver.\\n\\nLoopy's behavior created a toxic work environment for a while, making me not care about anything that had to do with the company. I am usually a person that voices their opinions and I am always happy to work as a team and not being able to do so just because I just wanted to not make them angry put a weight on me and I started feeling unhappy, depressed and wanting to quit because I felt I had no growth opportunities in the company. Their behavior had been like that for months and I did make an attempts to address issues directly and diplomatically, but their behavior persisted.\\n\\nAnyway, I guess I just wanted to vent a little and ask the Redditors on this board - AITA for reporting their behavior? Should I have done something differently? And what should I do now?\", flair_text='TL;DR'),\n",
       " RedditSubmission(title='AITA for cancelling my birthday plans because my friends have fallen out', author_name='Lis_wj', creation_datetime=datetime.datetime(2024, 7, 6, 5, 14, tzinfo=TzInfo(UTC)), subreddit_name='r/AmItheAsshole', num_comments=6, sfw=True, score=1, upvote_ratio=0.6, is_self=True, permalink='/r/AmItheAsshole/comments/1dwi28r/aita_for_cancelling_my_birthday_plans_because_my/', selftext='I (26F) have been really stressed about planning my birthday.\\n\\nLiv (fake name) introduced her long term friends (one being Tom) to some newer friends (e.g, Elle). Elle started talking to Tom, and although was very clear she didn’t want anything serious started to get feelings for Tom. They would organise dates, but the dates never happened (from what I gather anyway).\\n\\nAfter one night out, Liv and Tom hooked up. Liv felt terrible about it and told Elle straight away. \\n\\nSince then, Elle has completely cut Liv out her life and has told me she refuses to be in the same room as her.\\n\\nNow it’s come round to my birthday, I’m kinda fed up with everything and just wanted to have smaller separate celebrations with Liv and Elles side of the friendship group separately. I told them both that I have cancelled my Saturday plans altogether (replacing them with midweek dinners out or movie nights). \\n\\nWhen I told Elle and her side of the group, they reiterated that they had left Saturday free for me. When I told them it was making me really anxious, and I have changed plans for everyone (in case they thought I was picking favourites), have all left me on read.\\n\\nAITA?', flair_text='Not the A-hole'),\n",
       " RedditSubmission(title='AITA: I told my sister she has no friends or relationships because she is evil and selfish', author_name='dswizzle2', creation_datetime=datetime.datetime(2024, 7, 6, 5, 9, 53, tzinfo=TzInfo(UTC)), subreddit_name='r/AmItheAsshole', num_comments=14, sfw=True, score=18, upvote_ratio=0.8, is_self=True, permalink='/r/AmItheAsshole/comments/1dwhztp/aita_i_told_my_sister_she_has_no_friends_or/', selftext='For context, I(27F) and my sister (25F) have always had a bit of a complex relationship. She’s the youngest and I’m the oldest (three girls in total!) all three of us are very close, often FaceTiming 4-6 times a day and sharing a lot of our lives with each other. We’ve gotten into arguments before because I have always felt she was a mean person (I.e., always wanting others to do for her but not wanting to reciprocate, easily irritated and often quick to talk down about others not like her, complaining a lot, talking to people in very rude ways). She feels that I am too sensitive and take everything personally. The basis of this blowout is below:\\n\\nI invited my partner to my mom’s place with my mom’s permission and he would stay the night as he was driving 4 hours to come for the 4th of July. When I told my sister she got super irritated and told me she didn’t like that I invited him. I was put off, but I didn’t really say anything initially. The rest of the day went on but it was clear she was upset. Later on that night, I asked her directly if she was upset with me. \\n\\nAt first she said “no I’m just tired” and then launched into saying that it was wrong of me to invite my partner over to stay the night without checking with her to see if she was comfortable with him staying. She also said she’s been annoyed because I kept bringing him up in conversations for no reason and felt that I was “forcing him on them”. I apologized for this and acknowledged that I should have checked with everyone before assuming they’d be cool with him in the house. I told her I only talked about him because I liked him and was happy with him and wanted to share in that with my siblings but if she wasn’t okay with that I’d stop. She got upset and felt I was being condescending and said “there’s a lot I could say but I’m not going to, it’s your relationship, so I don’t care”.  I said that she always says there’s more she could say and asked her to explain what she means by that. She then got more upset and told me to leave her alone. \\n\\nHere’s where I may be the ahole: I said “okay, I’ll leave you alone since you clearly need to be left alone”. I was frustrated with her and snapped a little bit. This comment apparently made her “black out” and she started yelling at me, calling me sensitive, saying this is why she doesn’t talk to me and why she doesn’t like me, and I told her that she’s mean and evil and I don’t like her either so we don’t have to talk at all anymore. Words were exchanged. I hit below the belt by telling her that the reason all of her friendships and relationships have ended is because she’s mean and makes everything about herself. She called me a fat a** b*tch and told me she hated me. \\n\\nI have apologized for my comments as I knew it was hurtful. She apologized too but insists that she blacked out, didn’t remember what she said, and only reacted that way because I “egged her on” with my passive aggressive comment. Please someone tell me if I’m wrong. ', flair_text='Not the A-hole'),\n",
       " RedditSubmission(title='WIBTA for calling out my friend for a build up of small things', author_name='gremlinoverlord_420', creation_datetime=datetime.datetime(2024, 7, 6, 4, 57, 21, tzinfo=TzInfo(UTC)), subreddit_name='r/AmItheAsshole', num_comments=9, sfw=True, score=0, upvote_ratio=0.33, is_self=True, permalink='/r/AmItheAsshole/comments/1dwhsay/wibta_for_calling_out_my_friend_for_a_build_up_of/', selftext='I (f) have gotten fed up with the way one of my (m) friends has being behaving towards me recently and I\\'d like to address it and to call him out of it infront of all our friends. \\n\\nFor context, me and this person have only been friends for a few months but since than I feel like there\\'s something weird about the way he teats particularly me. I\\'m unsure if it\\'s due to immaturity (he\\'s a few years younger than me) or the fact that he might be harbouring a small crush but he\\'s been doing these small things that have built up over time and have made me come to resent him quite a bit.\\n\\nFor example, he told me infront of me and roughly 12 other people that his cousin could imagine me in a 🌽 video. This made me and his cousin visibly uncomfortable, with his cousin looking mad and imitately denying it and me telling him that that pretty disrespectful. He went of to tell the rest of us that his cousin was lying and he did say that whist hysterically laughing about what he said, (Thinking about him doing this makes me really mad and uncomfortable even now).\\n\\nHe also plays video games with me but likes to egg another friend of ours on to hold meaningless information from me that doesn\\'t need to be held. Whilst saying backhanded things to me and than telling me it was a joke. These things have happened infront of our friends which I don\\'t think they\\'ve noticed.\\n\\nAnd whilst we were playing i had alot of particularly rare items and he decided to push me into a thing that kills alot of players while laughing as I freaked out. I managed to survive but bearly and he didn\\'t even seem to care that much that I was upset as he said to our other friend \"I guess she\\'s mad at me\"\\n\\nHe accused me of sharing embarrassing pictures of him wearng a costume and when I denied it he very visibility didn\\'t believe and seemed to hold it against me for a while. And told me a (m) friend of mine seemed to hate me (we bicker alot).\\n\\nI\\'m not sure if I\\'m just being petty or overreacting but the build up of what happened bas really been bothering me and I just want to scream at him infront of everyone so they know how I feel, it\\'s just when I\\'ve tried to call him out before people have told me I\\'m overreacting. \\n\\n If you\\'ve read this far thank you and I\\'m sorry for it being quite long', flair_text='Not the A-hole')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "objects = adapter.validate_python(submissions)\n",
    "objects[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa32cf0-be82-464f-a6f7-2aa7f01ba4a1",
   "metadata": {},
   "source": [
    "Pydantic automatically converted the str datetime to an actual `datetime.datatime` object, as specified in the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9e932cb9-022d-4416-a83b-859b2e68908d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2024, 7, 1, 13, 15, 44, tzinfo=TzInfo(UTC))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "objects[-1].creation_datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4e87a89-e5b4-48ef-8e7e-4111a3980b09",
   "metadata": {},
   "source": [
    "We now use Polars https://pypi.org/project/polars/, a dataframe library with an expressive API and blazing fast performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "70b7d7cf-91a2-4bad-a603-71782e8a303c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>title</th><th>author_name</th><th>creation_datetime</th><th>subreddit_name</th><th>num_comments</th><th>sfw</th><th>score</th><th>upvote_ratio</th><th>is_self</th><th>permalink</th><th>selftext</th><th>flair_text</th></tr><tr><td>str</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>i64</td><td>bool</td><td>i64</td><td>f64</td><td>bool</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>&quot;AITA for kicking guests out of…</td><td>&quot;TheHylind&quot;</td><td>2024-07-06 05:36:33 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>7</td><td>true</td><td>1</td><td>1.0</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;For context, I work at a local…</td><td>&quot;Not enough info&quot;</td></tr><tr><td>&quot;AITA for reporting coworker&#x27;s …</td><td>&quot;Allethiia&quot;</td><td>2024-07-06 05:21:10 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>2</td><td>true</td><td>3</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (28) have been working at my…</td><td>&quot;TL;DR&quot;</td></tr><tr><td>&quot;AITA for cancelling my birthda…</td><td>&quot;Lis_wj&quot;</td><td>2024-07-06 05:14:00 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>6</td><td>true</td><td>1</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (26F) have been really stres…</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>&quot;AITA: I told my sister she has…</td><td>&quot;dswizzle2&quot;</td><td>2024-07-06 05:09:53 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>14</td><td>true</td><td>18</td><td>0.8</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;For context, I(27F) and my sis…</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>&quot;WIBTA for calling out my frien…</td><td>&quot;gremlinoverlord_420&quot;</td><td>2024-07-06 04:57:21 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>9</td><td>true</td><td>0</td><td>0.33</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;I (f) have gotten fed up with …</td><td>&quot;Not the A-hole&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 12)\n",
       "┌────────────┬───────────┬───────────┬───────────┬───┬─────────┬───────────┬───────────┬───────────┐\n",
       "│ title      ┆ author_na ┆ creation_ ┆ subreddit ┆ … ┆ is_self ┆ permalink ┆ selftext  ┆ flair_tex │\n",
       "│ ---        ┆ me        ┆ datetime  ┆ _name     ┆   ┆ ---     ┆ ---       ┆ ---       ┆ t         │\n",
       "│ str        ┆ ---       ┆ ---       ┆ ---       ┆   ┆ bool    ┆ str       ┆ str       ┆ ---       │\n",
       "│            ┆ str       ┆ datetime[ ┆ str       ┆   ┆         ┆           ┆           ┆ str       │\n",
       "│            ┆           ┆ μs, UTC]  ┆           ┆   ┆         ┆           ┆           ┆           │\n",
       "╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═════════╪═══════════╪═══════════╪═══════════╡\n",
       "│ AITA for   ┆ TheHylind ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ For       ┆ Not       │\n",
       "│ kicking    ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ context,  ┆ enough    │\n",
       "│ guests out ┆           ┆ 05:36:33  ┆           ┆   ┆         ┆ omments/1 ┆ I work at ┆ info      │\n",
       "│ of…        ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ a local…  ┆           │\n",
       "│ AITA for   ┆ Allethiia ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (28)    ┆ TL;DR     │\n",
       "│ reporting  ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have been ┆           │\n",
       "│ coworker's ┆           ┆ 05:21:10  ┆           ┆   ┆         ┆ omments/1 ┆ working   ┆           │\n",
       "│ …          ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ at my…    ┆           │\n",
       "│ AITA for   ┆ Lis_wj    ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (26F)   ┆ Not the   │\n",
       "│ cancelling ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have been ┆ A-hole    │\n",
       "│ my         ┆           ┆ 05:14:00  ┆           ┆   ┆         ┆ omments/1 ┆ really    ┆           │\n",
       "│ birthda…   ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ stres…    ┆           │\n",
       "│ AITA: I    ┆ dswizzle2 ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ For       ┆ Not the   │\n",
       "│ told my    ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ context,  ┆ A-hole    │\n",
       "│ sister she ┆           ┆ 05:09:53  ┆           ┆   ┆         ┆ omments/1 ┆ I(27F)    ┆           │\n",
       "│ has…       ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwh…      ┆ and my    ┆           │\n",
       "│            ┆           ┆           ┆           ┆   ┆         ┆           ┆ sis…      ┆           │\n",
       "│ WIBTA for  ┆ gremlinov ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (f)     ┆ Not the   │\n",
       "│ calling    ┆ erlord_42 ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have      ┆ A-hole    │\n",
       "│ out my     ┆ 0         ┆ 04:57:21  ┆           ┆   ┆         ┆ omments/1 ┆ gotten    ┆           │\n",
       "│ frien…     ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwh…      ┆ fed up    ┆           │\n",
       "│            ┆           ┆           ┆           ┆   ┆         ┆           ┆ with …    ┆           │\n",
       "└────────────┴───────────┴───────────┴───────────┴───┴─────────┴───────────┴───────────┴───────────┘"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = (\n",
    "    pl.from_dicts(objects)\n",
    "    .with_columns(\n",
    "        # Otherwise the timezone is lost\n",
    "        pl.col(\"creation_datetime\").dt.replace_time_zone(\"UTC\"),\n",
    "    )\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bdfc1cce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (11, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>flair_text</th><th>count</th><th>rel_pct</th></tr><tr><td>str</td><td>u32</td><td>f64</td></tr></thead><tbody><tr><td>&quot;Not the A-hole&quot;</td><td>480</td><td>71.9</td></tr><tr><td>&quot;Asshole&quot;</td><td>84</td><td>12.6</td></tr><tr><td>&quot;No A-holes here&quot;</td><td>32</td><td>4.8</td></tr><tr><td>&quot;Everyone Sucks&quot;</td><td>26</td><td>3.9</td></tr><tr><td>&quot;TL;DR&quot;</td><td>18</td><td>2.7</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>5</td><td>0.7</td></tr><tr><td>&quot;Not the A-hole POO Mode&quot;</td><td>4</td><td>0.6</td></tr><tr><td>&quot;Asshole POO Mode&quot;</td><td>3</td><td>0.4</td></tr><tr><td>&quot;UPDATE&quot;</td><td>1</td><td>0.1</td></tr><tr><td>&quot;POO Mode Activated 💩&quot;</td><td>1</td><td>0.1</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (11, 3)\n",
       "┌─────────────────────────┬───────┬─────────┐\n",
       "│ flair_text              ┆ count ┆ rel_pct │\n",
       "│ ---                     ┆ ---   ┆ ---     │\n",
       "│ str                     ┆ u32   ┆ f64     │\n",
       "╞═════════════════════════╪═══════╪═════════╡\n",
       "│ Not the A-hole          ┆ 480   ┆ 71.9    │\n",
       "│ Asshole                 ┆ 84    ┆ 12.6    │\n",
       "│ No A-holes here         ┆ 32    ┆ 4.8     │\n",
       "│ Everyone Sucks          ┆ 26    ┆ 3.9     │\n",
       "│ TL;DR                   ┆ 18    ┆ 2.7     │\n",
       "│ …                       ┆ …     ┆ …       │\n",
       "│ null                    ┆ 5     ┆ 0.7     │\n",
       "│ Not the A-hole POO Mode ┆ 4     ┆ 0.6     │\n",
       "│ Asshole POO Mode        ┆ 3     ┆ 0.4     │\n",
       "│ UPDATE                  ┆ 1     ┆ 0.1     │\n",
       "│ POO Mode Activated 💩   ┆ 1     ┆ 0.1     │\n",
       "└─────────────────────────┴───────┴─────────┘"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    df[\"flair_text\"]\n",
    "    .value_counts()\n",
    "    .sort(\"count\", descending=True)\n",
    "    .with_columns(\n",
    "        (pl.col(\"count\") / pl.sum(\"count\") * 100).round(1).alias(\"rel_pct\"),\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "352793ae",
   "metadata": {},
   "source": [
    "Approximately 85 % of the posts have a definitive result."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d7c6a38",
   "metadata": {},
   "source": [
    "# Store data\n",
    "\n",
    "Let's write it to a local Parquet file to avoid having to retrieve the data from the API again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0eb84d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.write_parquet(\"submissions.pq\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e68919f9",
   "metadata": {},
   "source": [
    "# Load raw data and do feature engineering\n",
    "\n",
    "We load the data again from the same Parquet file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d08f6ff8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>title</th><th>author_name</th><th>creation_datetime</th><th>subreddit_name</th><th>num_comments</th><th>sfw</th><th>score</th><th>upvote_ratio</th><th>is_self</th><th>permalink</th><th>selftext</th><th>flair_text</th></tr><tr><td>str</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>i64</td><td>bool</td><td>i64</td><td>f64</td><td>bool</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>&quot;AITA for kicking guests out of…</td><td>&quot;TheHylind&quot;</td><td>2024-07-06 05:36:33 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>7</td><td>true</td><td>1</td><td>1.0</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;For context, I work at a local…</td><td>&quot;Not enough info&quot;</td></tr><tr><td>&quot;AITA for reporting coworker&#x27;s …</td><td>&quot;Allethiia&quot;</td><td>2024-07-06 05:21:10 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>2</td><td>true</td><td>3</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (28) have been working at my…</td><td>&quot;TL;DR&quot;</td></tr><tr><td>&quot;AITA for cancelling my birthda…</td><td>&quot;Lis_wj&quot;</td><td>2024-07-06 05:14:00 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>6</td><td>true</td><td>1</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (26F) have been really stres…</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>&quot;AITA: I told my sister she has…</td><td>&quot;dswizzle2&quot;</td><td>2024-07-06 05:09:53 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>14</td><td>true</td><td>18</td><td>0.8</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;For context, I(27F) and my sis…</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>&quot;WIBTA for calling out my frien…</td><td>&quot;gremlinoverlord_420&quot;</td><td>2024-07-06 04:57:21 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>9</td><td>true</td><td>0</td><td>0.33</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;I (f) have gotten fed up with …</td><td>&quot;Not the A-hole&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 12)\n",
       "┌────────────┬───────────┬───────────┬───────────┬───┬─────────┬───────────┬───────────┬───────────┐\n",
       "│ title      ┆ author_na ┆ creation_ ┆ subreddit ┆ … ┆ is_self ┆ permalink ┆ selftext  ┆ flair_tex │\n",
       "│ ---        ┆ me        ┆ datetime  ┆ _name     ┆   ┆ ---     ┆ ---       ┆ ---       ┆ t         │\n",
       "│ str        ┆ ---       ┆ ---       ┆ ---       ┆   ┆ bool    ┆ str       ┆ str       ┆ ---       │\n",
       "│            ┆ str       ┆ datetime[ ┆ str       ┆   ┆         ┆           ┆           ┆ str       │\n",
       "│            ┆           ┆ μs, UTC]  ┆           ┆   ┆         ┆           ┆           ┆           │\n",
       "╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═════════╪═══════════╪═══════════╪═══════════╡\n",
       "│ AITA for   ┆ TheHylind ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ For       ┆ Not       │\n",
       "│ kicking    ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ context,  ┆ enough    │\n",
       "│ guests out ┆           ┆ 05:36:33  ┆           ┆   ┆         ┆ omments/1 ┆ I work at ┆ info      │\n",
       "│ of…        ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ a local…  ┆           │\n",
       "│ AITA for   ┆ Allethiia ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (28)    ┆ TL;DR     │\n",
       "│ reporting  ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have been ┆           │\n",
       "│ coworker's ┆           ┆ 05:21:10  ┆           ┆   ┆         ┆ omments/1 ┆ working   ┆           │\n",
       "│ …          ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ at my…    ┆           │\n",
       "│ AITA for   ┆ Lis_wj    ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (26F)   ┆ Not the   │\n",
       "│ cancelling ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have been ┆ A-hole    │\n",
       "│ my         ┆           ┆ 05:14:00  ┆           ┆   ┆         ┆ omments/1 ┆ really    ┆           │\n",
       "│ birthda…   ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwi…      ┆ stres…    ┆           │\n",
       "│ AITA: I    ┆ dswizzle2 ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ For       ┆ Not the   │\n",
       "│ told my    ┆           ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ context,  ┆ A-hole    │\n",
       "│ sister she ┆           ┆ 05:09:53  ┆           ┆   ┆         ┆ omments/1 ┆ I(27F)    ┆           │\n",
       "│ has…       ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwh…      ┆ and my    ┆           │\n",
       "│            ┆           ┆           ┆           ┆   ┆         ┆           ┆ sis…      ┆           │\n",
       "│ WIBTA for  ┆ gremlinov ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ true    ┆ /r/AmIthe ┆ I (f)     ┆ Not the   │\n",
       "│ calling    ┆ erlord_42 ┆ 6         ┆ sshole    ┆   ┆         ┆ Asshole/c ┆ have      ┆ A-hole    │\n",
       "│ out my     ┆ 0         ┆ 04:57:21  ┆           ┆   ┆         ┆ omments/1 ┆ gotten    ┆           │\n",
       "│ frien…     ┆           ┆ UTC       ┆           ┆   ┆         ┆ dwh…      ┆ fed up    ┆           │\n",
       "│            ┆           ┆           ┆           ┆   ┆         ┆           ┆ with …    ┆           │\n",
       "└────────────┴───────────┴───────────┴───────────┴───┴─────────┴───────────┴───────────┴───────────┘"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import polars as pl\n",
    "\n",
    "df = pl.read_parquet(\"submissions.pq\").sort(\"creation_datetime\", descending=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ebd0e584",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 14)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>title</th><th>author_name</th><th>creation_datetime</th><th>subreddit_name</th><th>num_comments</th><th>sfw</th><th>score</th><th>upvote_ratio</th><th>is_self</th><th>permalink</th><th>selftext</th><th>flair_text</th><th>post_type</th><th>text_length</th></tr><tr><td>str</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>i64</td><td>bool</td><td>i64</td><td>f64</td><td>bool</td><td>str</td><td>str</td><td>str</td><td>str</td><td>u32</td></tr></thead><tbody><tr><td>&quot;AITA for kicking guests out of…</td><td>&quot;TheHylind&quot;</td><td>2024-07-06 05:36:33 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>7</td><td>true</td><td>1</td><td>1.0</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;For context, I work at a local…</td><td>&quot;Not enough info&quot;</td><td>&quot;AITA&quot;</td><td>1546</td></tr><tr><td>&quot;AITA for reporting coworker&#x27;s …</td><td>&quot;Allethiia&quot;</td><td>2024-07-06 05:21:10 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>2</td><td>true</td><td>3</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (28) have been working at my…</td><td>&quot;TL;DR&quot;</td><td>&quot;AITA&quot;</td><td>4890</td></tr><tr><td>&quot;AITA for cancelling my birthda…</td><td>&quot;Lis_wj&quot;</td><td>2024-07-06 05:14:00 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>6</td><td>true</td><td>1</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (26F) have been really stres…</td><td>&quot;Not the A-hole&quot;</td><td>&quot;AITA&quot;</td><td>1169</td></tr><tr><td>&quot;AITA: I told my sister she has…</td><td>&quot;dswizzle2&quot;</td><td>2024-07-06 05:09:53 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>14</td><td>true</td><td>18</td><td>0.8</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;For context, I(27F) and my sis…</td><td>&quot;Not the A-hole&quot;</td><td>&quot;AITA&quot;</td><td>2996</td></tr><tr><td>&quot;WIBTA for calling out my frien…</td><td>&quot;gremlinoverlord_420&quot;</td><td>2024-07-06 04:57:21 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>9</td><td>true</td><td>0</td><td>0.33</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;I (f) have gotten fed up with …</td><td>&quot;Not the A-hole&quot;</td><td>&quot;WIBTA&quot;</td><td>2284</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 14)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ title     ┆ author_na ┆ creation_ ┆ subreddit ┆ … ┆ selftext  ┆ flair_tex ┆ post_type ┆ text_len │\n",
       "│ ---       ┆ me        ┆ datetime  ┆ _name     ┆   ┆ ---       ┆ t         ┆ ---       ┆ gth      │\n",
       "│ str       ┆ ---       ┆ ---       ┆ ---       ┆   ┆ str       ┆ ---       ┆ str       ┆ ---      │\n",
       "│           ┆ str       ┆ datetime[ ┆ str       ┆   ┆           ┆ str       ┆           ┆ u32      │\n",
       "│           ┆           ┆ μs, UTC]  ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ AITA for  ┆ TheHylind ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ For       ┆ Not       ┆ AITA      ┆ 1546     │\n",
       "│ kicking   ┆           ┆ 6         ┆ sshole    ┆   ┆ context,  ┆ enough    ┆           ┆          │\n",
       "│ guests    ┆           ┆ 05:36:33  ┆           ┆   ┆ I work at ┆ info      ┆           ┆          │\n",
       "│ out of…   ┆           ┆ UTC       ┆           ┆   ┆ a local…  ┆           ┆           ┆          │\n",
       "│ AITA for  ┆ Allethiia ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ I (28)    ┆ TL;DR     ┆ AITA      ┆ 4890     │\n",
       "│ reporting ┆           ┆ 6         ┆ sshole    ┆   ┆ have been ┆           ┆           ┆          │\n",
       "│ coworker' ┆           ┆ 05:21:10  ┆           ┆   ┆ working   ┆           ┆           ┆          │\n",
       "│ s …       ┆           ┆ UTC       ┆           ┆   ┆ at my…    ┆           ┆           ┆          │\n",
       "│ AITA for  ┆ Lis_wj    ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ I (26F)   ┆ Not the   ┆ AITA      ┆ 1169     │\n",
       "│ cancellin ┆           ┆ 6         ┆ sshole    ┆   ┆ have been ┆ A-hole    ┆           ┆          │\n",
       "│ g my      ┆           ┆ 05:14:00  ┆           ┆   ┆ really    ┆           ┆           ┆          │\n",
       "│ birthda…  ┆           ┆ UTC       ┆           ┆   ┆ stres…    ┆           ┆           ┆          │\n",
       "│ AITA: I   ┆ dswizzle2 ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ For       ┆ Not the   ┆ AITA      ┆ 2996     │\n",
       "│ told my   ┆           ┆ 6         ┆ sshole    ┆   ┆ context,  ┆ A-hole    ┆           ┆          │\n",
       "│ sister    ┆           ┆ 05:09:53  ┆           ┆   ┆ I(27F)    ┆           ┆           ┆          │\n",
       "│ she has…  ┆           ┆ UTC       ┆           ┆   ┆ and my    ┆           ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ sis…      ┆           ┆           ┆          │\n",
       "│ WIBTA for ┆ gremlinov ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ I (f)     ┆ Not the   ┆ WIBTA     ┆ 2284     │\n",
       "│ calling   ┆ erlord_42 ┆ 6         ┆ sshole    ┆   ┆ have      ┆ A-hole    ┆           ┆          │\n",
       "│ out my    ┆ 0         ┆ 04:57:21  ┆           ┆   ┆ gotten    ┆           ┆           ┆          │\n",
       "│ frien…    ┆           ┆ UTC       ┆           ┆   ┆ fed up    ┆           ┆           ┆          │\n",
       "│           ┆           ┆           ┆           ┆   ┆ with …    ┆           ┆           ┆          │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Two types of posts: AITA and WIBTA https://www.reddit.com/r/AmItheAsshole/wiki/howtopost/\n",
    "df = (\n",
    "    df.with_columns(\n",
    "        pl.col(\"title\").str.extract(r\"^(AITA|WIBTA)\", 1).alias(\"post_type\"),\n",
    "        pl.col(\"selftext\").str.len_chars().alias(\"text_length\"),\n",
    "    )\n",
    ")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ea26c9",
   "metadata": {},
   "source": [
    "Now let's perform some quick sentiment analysis using NLTK:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "39b9283b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package vader_lexicon to\n",
      "[nltk_data]     /Users/juan_cano/nltk_data...\n",
      "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "nltk.download('vader_lexicon')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "707307e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sia = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e4e0d692",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'neg': 0.076, 'neu': 0.828, 'pos': 0.096, 'compound': 0.7268}\n",
      "{'neg': 0.087, 'neu': 0.843, 'pos': 0.07, 'compound': -0.9138}\n",
      "{'neg': 0.071, 'neu': 0.849, 'pos': 0.081, 'compound': 0.2894}\n",
      "{'neg': 0.136, 'neu': 0.768, 'pos': 0.096, 'compound': -0.9803}\n",
      "{'neg': 0.153, 'neu': 0.722, 'pos': 0.125, 'compound': -0.9398}\n"
     ]
    }
   ],
   "source": [
    "for index in range(5):\n",
    "    print(sia.polarity_scores(df.item(index, \"selftext\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5088608",
   "metadata": {},
   "source": [
    "Better to do it by sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6f11e09f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>permalink</th><th>sentences</th><th>sentiment_scores</th></tr><tr><td>str</td><td>str</td><td>struct[4]</td></tr></thead><tbody><tr><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;For context, I work at a local…</td><td>{0.0,1.0,0.0,0.0}</td></tr><tr><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;Obviously I can&#x27;t disclose the…</td><td>{0.0,0.929,0.071,0.1901}</td></tr><tr><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;Most of our guests are either …</td><td>{0.0,1.0,0.0,0.0}</td></tr><tr><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;Because of this, we often have…</td><td>{0.076,0.859,0.064,-0.1263}</td></tr><tr><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I&#x27;ll be practically done with …</td><td>{0.138,0.769,0.092,-0.4215}</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 3)\n",
       "┌─────────────────────────────────┬─────────────────────────────────┬─────────────────────────────┐\n",
       "│ permalink                       ┆ sentences                       ┆ sentiment_scores            │\n",
       "│ ---                             ┆ ---                             ┆ ---                         │\n",
       "│ str                             ┆ str                             ┆ struct[4]                   │\n",
       "╞═════════════════════════════════╪═════════════════════════════════╪═════════════════════════════╡\n",
       "│ /r/AmItheAsshole/comments/1dwi… ┆ For context, I work at a local… ┆ {0.0,1.0,0.0,0.0}           │\n",
       "│ /r/AmItheAsshole/comments/1dwi… ┆ Obviously I can't disclose the… ┆ {0.0,0.929,0.071,0.1901}    │\n",
       "│ /r/AmItheAsshole/comments/1dwi… ┆ Most of our guests are either … ┆ {0.0,1.0,0.0,0.0}           │\n",
       "│ /r/AmItheAsshole/comments/1dwi… ┆ Because of this, we often have… ┆ {0.076,0.859,0.064,-0.1263} │\n",
       "│ /r/AmItheAsshole/comments/1dwi… ┆ I'll be practically done with … ┆ {0.138,0.769,0.092,-0.4215} │\n",
       "└─────────────────────────────────┴─────────────────────────────────┴─────────────────────────────┘"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = (\n",
    "    df.with_columns(\n",
    "        pl.col(\"selftext\").str.split(\".\").list.eval(pl.element().str.strip_chars()).alias(\"sentences\")\n",
    "    )\n",
    "    .select(pl.col(\"permalink\", \"sentences\"))\n",
    "    .explode(\"sentences\")\n",
    "    .with_columns(\n",
    "        pl.col(\"sentences\").map_elements(\n",
    "            lambda s: sia.polarity_scores(s),\n",
    "            return_dtype=pl.Struct({\"neg\": pl.Float64, \"neu\": pl.Float64, \"pos\": pl.Float64, \"compound\": pl.Float64}),\n",
    "        ).alias(\"sentiment_scores\"),\n",
    "    )\n",
    ")\n",
    "sentences.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cfca063b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 15)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>title</th><th>author_name</th><th>creation_datetime</th><th>subreddit_name</th><th>num_comments</th><th>sfw</th><th>score</th><th>upvote_ratio</th><th>is_self</th><th>permalink</th><th>selftext</th><th>flair_text</th><th>post_type</th><th>text_length</th><th>compound_sentiment</th></tr><tr><td>str</td><td>str</td><td>datetime[μs, UTC]</td><td>str</td><td>i64</td><td>bool</td><td>i64</td><td>f64</td><td>bool</td><td>str</td><td>str</td><td>str</td><td>str</td><td>u32</td><td>f64</td></tr></thead><tbody><tr><td>&quot;AITA for kicking guests out of…</td><td>&quot;TheHylind&quot;</td><td>2024-07-06 05:36:33 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>7</td><td>true</td><td>1</td><td>1.0</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;For context, I work at a local…</td><td>&quot;Not enough info&quot;</td><td>&quot;AITA&quot;</td><td>1546</td><td>-0.0193</td></tr><tr><td>&quot;AITA for reporting coworker&#x27;s …</td><td>&quot;Allethiia&quot;</td><td>2024-07-06 05:21:10 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>2</td><td>true</td><td>3</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (28) have been working at my…</td><td>&quot;TL;DR&quot;</td><td>&quot;AITA&quot;</td><td>4890</td><td>-0.094389</td></tr><tr><td>&quot;AITA for cancelling my birthda…</td><td>&quot;Lis_wj&quot;</td><td>2024-07-06 05:14:00 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>6</td><td>true</td><td>1</td><td>0.6</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwi…</td><td>&quot;I (26F) have been really stres…</td><td>&quot;Not the A-hole&quot;</td><td>&quot;AITA&quot;</td><td>1169</td><td>0.040277</td></tr><tr><td>&quot;AITA: I told my sister she has…</td><td>&quot;dswizzle2&quot;</td><td>2024-07-06 05:09:53 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>14</td><td>true</td><td>18</td><td>0.8</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;For context, I(27F) and my sis…</td><td>&quot;Not the A-hole&quot;</td><td>&quot;AITA&quot;</td><td>2996</td><td>-0.056525</td></tr><tr><td>&quot;WIBTA for calling out my frien…</td><td>&quot;gremlinoverlord_420&quot;</td><td>2024-07-06 04:57:21 UTC</td><td>&quot;r/AmItheAsshole&quot;</td><td>9</td><td>true</td><td>0</td><td>0.33</td><td>true</td><td>&quot;/r/AmItheAsshole/comments/1dwh…</td><td>&quot;I (f) have gotten fed up with …</td><td>&quot;Not the A-hole&quot;</td><td>&quot;WIBTA&quot;</td><td>2284</td><td>-0.037443</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 15)\n",
       "┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐\n",
       "│ title     ┆ author_na ┆ creation_ ┆ subreddit ┆ … ┆ flair_tex ┆ post_type ┆ text_leng ┆ compound │\n",
       "│ ---       ┆ me        ┆ datetime  ┆ _name     ┆   ┆ t         ┆ ---       ┆ th        ┆ _sentime │\n",
       "│ str       ┆ ---       ┆ ---       ┆ ---       ┆   ┆ ---       ┆ str       ┆ ---       ┆ nt       │\n",
       "│           ┆ str       ┆ datetime[ ┆ str       ┆   ┆ str       ┆           ┆ u32       ┆ ---      │\n",
       "│           ┆           ┆ μs, UTC]  ┆           ┆   ┆           ┆           ┆           ┆ f64      │\n",
       "╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡\n",
       "│ AITA for  ┆ TheHylind ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ Not       ┆ AITA      ┆ 1546      ┆ -0.0193  │\n",
       "│ kicking   ┆           ┆ 6         ┆ sshole    ┆   ┆ enough    ┆           ┆           ┆          │\n",
       "│ guests    ┆           ┆ 05:36:33  ┆           ┆   ┆ info      ┆           ┆           ┆          │\n",
       "│ out of…   ┆           ┆ UTC       ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ AITA for  ┆ Allethiia ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ TL;DR     ┆ AITA      ┆ 4890      ┆ -0.09438 │\n",
       "│ reporting ┆           ┆ 6         ┆ sshole    ┆   ┆           ┆           ┆           ┆ 9        │\n",
       "│ coworker' ┆           ┆ 05:21:10  ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ s …       ┆           ┆ UTC       ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ AITA for  ┆ Lis_wj    ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ Not the   ┆ AITA      ┆ 1169      ┆ 0.040277 │\n",
       "│ cancellin ┆           ┆ 6         ┆ sshole    ┆   ┆ A-hole    ┆           ┆           ┆          │\n",
       "│ g my      ┆           ┆ 05:14:00  ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ birthda…  ┆           ┆ UTC       ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ AITA: I   ┆ dswizzle2 ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ Not the   ┆ AITA      ┆ 2996      ┆ -0.05652 │\n",
       "│ told my   ┆           ┆ 6         ┆ sshole    ┆   ┆ A-hole    ┆           ┆           ┆ 5        │\n",
       "│ sister    ┆           ┆ 05:09:53  ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ she has…  ┆           ┆ UTC       ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ WIBTA for ┆ gremlinov ┆ 2024-07-0 ┆ r/AmItheA ┆ … ┆ Not the   ┆ WIBTA     ┆ 2284      ┆ -0.03744 │\n",
       "│ calling   ┆ erlord_42 ┆ 6         ┆ sshole    ┆   ┆ A-hole    ┆           ┆           ┆ 3        │\n",
       "│ out my    ┆ 0         ┆ 04:57:21  ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "│ frien…    ┆           ┆ UTC       ┆           ┆   ┆           ┆           ┆           ┆          │\n",
       "└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.join(\n",
    "    (\n",
    "        sentences\n",
    "        .group_by(\"permalink\")\n",
    "        .agg(\n",
    "            pl.col(\"sentiment_scores\").struct.field(\"compound\").mean().alias(\"compound_sentiment\"),\n",
    "        )\n",
    "    ),\n",
    "    on=\"permalink\",\n",
    "    how=\"left\",\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "959e170a",
   "metadata": {},
   "source": [
    "## Predict result\n",
    "\n",
    "Now, the interesting part: can we create a machine learning model that can predict, _before the vote happens_, who is the a-hole of the story?\n",
    "\n",
    "Let's start with an extremely simple model and very few features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "08fc12a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.compose import make_column_transformer\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.model_selection import GridSearchCV, StratifiedKFold\n",
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "df6e0c5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;transformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;text_length&#x27;]),\n",
       "                                                 (&#x27;minmaxscaler&#x27;,\n",
       "                                                  MinMaxScaler(feature_range=(-1,\n",
       "                                                                              1)),\n",
       "                                                  [&#x27;compound_sentiment&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse_output=False),\n",
       "                                                  [&#x27;sfw&#x27;, &#x27;post_type&#x27;])])),\n",
       "                (&#x27;classifier&#x27;,\n",
       "                 DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;&nbsp;Pipeline<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.Pipeline.html\">?<span>Documentation for Pipeline</span></a><span class=\"sk-estimator-doc-link \">i<span>Not fitted</span></span></label><div class=\"sk-toggleable__content \"><pre>Pipeline(steps=[(&#x27;transformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;text_length&#x27;]),\n",
       "                                                 (&#x27;minmaxscaler&#x27;,\n",
       "                                                  MinMaxScaler(feature_range=(-1,\n",
       "                                                                              1)),\n",
       "                                                  [&#x27;compound_sentiment&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse_output=False),\n",
       "                                                  [&#x27;sfw&#x27;, &#x27;post_type&#x27;])])),\n",
       "                (&#x27;classifier&#x27;,\n",
       "                 DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;))])</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;transformer: ColumnTransformer<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.compose.ColumnTransformer.html\">?<span>Documentation for transformer: ColumnTransformer</span></a></label><div class=\"sk-toggleable__content \"><pre>ColumnTransformer(transformers=[(&#x27;passthrough&#x27;, &#x27;passthrough&#x27;, [&#x27;text_length&#x27;]),\n",
       "                                (&#x27;minmaxscaler&#x27;,\n",
       "                                 MinMaxScaler(feature_range=(-1, 1)),\n",
       "                                 [&#x27;compound_sentiment&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                               sparse_output=False),\n",
       "                                 [&#x27;sfw&#x27;, &#x27;post_type&#x27;])])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">passthrough</label><div class=\"sk-toggleable__content \"><pre>[&#x27;text_length&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">passthrough</label><div class=\"sk-toggleable__content \"><pre>passthrough</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">minmaxscaler</label><div class=\"sk-toggleable__content \"><pre>[&#x27;compound_sentiment&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;MinMaxScaler<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.MinMaxScaler.html\">?<span>Documentation for MinMaxScaler</span></a></label><div class=\"sk-toggleable__content \"><pre>MinMaxScaler(feature_range=(-1, 1))</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">onehotencoder</label><div class=\"sk-toggleable__content \"><pre>[&#x27;sfw&#x27;, &#x27;post_type&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;OneHotEncoder<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.OneHotEncoder.html\">?<span>Documentation for OneHotEncoder</span></a></label><div class=\"sk-toggleable__content \"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse_output=False)</pre></div> </div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;DecisionTreeClassifier<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.tree.DecisionTreeClassifier.html\">?<span>Documentation for DecisionTreeClassifier</span></a></label><div class=\"sk-toggleable__content \"><pre>DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;)</pre></div> </div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('transformer',\n",
       "                 ColumnTransformer(transformers=[('passthrough', 'passthrough',\n",
       "                                                  ['text_length']),\n",
       "                                                 ('minmaxscaler',\n",
       "                                                  MinMaxScaler(feature_range=(-1,\n",
       "                                                                              1)),\n",
       "                                                  ['compound_sentiment']),\n",
       "                                                 ('onehotencoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                sparse_output=False),\n",
       "                                                  ['sfw', 'post_type'])])),\n",
       "                ('classifier',\n",
       "                 DecisionTreeClassifier(class_weight='balanced'))])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = Pipeline([\n",
    "    (\"transformer\", make_column_transformer(\n",
    "        (\"passthrough\", [\"text_length\"]),\n",
    "        (MinMaxScaler((-1, 1)), [\"compound_sentiment\"]),\n",
    "        (OneHotEncoder(sparse_output=False, handle_unknown=\"ignore\"), [\"sfw\", \"post_type\"]),\n",
    "        remainder=\"drop\",  # Important to avoid accidental data leakage\n",
    "    ).set_output(transform=\"polars\")),\n",
    "    (\"classifier\", DecisionTreeClassifier(class_weight=\"balanced\")),\n",
    "])\n",
    "classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ee13645",
   "metadata": {},
   "source": [
    "We will only show the model entries with a definitive result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6dabb1bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "564"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = (\n",
    "    df\n",
    "    .with_columns(\n",
    "        pl.when(pl.col(\"author_name\").is_null())\n",
    "        .then(pl.lit(\"<UNKNOWN>\"))\n",
    "        .otherwise(pl.col(\"author_name\"))\n",
    "        .alias(\"author_name\"),\n",
    "    )\n",
    "    .filter(pl.col(\"flair_text\").is_in([\"Not the A-hole\", \"Asshole\"]))\n",
    ")\n",
    "len(df_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "4a0c1e11",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>passthrough__text_length</th><th>minmaxscaler__compound_sentiment</th><th>onehotencoder__sfw_False</th><th>onehotencoder__sfw_True</th><th>onehotencoder__post_type_AITA</th><th>onehotencoder__post_type_WIBTA</th></tr><tr><td>u32</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>1169</td><td>0.019746</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.0</td></tr><tr><td>2996</td><td>-0.081782</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.0</td></tr><tr><td>2284</td><td>-0.061768</td><td>0.0</td><td>1.0</td><td>0.0</td><td>1.0</td></tr><tr><td>2710</td><td>-0.292353</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.0</td></tr><tr><td>1651</td><td>-0.470861</td><td>0.0</td><td>1.0</td><td>1.0</td><td>0.0</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 6)\n",
       "┌────────────────┬────────────────┬────────────────┬───────────────┬───────────────┬───────────────┐\n",
       "│ passthrough__t ┆ minmaxscaler__ ┆ onehotencoder_ ┆ onehotencoder ┆ onehotencoder ┆ onehotencoder │\n",
       "│ ext_length     ┆ compound_senti ┆ _sfw_False     ┆ __sfw_True    ┆ __post_type_A ┆ __post_type_W │\n",
       "│ ---            ┆ me…            ┆ ---            ┆ ---           ┆ ITA           ┆ IBTA          │\n",
       "│ u32            ┆ ---            ┆ f64            ┆ f64           ┆ ---           ┆ ---           │\n",
       "│                ┆ f64            ┆                ┆               ┆ f64           ┆ f64           │\n",
       "╞════════════════╪════════════════╪════════════════╪═══════════════╪═══════════════╪═══════════════╡\n",
       "│ 1169           ┆ 0.019746       ┆ 0.0            ┆ 1.0           ┆ 1.0           ┆ 0.0           │\n",
       "│ 2996           ┆ -0.081782      ┆ 0.0            ┆ 1.0           ┆ 1.0           ┆ 0.0           │\n",
       "│ 2284           ┆ -0.061768      ┆ 0.0            ┆ 1.0           ┆ 0.0           ┆ 1.0           │\n",
       "│ 2710           ┆ -0.292353      ┆ 0.0            ┆ 1.0           ┆ 1.0           ┆ 0.0           │\n",
       "│ 1651           ┆ -0.470861      ┆ 0.0            ┆ 1.0           ┆ 1.0           ┆ 0.0           │\n",
       "└────────────────┴────────────────┴────────────────┴───────────────┴───────────────┴───────────────┘"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier[:-1].fit_transform(df_train).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a0ecc5f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=None, shuffle=False),\n",
       "             estimator=Pipeline(steps=[(&#x27;transformer&#x27;,\n",
       "                                        ColumnTransformer(transformers=[(&#x27;passthrough&#x27;,\n",
       "                                                                         &#x27;passthrough&#x27;,\n",
       "                                                                         [&#x27;text_length&#x27;]),\n",
       "                                                                        (&#x27;minmaxscaler&#x27;,\n",
       "                                                                         MinMaxScaler(feature_range=(-1,\n",
       "                                                                                                     1)),\n",
       "                                                                         [&#x27;compound_sentiment&#x27;]),\n",
       "                                                                        (&#x27;onehotencoder&#x27;,\n",
       "                                                                         OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                                       sparse_output=False),\n",
       "                                                                         [&#x27;sfw&#x27;,\n",
       "                                                                          &#x27;post_type&#x27;])])),\n",
       "                                       (&#x27;classifier&#x27;,\n",
       "                                        DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;))]),\n",
       "             param_grid={&#x27;classifier__max_depth&#x27;: range(1, 6),\n",
       "                         &#x27;transformer__minmaxscaler&#x27;: [MinMaxScaler(feature_range=(-1,\n",
       "                                                                                   1)),\n",
       "                                                       &#x27;drop&#x27;]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;GridSearchCV<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.model_selection.GridSearchCV.html\">?<span>Documentation for GridSearchCV</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=None, shuffle=False),\n",
       "             estimator=Pipeline(steps=[(&#x27;transformer&#x27;,\n",
       "                                        ColumnTransformer(transformers=[(&#x27;passthrough&#x27;,\n",
       "                                                                         &#x27;passthrough&#x27;,\n",
       "                                                                         [&#x27;text_length&#x27;]),\n",
       "                                                                        (&#x27;minmaxscaler&#x27;,\n",
       "                                                                         MinMaxScaler(feature_range=(-1,\n",
       "                                                                                                     1)),\n",
       "                                                                         [&#x27;compound_sentiment&#x27;]),\n",
       "                                                                        (&#x27;onehotencoder&#x27;,\n",
       "                                                                         OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                                       sparse_output=False),\n",
       "                                                                         [&#x27;sfw&#x27;,\n",
       "                                                                          &#x27;post_type&#x27;])])),\n",
       "                                       (&#x27;classifier&#x27;,\n",
       "                                        DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;))]),\n",
       "             param_grid={&#x27;classifier__max_depth&#x27;: range(1, 6),\n",
       "                         &#x27;transformer__minmaxscaler&#x27;: [MinMaxScaler(feature_range=(-1,\n",
       "                                                                                   1)),\n",
       "                                                       &#x27;drop&#x27;]})</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">best_estimator_: Pipeline</label><div class=\"sk-toggleable__content fitted\"><pre>Pipeline(steps=[(&#x27;transformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;text_length&#x27;]),\n",
       "                                                 (&#x27;minmaxscaler&#x27;, &#x27;drop&#x27;,\n",
       "                                                  [&#x27;compound_sentiment&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse_output=False),\n",
       "                                                  [&#x27;sfw&#x27;, &#x27;post_type&#x27;])])),\n",
       "                (&#x27;classifier&#x27;,\n",
       "                 DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;, max_depth=4))])</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;transformer: ColumnTransformer<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.compose.ColumnTransformer.html\">?<span>Documentation for transformer: ColumnTransformer</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>ColumnTransformer(transformers=[(&#x27;passthrough&#x27;, &#x27;passthrough&#x27;, [&#x27;text_length&#x27;]),\n",
       "                                (&#x27;minmaxscaler&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;compound_sentiment&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                               sparse_output=False),\n",
       "                                 [&#x27;sfw&#x27;, &#x27;post_type&#x27;])])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">passthrough</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;text_length&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">passthrough</label><div class=\"sk-toggleable__content fitted\"><pre>passthrough</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">minmaxscaler</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;compound_sentiment&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">drop</label><div class=\"sk-toggleable__content fitted\"><pre>drop</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">onehotencoder</label><div class=\"sk-toggleable__content fitted\"><pre>[&#x27;sfw&#x27;, &#x27;post_type&#x27;]</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;OneHotEncoder<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.OneHotEncoder.html\">?<span>Documentation for OneHotEncoder</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse_output=False)</pre></div> </div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;DecisionTreeClassifier<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.tree.DecisionTreeClassifier.html\">?<span>Documentation for DecisionTreeClassifier</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>DecisionTreeClassifier(class_weight=&#x27;balanced&#x27;, max_depth=4)</pre></div> </div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=None, shuffle=False),\n",
       "             estimator=Pipeline(steps=[('transformer',\n",
       "                                        ColumnTransformer(transformers=[('passthrough',\n",
       "                                                                         'passthrough',\n",
       "                                                                         ['text_length']),\n",
       "                                                                        ('minmaxscaler',\n",
       "                                                                         MinMaxScaler(feature_range=(-1,\n",
       "                                                                                                     1)),\n",
       "                                                                         ['compound_sentiment']),\n",
       "                                                                        ('onehotencoder',\n",
       "                                                                         OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                                       sparse_output=False),\n",
       "                                                                         ['sfw',\n",
       "                                                                          'post_type'])])),\n",
       "                                       ('classifier',\n",
       "                                        DecisionTreeClassifier(class_weight='balanced'))]),\n",
       "             param_grid={'classifier__max_depth': range(1, 6),\n",
       "                         'transformer__minmaxscaler': [MinMaxScaler(feature_range=(-1,\n",
       "                                                                                   1)),\n",
       "                                                       'drop']})"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, y_train = df_train.drop(\"flair_text\"), df_train[\"flair_text\"]\n",
    "\n",
    "cv = GridSearchCV(\n",
    "    classifier,\n",
    "    param_grid={\n",
    "        \"classifier__max_depth\": range(1, 6),\n",
    "        \"transformer__minmaxscaler\": [MinMaxScaler((-1, 1)), \"drop\"],\n",
    "    },\n",
    "    cv=StratifiedKFold(5)\n",
    ")\n",
    "cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ba597439",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (564, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>upvote_ratio</th><th>post_type</th><th>flair_text</th><th>flair_text_predicted</th></tr><tr><td>f64</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>0.6</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.8</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.33</td><td>&quot;WIBTA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.91</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.75</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.29</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Asshole&quot;</td></tr><tr><td>0.92</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.95</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.95</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr><tr><td>0.93</td><td>&quot;AITA&quot;</td><td>&quot;Not the A-hole&quot;</td><td>&quot;Not the A-hole&quot;</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (564, 4)\n",
       "┌──────────────┬───────────┬────────────────┬──────────────────────┐\n",
       "│ upvote_ratio ┆ post_type ┆ flair_text     ┆ flair_text_predicted │\n",
       "│ ---          ┆ ---       ┆ ---            ┆ ---                  │\n",
       "│ f64          ┆ str       ┆ str            ┆ str                  │\n",
       "╞══════════════╪═══════════╪════════════════╪══════════════════════╡\n",
       "│ 0.6          ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.8          ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.33         ┆ WIBTA     ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.91         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.75         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ …            ┆ …         ┆ …              ┆ …                    │\n",
       "│ 0.29         ┆ AITA      ┆ Not the A-hole ┆ Asshole              │\n",
       "│ 0.92         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.95         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.95         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "│ 0.93         ┆ AITA      ┆ Not the A-hole ┆ Not the A-hole       │\n",
       "└──────────────┴───────────┴────────────────┴──────────────────────┘"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = (\n",
    "    df_train\n",
    "    .select(pl.col(\"upvote_ratio\", \"post_type\", \"flair_text\"))\n",
    "    .with_columns(\n",
    "        pl.Series(name=\"flair_text_predicted\", values=cv.predict(df_train))\n",
    "    )\n",
    ")\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "6288f8ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (2, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>flair_text_predicted</th><th>count</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>&quot;Not the A-hole&quot;</td><td>450</td></tr><tr><td>&quot;Asshole&quot;</td><td>114</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (2, 2)\n",
       "┌──────────────────────┬───────┐\n",
       "│ flair_text_predicted ┆ count │\n",
       "│ ---                  ┆ ---   │\n",
       "│ str                  ┆ u32   │\n",
       "╞══════════════════════╪═══════╡\n",
       "│ Not the A-hole       ┆ 450   │\n",
       "│ Asshole              ┆ 114   │\n",
       "└──────────────────────┴───────┘"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[\"flair_text_predicted\"].value_counts().sort(\"count\", descending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "581bbabd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classifier__max_depth': 4, 'transformer__minmaxscaler': 'drop'}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c3c840ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><style>\n",
       ".dataframe > thead > tr,\n",
       ".dataframe > tbody > tr {\n",
       "  text-align: right;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       "</style>\n",
       "<small>shape: (5, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>feature_names_out</th><th>importances</th></tr><tr><td>str</td><td>f64</td></tr></thead><tbody><tr><td>&quot;passthrough__text_length&quot;</td><td>0.945209</td></tr><tr><td>&quot;onehotencoder__sfw_False&quot;</td><td>0.0</td></tr><tr><td>&quot;onehotencoder__sfw_True&quot;</td><td>0.0</td></tr><tr><td>&quot;onehotencoder__post_type_AITA&quot;</td><td>0.054791</td></tr><tr><td>&quot;onehotencoder__post_type_WIBTA&quot;</td><td>0.0</td></tr></tbody></table></div>"
      ],
      "text/plain": [
       "shape: (5, 2)\n",
       "┌────────────────────────────────┬─────────────┐\n",
       "│ feature_names_out              ┆ importances │\n",
       "│ ---                            ┆ ---         │\n",
       "│ str                            ┆ f64         │\n",
       "╞════════════════════════════════╪═════════════╡\n",
       "│ passthrough__text_length       ┆ 0.945209    │\n",
       "│ onehotencoder__sfw_False       ┆ 0.0         │\n",
       "│ onehotencoder__sfw_True        ┆ 0.0         │\n",
       "│ onehotencoder__post_type_AITA  ┆ 0.054791    │\n",
       "│ onehotencoder__post_type_WIBTA ┆ 0.0         │\n",
       "└────────────────────────────────┴─────────────┘"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importances = pl.DataFrame({\n",
    "    \"feature_names_out\": cv.best_estimator_[:-1].get_feature_names_out(),\n",
    "    \"importances\": cv.best_estimator_[-1].feature_importances_,\n",
    "})\n",
    "importances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8da8b2da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7553191489361702"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.score(df_train.drop(\"flair_text\"), df_train[\"flair_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79e1a70f",
   "metadata": {},
   "source": [
    "Well, we have a prediction... but there's lots of work to do!\n",
    "\n",
    "- The score is okay, but probably can be improved\n",
    "- We only used a handful of features\n",
    "- We did very little feature engineering\n",
    "- We would like to try out other models\n",
    "- What if we have more data?\n",
    "- How do we keep track of experiments?\n",
    "\n",
    "We could complicate this notebook more and more, but it's already getting out of hand...\n",
    "\n",
    "Let's start taking steps towards a more structured approach! ✨"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}