hackernewsposts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exploring Hacker News Posts\n",
    "\n",
    "This is a Data analysis project where we will be analyzing Hacker News posts. We are specifically interested in posts where the titles begin with 'Ask HN' or 'Show HN'.\n",
    "\n",
    "We will be comparing these two types of posts and try to see if Ask HN or Show HN receive more comments on average.\n",
    "\n",
    "And, Do posts created at a certain time receive more comments on average compared to posting on other times.\n",
    "\n",
    "Dataset documentation found [here](https://www.kaggle.com/hacker-news/hacker-news-posts)\n",
    "\n",
    "# Data Collection (kaggle set)\n",
    "\n",
    "Below we will,\n",
    "- import the needed modules\n",
    "- open the dataset\n",
    "- read the csv file\n",
    "- save the data as a list of lists\n",
    "- separate the header from the data for ease of manipulation\n",
    "- print the header\n",
    "- print first 5 rows to get a feel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']\n",
      "\n",
      "\n",
      "[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]\n"
     ]
    }
   ],
   "source": [
    "from csv import reader\n",
    "import datetime as dt\n",
    "\n",
    "opened_file = open('hacker_news.csv')\n",
    "read_file = reader(opened_file)\n",
    "hn = list(read_file)\n",
    "headers = hn[0]\n",
    "hn = hn[1:]\n",
    "\n",
    "print(headers) # Header row\n",
    "print('\\n')\n",
    "print(hn[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning\n",
    "\n",
    "`Modelling the data to fit our need`\n",
    "\n",
    "Below we will isolate only data with the Ask HN or Show HN starting keywords, then we will store them in a list of lists.\n",
    "\n",
    "We will separate each category to ensure understandability in the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ask hn: 1744\n",
      "show hn: 1162\n",
      "others: 17194\n"
     ]
    }
   ],
   "source": [
    "# storage lists\n",
    "ask_posts = []\n",
    "show_posts = []\n",
    "other_posts = []\n",
    "\n",
    "# NOTE: append posts not the title.\n",
    "for row in hn:\n",
    "    title = row[1] # Title is in second column index 1.\n",
    "    if title.lower().startswith('ask hn'):\n",
    "        ask_posts.append(row)\n",
    "    elif title.lower().startswith('show hn'):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)\n",
    "        \n",
    "# Checking the number of each category\n",
    "print('ask hn: ' + str(len(ask_posts)))\n",
    "print('show hn: ' + str(len(show_posts)))\n",
    "print('others: ' + str(len(other_posts)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['12296411',\n",
       "  'Ask HN: How to improve my personal website?',\n",
       "  '',\n",
       "  '2',\n",
       "  '6',\n",
       "  'ahmedbaracat',\n",
       "  '8/16/2016 9:55'],\n",
       " ['10610020',\n",
       "  'Ask HN: Am I the only one outraged by Twitter shutting down share counts?',\n",
       "  '',\n",
       "  '28',\n",
       "  '29',\n",
       "  'tkfx',\n",
       "  '11/22/2015 13:43'],\n",
       " ['11610310',\n",
       "  'Ask HN: Aby recent changes to CSS that broke mobile?',\n",
       "  '',\n",
       "  '1',\n",
       "  '1',\n",
       "  'polskibus',\n",
       "  '5/2/2016 10:14'],\n",
       " ['12210105',\n",
       "  'Ask HN: Looking for Employee #3 How do I do it?',\n",
       "  '',\n",
       "  '1',\n",
       "  '3',\n",
       "  'sph130',\n",
       "  '8/2/2016 14:20'],\n",
       " ['10394168',\n",
       "  'Ask HN: Someone offered to buy my browser extension from me. What now?',\n",
       "  '',\n",
       "  '28',\n",
       "  '17',\n",
       "  'roykolak',\n",
       "  '10/15/2015 16:38']]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First 5 in ask posts\n",
    "ask_posts[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['10627194',\n",
       "  'Show HN: Wio Link  ESP8266 Based Web of Things Hardware Development Platform',\n",
       "  'https://iot.seeed.cc',\n",
       "  '26',\n",
       "  '22',\n",
       "  'kfihihc',\n",
       "  '11/25/2015 14:03'],\n",
       " ['10646440',\n",
       "  'Show HN: Something pointless I made',\n",
       "  'http://dn.ht/picklecat/',\n",
       "  '747',\n",
       "  '102',\n",
       "  'dhotson',\n",
       "  '11/29/2015 22:46'],\n",
       " ['11590768',\n",
       "  'Show HN: Shanhu.io, a programming playground powered by e8vm',\n",
       "  'https://shanhu.io',\n",
       "  '1',\n",
       "  '1',\n",
       "  'h8liu',\n",
       "  '4/28/2016 18:05'],\n",
       " ['12178806',\n",
       "  'Show HN: Webscope  Easy way for web developers to communicate with Clients',\n",
       "  'http://webscopeapp.com',\n",
       "  '3',\n",
       "  '3',\n",
       "  'fastbrick',\n",
       "  '7/28/2016 7:11'],\n",
       " ['10872799',\n",
       "  'Show HN: GeoScreenshot  Easily test Geo-IP based web pages',\n",
       "  'https://www.geoscreenshot.com/',\n",
       "  '1',\n",
       "  '9',\n",
       "  'kpsychwave',\n",
       "  '1/9/2016 20:45']]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First 5 in show posts\n",
    "show_posts[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n",
    "\n",
    "Below we shall determine if ask posts or show posts receive more comments on average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.038417431192661\n"
     ]
    }
   ],
   "source": [
    "# Find total number of comments in ask posts.\n",
    "total_ask_comments = 0\n",
    "\n",
    "for post in ask_posts:\n",
    "    num_comments = int(post[4])\n",
    "    total_ask_comments += num_comments\n",
    "\n",
    "# Compute the average comments in ask posts.\n",
    "avg_ask_comments = total_ask_comments / len(ask_posts)\n",
    "# show\n",
    "print(avg_ask_comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.31669535283993\n"
     ]
    }
   ],
   "source": [
    "# Find total number of comments in show posts.\n",
    "total_show_comments = 0\n",
    "\n",
    "for post in show_posts:\n",
    "    num_comments = int(post[4])\n",
    "    total_show_comments += num_comments\n",
    "\n",
    "# Compute the average comments in show posts.\n",
    "avg_show_comments = total_show_comments / len(show_posts)\n",
    "# show\n",
    "print(avg_show_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Findings\n",
    "\n",
    "As we can observe from the computations above, `Ask Posts` with an Average value of 14 (approx.) does have a larger number of commenters per post on average versus `Show Posts` with 10 (approx.).\n",
    "\n",
    "Therefore, here on out we shall now focus only on the Ask Posts for further deeper analysis.\n",
    "\n",
    "We shall now check if posts created at a certain `time` are more likely to get or attract more comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'09': 45, '13': 85, '10': 59, '14': 107, '16': 108, '23': 68, '12': 73, '17': 100, '15': 116, '21': 109, '20': 80, '02': 58, '18': 109, '03': 54, '05': 46, '19': 110, '01': 60, '22': 71, '08': 48, '04': 47, '00': 55, '06': 44, '07': 34, '11': 58}\n",
      "\n",
      "\n",
      "{'09': 251, '13': 1253, '10': 793, '14': 1416, '16': 1814, '23': 543, '12': 687, '17': 1146, '15': 4477, '21': 1745, '20': 1722, '02': 1381, '18': 1439, '03': 421, '05': 464, '19': 1188, '01': 683, '22': 479, '08': 492, '04': 337, '00': 447, '06': 397, '07': 267, '11': 641}\n",
      "\n",
      "\n",
      "24\n",
      "24\n"
     ]
    }
   ],
   "source": [
    "result_list = []\n",
    "\n",
    "for post in ask_posts:\n",
    "    created_at = post[6]\n",
    "    num_comments = int(post[4])\n",
    "    list_append = [created_at, num_comments]\n",
    "    result_list.append(list_append)\n",
    "\n",
    "# print(result_list[:10])\n",
    "\n",
    "counts_by_hour = {}\n",
    "comments_by_hour = {}\n",
    "\n",
    "for row in result_list:\n",
    "    date_time = row[0]\n",
    "    num_comment = row[1]\n",
    "    date_time = dt.datetime.strptime(date_time, \"%m/%d/%Y %H:%M\")\n",
    "    hour = date_time.strftime('%H')\n",
    "    if hour not in counts_by_hour:\n",
    "        counts_by_hour[hour] = 1\n",
    "        comments_by_hour[hour] = num_comment\n",
    "    else:\n",
    "        counts_by_hour[hour] += 1\n",
    "        comments_by_hour[hour] += num_comment\n",
    "       \n",
    "print(counts_by_hour)\n",
    "print('\\n')\n",
    "print(comments_by_hour)\n",
    "print('\\n')\n",
    "print(len(counts_by_hour))\n",
    "print(len(comments_by_hour))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Average comments per post in each Hour of the Day\n",
    "\n",
    "The two dictionaries above namely:\n",
    "\n",
    "`counts_by_hour` - contains number of ask posts created during each hour of the day.\n",
    "\n",
    "`comments_by_hour` - contains number of comments ask posts created each hour of the day.\n",
    "\n",
    "We will now use these dictionaries to calculate the average number of comments for posts created during each hour of the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24\n",
      "\n",
      "\n",
      "[['09', 5.5777777777777775], ['13', 14.741176470588234], ['10', 13.440677966101696], ['14', 13.233644859813085], ['16', 16.796296296296298], ['23', 7.985294117647059], ['12', 9.41095890410959], ['17', 11.46], ['15', 38.5948275862069], ['21', 16.009174311926607], ['20', 21.525], ['02', 23.810344827586206], ['18', 13.20183486238532], ['03', 7.796296296296297], ['05', 10.08695652173913], ['19', 10.8], ['01', 11.383333333333333], ['22', 6.746478873239437], ['08', 10.25], ['04', 7.170212765957447], ['00', 8.127272727272727], ['06', 9.022727272727273], ['07', 7.852941176470588], ['11', 11.051724137931034]]\n"
     ]
    }
   ],
   "source": [
    "avg_by_hour = []\n",
    "\n",
    "for hour in counts_by_hour:\n",
    "    avg_by_hour.append([hour, comments_by_hour[hour] / counts_by_hour[hour]])\n",
    " \n",
    "# Testing the list of list results        \n",
    "print(len(avg_by_hour)) # if 24 - complete.\n",
    "print('\\n')\n",
    "print(avg_by_hour) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the results needed But we need to enhance it's readability. Therefore, we will sort the data further to enhance ease of use and readability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5.5777777777777775, '09'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [16.796296296296298, '16'], [7.985294117647059, '23'], [9.41095890410959, '12'], [11.46, '17'], [38.5948275862069, '15'], [16.009174311926607, '21'], [21.525, '20'], [23.810344827586206, '02'], [13.20183486238532, '18'], [7.796296296296297, '03'], [10.08695652173913, '05'], [10.8, '19'], [11.383333333333333, '01'], [6.746478873239437, '22'], [10.25, '08'], [7.170212765957447, '04'], [8.127272727272727, '00'], [9.022727272727273, '06'], [7.852941176470588, '07'], [11.051724137931034, '11']]\n",
      "\n",
      "\n",
      "24\n"
     ]
    }
   ],
   "source": [
    "# Column swap avg_by_hour\n",
    "swap_avg_by_hour = []\n",
    "\n",
    "for row in avg_by_hour:\n",
    "    swap_avg_by_hour.append([row[1], row[0]])\n",
    "\n",
    "print(swap_avg_by_hour)\n",
    "print('\\n')\n",
    "print(len(swap_avg_by_hour))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[38.5948275862069, '15'], [23.810344827586206, '02'], [21.525, '20'], [16.796296296296298, '16'], [16.009174311926607, '21'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [13.20183486238532, '18'], [11.46, '17'], [11.383333333333333, '01'], [11.051724137931034, '11'], [10.8, '19'], [10.25, '08'], [10.08695652173913, '05'], [9.41095890410959, '12'], [9.022727272727273, '06'], [8.127272727272727, '00'], [7.985294117647059, '23'], [7.852941176470588, '07'], [7.796296296296297, '03'], [7.170212765957447, '04'], [6.746478873239437, '22'], [5.5777777777777775, '09']]\n",
      "Top 5\n",
      "[[38.5948275862069, '15'], [23.810344827586206, '02'], [21.525, '20'], [16.796296296296298, '16'], [16.009174311926607, '21']]\n"
     ]
    }
   ],
   "source": [
    "# Sorting Algorithm to show in descending order\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse = True)\n",
    "\n",
    "print(sorted_swap)\n",
    "print('Top 5')\n",
    "print(sorted_swap[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 Hours for Ask Posts Comments\n",
      "\n",
      "\n",
      "15:00: 38.59 average comments per post\n",
      "02:00: 23.81 average comments per post\n",
      "20:00: 21.52 average comments per post\n",
      "16:00: 16.80 average comments per post\n",
      "21:00: 16.01 average comments per post\n"
     ]
    }
   ],
   "source": [
    "print(\"Top 5 Hours for Ask Posts Comments\")\n",
    "print('\\n')\n",
    "for row in sorted_swap[:5]:\n",
    "    hour = row[1]\n",
    "    avg_comments = row[0]\n",
    "    time_obj = dt.datetime.strptime(hour, '%H')\n",
    "    time = time_obj.strftime('%H:%M')\n",
    "    result = '{}: {:.2f} average comments per post'.format(time, avg_comments)\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Based on the results of our analysis we can now conclude that the `Hour of the day` with the highest chance of receiving comments is at `3:00 P.M. Eastern Time, US`\n",
    "\n",
    "Locally this is at exactly `3:00 A.M. PH Time`.\n",
    "\n",
    "Other times of the day (EST) are 2 AM, 8 PM, 4 PM and lastly 9 PM. \n",
    "\n",
    "Conversion to Philippine time is just the opposite meridian of EST e.g. PM becomes AM or AM becomes PM."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}