named_entity_recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hello everyone,\n",
    "\n",
    "#### Welcome in this place. \n",
    "\n",
    "In this notebook I'll show you a really simple example of Named Entity Recognition (NER).\n",
    "\n",
    "Using NER it's possible to detect and classify named entities in a sentence.\n",
    "\n",
    "Wiki: https://en.wikipedia.org/wiki/Named-entity_recognition\n",
    "\n",
    "I'll make a comparison between NER implemented in NLTK ( Natural Language ToolKit ) and StanfordNERTagger. They are available both inside nltk in Python.\n",
    "\n",
    "NLTK Documentation: http://www.nltk.org/\n",
    "\n",
    "NLTK Book: http://www.nltk.org/book/\n",
    "\n",
    "Stanford NER Documentation: https://nlp.stanford.edu/software/CRF-NER.shtml\n",
    "\n",
    "###### If you want to run this code you have to download the stanford's component from the webpage I linked\n",
    "\n",
    "#### Let's start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:44.703933Z",
     "start_time": "2017-07-19T12:13:43.840846Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tag import StanfordNERTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:46.206083Z",
     "start_time": "2017-07-19T12:13:46.200082Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentences = [\n",
    "    'The Facebook website was launched on February 4, 2004, by Mark Zuckerberg, along with fellow Harvard College students and roommates, Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, and Chris Hughes.',\n",
    "    'Apple Inc. is an American multinational technology company headquartered in Cupertino, California.',\n",
    "    'Microsoft was founded by Paul Allen and Bill Gates on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800.',\n",
    "    'Saul Hudson better known by his stage name Slash, is a British-American musician and songwriter. He is best known as the lead guitarist of the American rock band Guns N\\' Roses.',\n",
    "    'Mario Draghi is an Italian economist who has served as President of the European Central Bank since November 2011.',\n",
    "    'Italy is a unitary parliamentary republic in Europe. Italy shares open land borders with France, Switzerland, Austria, Slovenia, San Marino and Vatican City.',\n",
    "    'Tesla is an American automaker, energy storage company, and solar panel manufacturer based in Palo Alto.',\n",
    "    'Elon Reeve Musk is a South African-born Canadian-American business magnate, investor, engineer, and inventor.',\n",
    "    'Frank Underwood is the main character of House of Cards',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:47.068169Z",
     "start_time": "2017-07-19T12:13:47.063169Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "# path = 'your_path_here'\n",
    "java_path = path + \"java.exe\"\n",
    "os.environ['JAVAHOME'] = java_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:49.413404Z",
     "start_time": "2017-07-19T12:13:49.407403Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stanfordtagger = StanfordNERTagger(\n",
    "    './stanford/classifiers/english.all.3class.distsim.crf.ser.gz',\n",
    "    './stanford/stanford-ner.jar',\n",
    "    encoding='latin1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:51.303593Z",
     "start_time": "2017-07-19T12:13:51.288591Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split2(x,by=2):\n",
    "    out = []\n",
    "    for i in range(0,len(x)):\n",
    "        if i % 2 == 0:\n",
    "            out.append(tuple(x[i:i+2]))\n",
    "    return(out)\n",
    "\n",
    "def from_sentence_to_ne(x,method='nltk'):\n",
    "    token = nltk.word_tokenize(x)\n",
    "    if method == 'nltk':\n",
    "        tag = nltk.pos_tag(token)\n",
    "        ne = nltk.ne_chunk(tag)\n",
    "        out = nltk.chunk.tree2conllstr(ne)\n",
    "        \n",
    "        out = out.split()\n",
    "        length = len(out)\n",
    "        for i in range(0,length):\n",
    "            if i % 2 != 0:\n",
    "                try:\n",
    "                    del out[i]\n",
    "                except IndexError:\n",
    "                    break\n",
    "        out = split2(out)\n",
    "    else:\n",
    "        out = stanfordtagger.tag(token) \n",
    "    return(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:13:53.838846Z",
     "start_time": "2017-07-19T12:13:53.504813Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Facebook website was launched on February 4, 2004, by Mark Zuckerberg, along with fellow Harvard College students and roommates, Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, and Chris Hughes.\n",
      "\n",
      "\n",
      "('Facebook', 'B-ORGANIZATION')\n",
      "('Mark', 'B-PERSON')\n",
      "('Zuckerberg', 'I-PERSON')\n",
      "('Harvard', 'B-ORGANIZATION')\n",
      "('College', 'I-ORGANIZATION')\n",
      "('Eduardo', 'B-PERSON')\n",
      "('Saverin', 'I-PERSON')\n",
      "('Andrew', 'B-PERSON')\n",
      "('McCollum', 'I-PERSON')\n",
      "('Dustin', 'B-PERSON')\n",
      "('Moskovitz', 'I-PERSON')\n",
      "('Chris', 'B-PERSON')\n",
      "('Hughes', 'I-PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Apple Inc. is an American multinational technology company headquartered in Cupertino, California.\n",
      "\n",
      "\n",
      "('Apple', 'B-PERSON')\n",
      "('Inc.', 'B-ORGANIZATION')\n",
      "('American', 'B-GPE')\n",
      "('Cupertino', 'B-GPE')\n",
      "('California', 'B-GPE')\n",
      "\n",
      "\n",
      "\n",
      "Microsoft was founded by Paul Allen and Bill Gates on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800.\n",
      "\n",
      "\n",
      "('Microsoft', 'B-PERSON')\n",
      "('Paul', 'B-PERSON')\n",
      "('Allen', 'I-PERSON')\n",
      "('Bill', 'B-PERSON')\n",
      "('Gates', 'I-PERSON')\n",
      "('BASIC', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "Saul Hudson better known by his stage name Slash, is a British-American musician and songwriter. He is best known as the lead guitarist of the American rock band Guns N' Roses.\n",
      "\n",
      "\n",
      "('Saul', 'B-PERSON')\n",
      "('Hudson', 'B-PERSON')\n",
      "('Slash', 'B-PERSON')\n",
      "('American', 'B-GPE')\n",
      "('Guns', 'B-PERSON')\n",
      "('N', 'I-PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Mario Draghi is an Italian economist who has served as President of the European Central Bank since November 2011.\n",
      "\n",
      "\n",
      "('Mario', 'B-PERSON')\n",
      "('Draghi', 'B-ORGANIZATION')\n",
      "('Italian', 'B-GPE')\n",
      "('European', 'B-ORGANIZATION')\n",
      "('Central', 'I-ORGANIZATION')\n",
      "('Bank', 'I-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "Italy is a unitary parliamentary republic in Europe. Italy shares open land borders with France, Switzerland, Austria, Slovenia, San Marino and Vatican City.\n",
      "\n",
      "\n",
      "('Italy', 'B-GPE')\n",
      "('Europe', 'B-GPE')\n",
      "('Italy', 'B-GPE')\n",
      "('France', 'B-GPE')\n",
      "('Switzerland', 'B-GPE')\n",
      "('Austria', 'B-GSP')\n",
      "('Slovenia', 'B-GPE')\n",
      "('San', 'B-PERSON')\n",
      "('Marino', 'I-PERSON')\n",
      "('Vatican', 'B-PERSON')\n",
      "('City', 'I-PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Tesla is an American automaker, energy storage company, and solar panel manufacturer based in Palo Alto.\n",
      "\n",
      "\n",
      "('Tesla', 'B-GPE')\n",
      "('American', 'B-GPE')\n",
      "('Palo', 'B-GPE')\n",
      "('Alto', 'I-GPE')\n",
      "\n",
      "\n",
      "\n",
      "Elon Reeve Musk is a South African-born Canadian-American business magnate, investor, engineer, and inventor.\n",
      "\n",
      "\n",
      "('Elon', 'B-PERSON')\n",
      "('Reeve', 'B-PERSON')\n",
      "('Musk', 'I-PERSON')\n",
      "('South', 'B-GPE')\n",
      "\n",
      "\n",
      "\n",
      "Frank Underwood is the main character of House of Cards\n",
      "\n",
      "\n",
      "('Frank', 'B-PERSON')\n",
      "('Underwood', 'I-PERSON')\n",
      "('House', 'B-ORGANIZATION')\n",
      "('Cards', 'B-GPE')\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 314 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'nltk'\n",
    "for s in sentences:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:14:18.605323Z",
     "start_time": "2017-07-19T12:13:56.013064Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Facebook website was launched on February 4, 2004, by Mark Zuckerberg, along with fellow Harvard College students and roommates, Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, and Chris Hughes.\n",
      "\n",
      "\n",
      "('Facebook', 'ORGANIZATION')\n",
      "('Mark', 'PERSON')\n",
      "('Zuckerberg', 'PERSON')\n",
      "('Harvard', 'ORGANIZATION')\n",
      "('College', 'ORGANIZATION')\n",
      "('Eduardo', 'PERSON')\n",
      "('Saverin', 'PERSON')\n",
      "('Andrew', 'PERSON')\n",
      "('McCollum', 'PERSON')\n",
      "('Dustin', 'PERSON')\n",
      "('Moskovitz', 'PERSON')\n",
      "('Chris', 'PERSON')\n",
      "('Hughes', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Apple Inc. is an American multinational technology company headquartered in Cupertino, California.\n",
      "\n",
      "\n",
      "('Apple', 'ORGANIZATION')\n",
      "('Inc.', 'ORGANIZATION')\n",
      "('Cupertino', 'LOCATION')\n",
      "('California', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "Microsoft was founded by Paul Allen and Bill Gates on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800.\n",
      "\n",
      "\n",
      "('Microsoft', 'ORGANIZATION')\n",
      "('Paul', 'PERSON')\n",
      "('Allen', 'PERSON')\n",
      "('Bill', 'PERSON')\n",
      "('Gates', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Saul Hudson better known by his stage name Slash, is a British-American musician and songwriter. He is best known as the lead guitarist of the American rock band Guns N' Roses.\n",
      "\n",
      "\n",
      "('Saul', 'PERSON')\n",
      "('Hudson', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Mario Draghi is an Italian economist who has served as President of the European Central Bank since November 2011.\n",
      "\n",
      "\n",
      "('Mario', 'PERSON')\n",
      "('Draghi', 'PERSON')\n",
      "('Central', 'ORGANIZATION')\n",
      "('Bank', 'ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "Italy is a unitary parliamentary republic in Europe. Italy shares open land borders with France, Switzerland, Austria, Slovenia, San Marino and Vatican City.\n",
      "\n",
      "\n",
      "('Italy', 'LOCATION')\n",
      "('Europe', 'LOCATION')\n",
      "('Italy', 'LOCATION')\n",
      "('France', 'LOCATION')\n",
      "('Switzerland', 'LOCATION')\n",
      "('Austria', 'LOCATION')\n",
      "('Slovenia', 'LOCATION')\n",
      "('San', 'LOCATION')\n",
      "('Marino', 'LOCATION')\n",
      "('Vatican', 'LOCATION')\n",
      "('City', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "Tesla is an American automaker, energy storage company, and solar panel manufacturer based in Palo Alto.\n",
      "\n",
      "\n",
      "('Tesla', 'PERSON')\n",
      "('Palo', 'LOCATION')\n",
      "('Alto', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "Elon Reeve Musk is a South African-born Canadian-American business magnate, investor, engineer, and inventor.\n",
      "\n",
      "\n",
      "('Elon', 'PERSON')\n",
      "('Reeve', 'PERSON')\n",
      "('Musk', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Frank Underwood is the main character of House of Cards\n",
      "\n",
      "\n",
      "('Frank', 'PERSON')\n",
      "('Underwood', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 22.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'stanford'\n",
    "for s in sentences:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NOTES\n",
    "\n",
    "StanfordNERTagger looks quite better than the other. Unfortunally the prediction of this model is expensive in terms of time.\n",
    "\n",
    "Let's have fun..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:14:18.611323Z",
     "start_time": "2017-07-19T12:14:18.607323Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lower_case = [s.lower() for s in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:14:18.730335Z",
     "start_time": "2017-07-19T12:14:18.615324Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the facebook website was launched on february 4, 2004, by mark zuckerberg, along with fellow harvard college students and roommates, eduardo saverin, andrew mccollum, dustin moskovitz, and chris hughes.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "apple inc. is an american multinational technology company headquartered in cupertino, california.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "microsoft was founded by paul allen and bill gates on april 4, 1975, to develop and sell basic interpreters for the altair 8800.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "saul hudson better known by his stage name slash, is a british-american musician and songwriter. he is best known as the lead guitarist of the american rock band guns n' roses.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "mario draghi is an italian economist who has served as president of the european central bank since november 2011.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "italy is a unitary parliamentary republic in europe. italy shares open land borders with france, switzerland, austria, slovenia, san marino and vatican city.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "tesla is an american automaker, energy storage company, and solar panel manufacturer based in palo alto.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "elon reeve musk is a south african-born canadian-american business magnate, investor, engineer, and inventor.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "frank underwood is the main character of house of cards\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 105 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'nltk'\n",
    "for s in lower_case:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:14:42.937756Z",
     "start_time": "2017-07-19T12:14:18.732335Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the facebook website was launched on february 4, 2004, by mark zuckerberg, along with fellow harvard college students and roommates, eduardo saverin, andrew mccollum, dustin moskovitz, and chris hughes.\n",
      "\n",
      "\n",
      "('mark', 'PERSON')\n",
      "('zuckerberg', 'PERSON')\n",
      "('andrew', 'PERSON')\n",
      "('mccollum', 'PERSON')\n",
      "('dustin', 'PERSON')\n",
      "('moskovitz', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "apple inc. is an american multinational technology company headquartered in cupertino, california.\n",
      "\n",
      "\n",
      "('american', 'LOCATION')\n",
      "('california', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "microsoft was founded by paul allen and bill gates on april 4, 1975, to develop and sell basic interpreters for the altair 8800.\n",
      "\n",
      "\n",
      "('microsoft', 'ORGANIZATION')\n",
      "('paul', 'PERSON')\n",
      "('allen', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "saul hudson better known by his stage name slash, is a british-american musician and songwriter. he is best known as the lead guitarist of the american rock band guns n' roses.\n",
      "\n",
      "\n",
      "('american', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "mario draghi is an italian economist who has served as president of the european central bank since november 2011.\n",
      "\n",
      "\n",
      "('mario', 'PERSON')\n",
      "('draghi', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "italy is a unitary parliamentary republic in europe. italy shares open land borders with france, switzerland, austria, slovenia, san marino and vatican city.\n",
      "\n",
      "\n",
      "('italy', 'LOCATION')\n",
      "('italy', 'LOCATION')\n",
      "('france', 'LOCATION')\n",
      "('switzerland', 'LOCATION')\n",
      "('austria', 'LOCATION')\n",
      "('slovenia', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "tesla is an american automaker, energy storage company, and solar panel manufacturer based in palo alto.\n",
      "\n",
      "\n",
      "('american', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "elon reeve musk is a south african-born canadian-american business magnate, investor, engineer, and inventor.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "frank underwood is the main character of house of cards\n",
      "\n",
      "\n",
      "('frank', 'PERSON')\n",
      "('underwood', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 24.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'stanford'\n",
    "for s in lower_case:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:14:42.944756Z",
     "start_time": "2017-07-19T12:14:42.939756Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "upper_case = [s.upper() for s in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:15:05.203982Z",
     "start_time": "2017-07-19T12:14:42.946756Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THE FACEBOOK WEBSITE WAS LAUNCHED ON FEBRUARY 4, 2004, BY MARK ZUCKERBERG, ALONG WITH FELLOW HARVARD COLLEGE STUDENTS AND ROOMMATES, EDUARDO SAVERIN, ANDREW MCCOLLUM, DUSTIN MOSKOVITZ, AND CHRIS HUGHES.\n",
      "\n",
      "\n",
      "('MARK', 'PERSON')\n",
      "('ZUCKERBERG', 'PERSON')\n",
      "('EDUARDO', 'PERSON')\n",
      "('SAVERIN', 'PERSON')\n",
      "('ANDREW', 'PERSON')\n",
      "('MCCOLLUM', 'PERSON')\n",
      "('DUSTIN', 'PERSON')\n",
      "('MOSKOVITZ', 'PERSON')\n",
      "('CHRIS', 'PERSON')\n",
      "('HUGHES', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "APPLE INC. IS AN AMERICAN MULTINATIONAL TECHNOLOGY COMPANY HEADQUARTERED IN CUPERTINO, CALIFORNIA.\n",
      "\n",
      "\n",
      "('APPLE', 'ORGANIZATION')\n",
      "('INC.', 'ORGANIZATION')\n",
      "('CUPERTINO', 'LOCATION')\n",
      "('CALIFORNIA', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "MICROSOFT WAS FOUNDED BY PAUL ALLEN AND BILL GATES ON APRIL 4, 1975, TO DEVELOP AND SELL BASIC INTERPRETERS FOR THE ALTAIR 8800.\n",
      "\n",
      "\n",
      "('PAUL', 'PERSON')\n",
      "('ALLEN', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "SAUL HUDSON BETTER KNOWN BY HIS STAGE NAME SLASH, IS A BRITISH-AMERICAN MUSICIAN AND SONGWRITER. HE IS BEST KNOWN AS THE LEAD GUITARIST OF THE AMERICAN ROCK BAND GUNS N' ROSES.\n",
      "\n",
      "\n",
      "('SAUL', 'PERSON')\n",
      "('HUDSON', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "MARIO DRAGHI IS AN ITALIAN ECONOMIST WHO HAS SERVED AS PRESIDENT OF THE EUROPEAN CENTRAL BANK SINCE NOVEMBER 2011.\n",
      "\n",
      "\n",
      "('MARIO', 'PERSON')\n",
      "('DRAGHI', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "ITALY IS A UNITARY PARLIAMENTARY REPUBLIC IN EUROPE. ITALY SHARES OPEN LAND BORDERS WITH FRANCE, SWITZERLAND, AUSTRIA, SLOVENIA, SAN MARINO AND VATICAN CITY.\n",
      "\n",
      "\n",
      "('ITALY', 'LOCATION')\n",
      "('EUROPE', 'LOCATION')\n",
      "('ITALY', 'LOCATION')\n",
      "('FRANCE', 'LOCATION')\n",
      "('SWITZERLAND', 'LOCATION')\n",
      "('AUSTRIA', 'LOCATION')\n",
      "('SLOVENIA', 'LOCATION')\n",
      "('SAN', 'LOCATION')\n",
      "('MARINO', 'LOCATION')\n",
      "('VATICAN', 'ORGANIZATION')\n",
      "('CITY', 'ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "TESLA IS AN AMERICAN AUTOMAKER, ENERGY STORAGE COMPANY, AND SOLAR PANEL MANUFACTURER BASED IN PALO ALTO.\n",
      "\n",
      "\n",
      "('TESLA', 'ORGANIZATION')\n",
      "('PALO', 'LOCATION')\n",
      "('ALTO', 'LOCATION')\n",
      "\n",
      "\n",
      "\n",
      "ELON REEVE MUSK IS A SOUTH AFRICAN-BORN CANADIAN-AMERICAN BUSINESS MAGNATE, INVESTOR, ENGINEER, AND INVENTOR.\n",
      "\n",
      "\n",
      "('ELON', 'PERSON')\n",
      "('REEVE', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "FRANK UNDERWOOD IS THE MAIN CHARACTER OF HOUSE OF CARDS\n",
      "\n",
      "\n",
      "('FRANK', 'PERSON')\n",
      "('UNDERWOOD', 'PERSON')\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 22.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'stanford'\n",
    "for s in upper_case:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:15:05.309993Z",
     "start_time": "2017-07-19T12:15:05.205982Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THE FACEBOOK WEBSITE WAS LAUNCHED ON FEBRUARY 4, 2004, BY MARK ZUCKERBERG, ALONG WITH FELLOW HARVARD COLLEGE STUDENTS AND ROOMMATES, EDUARDO SAVERIN, ANDREW MCCOLLUM, DUSTIN MOSKOVITZ, AND CHRIS HUGHES.\n",
      "\n",
      "\n",
      "('FACEBOOK', 'B-ORGANIZATION')\n",
      "('BY', 'B-ORGANIZATION')\n",
      "('MARK', 'B-ORGANIZATION')\n",
      "('ALONG', 'B-ORGANIZATION')\n",
      "('FELLOW', 'B-ORGANIZATION')\n",
      "('HARVARD', 'B-ORGANIZATION')\n",
      "('STUDENTS', 'B-ORGANIZATION')\n",
      "('EDUARDO', 'B-ORGANIZATION')\n",
      "('ANDREW', 'B-ORGANIZATION')\n",
      "('DUSTIN', 'B-ORGANIZATION')\n",
      "('AND', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "APPLE INC. IS AN AMERICAN MULTINATIONAL TECHNOLOGY COMPANY HEADQUARTERED IN CUPERTINO, CALIFORNIA.\n",
      "\n",
      "\n",
      "('APPLE', 'B-ORGANIZATION')\n",
      "('AMERICAN', 'B-ORGANIZATION')\n",
      "('TECHNOLOGY', 'B-ORGANIZATION')\n",
      "('CALIFORNIA', 'B-GPE')\n",
      "\n",
      "\n",
      "\n",
      "MICROSOFT WAS FOUNDED BY PAUL ALLEN AND BILL GATES ON APRIL 4, 1975, TO DEVELOP AND SELL BASIC INTERPRETERS FOR THE ALTAIR 8800.\n",
      "\n",
      "\n",
      "('MICROSOFT', 'B-ORGANIZATION')\n",
      "('WAS', 'B-ORGANIZATION')\n",
      "('PAUL', 'B-ORGANIZATION')\n",
      "('BILL', 'B-PERSON')\n",
      "('GATES', 'I-PERSON')\n",
      "('TO', 'B-ORGANIZATION')\n",
      "('DEVELOP', 'B-ORGANIZATION')\n",
      "('SELL', 'B-ORGANIZATION')\n",
      "('THE', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "SAUL HUDSON BETTER KNOWN BY HIS STAGE NAME SLASH, IS A BRITISH-AMERICAN MUSICIAN AND SONGWRITER. HE IS BEST KNOWN AS THE LEAD GUITARIST OF THE AMERICAN ROCK BAND GUNS N' ROSES.\n",
      "\n",
      "\n",
      "('SAUL', 'B-ORGANIZATION')\n",
      "('HUDSON', 'B-ORGANIZATION')\n",
      "('MUSICIAN', 'B-ORGANIZATION')\n",
      "('BEST', 'B-ORGANIZATION')\n",
      "('THE', 'B-ORGANIZATION')\n",
      "('LEAD', 'B-ORGANIZATION')\n",
      "('THE', 'B-ORGANIZATION')\n",
      "('AMERICAN', 'B-ORGANIZATION')\n",
      "('ROSES', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "MARIO DRAGHI IS AN ITALIAN ECONOMIST WHO HAS SERVED AS PRESIDENT OF THE EUROPEAN CENTRAL BANK SINCE NOVEMBER 2011.\n",
      "\n",
      "\n",
      "('MARIO', 'B-ORGANIZATION')\n",
      "('DRAGHI', 'B-ORGANIZATION')\n",
      "('ITALIAN', 'B-ORGANIZATION')\n",
      "('THE', 'B-ORGANIZATION')\n",
      "('EUROPEAN', 'B-ORGANIZATION')\n",
      "('CENTRAL', 'B-ORGANIZATION')\n",
      "('BANK', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "ITALY IS A UNITARY PARLIAMENTARY REPUBLIC IN EUROPE. ITALY SHARES OPEN LAND BORDERS WITH FRANCE, SWITZERLAND, AUSTRIA, SLOVENIA, SAN MARINO AND VATICAN CITY.\n",
      "\n",
      "\n",
      "('ITALY', 'B-GPE')\n",
      "('UNITARY', 'B-ORGANIZATION')\n",
      "('REPUBLIC', 'B-ORGANIZATION')\n",
      "('ITALY', 'B-ORGANIZATION')\n",
      "('SWITZERLAND', 'B-GPE')\n",
      "('AUSTRIA', 'B-GSP')\n",
      "('SLOVENIA', 'B-GPE')\n",
      "('SAN', 'B-ORGANIZATION')\n",
      "('MARINO', 'B-ORGANIZATION')\n",
      "('VATICAN', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "TESLA IS AN AMERICAN AUTOMAKER, ENERGY STORAGE COMPANY, AND SOLAR PANEL MANUFACTURER BASED IN PALO ALTO.\n",
      "\n",
      "\n",
      "('TESLA', 'B-GPE')\n",
      "('AMERICAN', 'B-ORGANIZATION')\n",
      "('ENERGY', 'B-ORGANIZATION')\n",
      "('AND', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "ELON REEVE MUSK IS A SOUTH AFRICAN-BORN CANADIAN-AMERICAN BUSINESS MAGNATE, INVESTOR, ENGINEER, AND INVENTOR.\n",
      "\n",
      "\n",
      "('ELON', 'B-ORGANIZATION')\n",
      "('REEVE', 'B-ORGANIZATION')\n",
      "('SOUTH', 'B-ORGANIZATION')\n",
      "('BUSINESS', 'B-ORGANIZATION')\n",
      "('MAGNATE', 'B-ORGANIZATION')\n",
      "('INVESTOR', 'B-ORGANIZATION')\n",
      "('ENGINEER', 'B-ORGANIZATION')\n",
      "('AND', 'B-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "FRANK UNDERWOOD IS THE MAIN CHARACTER OF HOUSE OF CARDS\n",
      "\n",
      "\n",
      "('FRANK', 'B-ORGANIZATION')\n",
      "('UNDERWOOD', 'B-ORGANIZATION')\n",
      "('THE', 'B-ORGANIZATION')\n",
      "('MAIN', 'B-ORGANIZATION')\n",
      "('HOUSE', 'B-ORGANIZATION')\n",
      "('OF', 'I-ORGANIZATION')\n",
      "('CARDS', 'I-ORGANIZATION')\n",
      "\n",
      "\n",
      "\n",
      "Wall time: 88 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "method = 'nltk'\n",
    "for s in upper_case:\n",
    "    app = from_sentence_to_ne(s, method=method)\n",
    "    print(s)\n",
    "    print('\\n')\n",
    "    for i in app:\n",
    "        if i[1]!='O':\n",
    "            print(i)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "StanfordNERTagger looks more efficient and also more robust. \n",
    "\n",
    "However both of them look inefficient in not standard cases.\n",
    "\n",
    "My questions are: \n",
    "\n",
    "    1) is it possible to train a NER insensitive to lower/upper-case?\n",
    "\n",
    "    2) if it's possible, how many data it requires?\n",
    "\n",
    "###### I hope you appreciate these examples. See you around,\n",
    "###### Federico"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}