single_kanji_data_gen.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DaKanjiRecognizer - Single Kanji CNN : Create dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import the needed libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "#std lib\n",
    "import sys\n",
    "import os\n",
    "import random\n",
    "import math\n",
    "import multiprocessing as mp\n",
    "import gc\n",
    "import time\n",
    "import datetime\n",
    "from typing import Tuple, List\n",
    "from shutil import copy\n",
    "\n",
    "from tqdm import tqdm\n",
    "import tensorflow as tf\n",
    "\n",
    "#reading the dataset\n",
    "from etldr.etl_data_reader import ETLDataReader\n",
    "from etldr.etl_character_groups import ETLCharacterGroups\n",
    "from etldr.etl_data_names import ETLDataNames\n",
    "\n",
    "from DataGenerator import generate_images, check_font_char_support\n",
    "\n",
    "#data handling\n",
    "import PIL\n",
    "from PIL import Image as PImage\n",
    "from PIL import ImageFilter, ImageFont, ImageDraw\n",
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "#plotting/showing graphics\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import Image\n",
    "\n",
    "#define a font to show japanese characters in matplotlib figures\n",
    "import matplotlib.font_manager as fm\n",
    "show_sample_font = fm.FontProperties(fname=os.path.join(\"..\", \"fonts\", \"NotoSerifCJKjp-Regular.otf\"), size=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [ETL Character data set](http://etlcdb.db.aist.go.jp/) which I am using is a data set with multiple sub sets (ETL1 - ETL7, ETL8B, ETL8G, ETL9B and ETL9G). <br/>\n",
    "After unpacking the data set I renamed all folders and files to have a uniform naming scheme: \"ETLX/ETLX_Y\". \"X\" is the number of the subset and Y the part of the subset. Also ETL7S was removed (ETL7L just smaller), the following renaming was also done: <br/>\n",
    "ETL8B $\\rightarrow$ ETL1, ETL8G $\\rightarrow$ ETL9, ETL9B $\\rightarrow$ ETL10 and ETL9G $\\rightarrow$ ETL11.<br/>\n",
    "This leads to the following data set structure: <br/> \n",
    "\n",
    "| name  |   type  |                    content                                              |   res   | Bit depth |    code    | samples perlabel | total samples |\n",
    "|:-----:|:-------:|:-----------------------------------------------------------------------:|:-------:|:---------:|:----------:|:----------------:|:-------------:|\n",
    "| ETL1  | M-Type  | Numbers <br/> Roman <br/> Symbols <br/> Katakana                        |  64x63  |     4     | JIS X 0201 |   ~1400          |     141319    |\n",
    "| ETL2  | K-Type  | Hiragana <br/> Katakana <br/> Kanji <br/> Roman <br/> Symbols           |  60x60  |     6     |    CO59    |     ~24          |      52796    |\n",
    "| ETL3  | C-Type  | Numeric <br/> Capital Roman <br/> Symbols                               |  72x76  |     4     | JIS X 0201 |     200          |       9600    |\n",
    "| ETL4  | C-Type  | Hiragana                                                                |  72x76  |     4     | JIS X 0201 |     120          |       6120    |\n",
    "| ETL5  | C-Type  | Katakana                                                                |  72x76  |     4     | JIS X 0201 |    ~200          |      10608    |\n",
    "| ETL6  | M-Type  | Katakana <br/> Symbols                                                  |  64x63  |     4     | JIS X 0201 |    1383          |     157662    |\n",
    "| ETL7  | M-Type  | Hiragana <br/> Symbols                                                  |  64x63  |     4     | JIS X 0201 |     160          |      16800    |\n",
    "| ETL8  | 8B-Type | Hiragana <br/> Kanji                                                    |  64x63  |     1     | JIS X 0208 |     160          |     157662    |\n",
    "| ETL9  | 8G-Type | Hiragana <br/> Kanji                                                    | 128x127 |     4     | JIS X 0208 |     200          |     607200    |\n",
    "| ETL10 | 9B-Type | Hiragana <br/> Kanji                                                    |  64x63  |     1     | JIS X 0208 |     160          |     152960    |\n",
    "| ETL11 | 9G-Type | Hiragana <br/> Kanji                                                    | 128x127 |     4     | JIS X 0208 |     200          |     607200    |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the provided data set is distributed in a proprietary binary data format and therefore hard to handle I created a ```ETL_data_reader```-package. This package can be found [here](https://github.com/CaptainDario/ETLCDB_data_reader).\n",
    "The specific dataformat is C-struct like for types: M, 8B, 8G, 9B, 9G. But the types C and K are 6-bit encoded. All codes can be found on the [official website.](http://etlcdb.db.aist.go.jp/file-formats-and-sample-unpacking-code)\n",
    "I used the [struct module](https://docs.python.org/3/library/struct.html) and the [bitstring module](https://pypi.org/project/bitstring/) to unpack the binary data. <br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First an instance of the ```ERL_data_reader``` -class is needed.\n",
    "The path parameter should lead to the folder in which all parts of the ETL data set can be found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "path = \"Z:\\data_sets\\etlcdb_binary\"\n",
    "reader = ETLDataReader(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a convenience function for showing characters and their label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_image(img : np.array, label : str):\n",
    "    plt.figure(figsize=(2.2, 2.2))\n",
    "    plt.title(label=label, font=show_sample_font)\n",
    "    plt.axis(\"off\")\n",
    "    plt.imshow(img.astype(np.float64), cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now load all samples which contain Kanji, Hiragana and Katakana."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ffe4168d0c4c44f7bc8301bc3315ffe7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/127 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1745185, 64, 64, 1) (1745185,)\n"
     ]
    }
   ],
   "source": [
    "types = [ETLCharacterGroups.kanji, ETLCharacterGroups.katakana, ETLCharacterGroups.hiragana]\n",
    "x, y = reader.read_dataset_whole(types, 16)\n",
    "print(x.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the loaded data we can take a look at the class distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAR8klEQVR4nO3df6hkZ33H8ffHJEZRMRuzXcJu6CZ1QSK0MVxiRJHW4OaHpZuClUipiw0stBEUWtq1QuNP0EK1DWgkbUI3Yo2pP0gwtnEbA9I/THJXY8yPxr1qJFlidnWTqEhto9/+Mc+N43Xm3rm7c3/t837BMOc855lzvs+cuZ+Ze+bMTKoKSVIfnrPWBUiSVo+hL0kdMfQlqSOGviR1xNCXpI6cvNYFLOaMM86o7du3r3UZkrShHDhw4AdVtXnUsnUd+tu3b2d2dnaty5CkDSXJ98Yt8/COJHXE0Jekjhj6ktQRQ1+SOmLoS1JHJgr9JI8k+WaSe5PMtrbTk+xPcrBdb2rtSXJNkrkk9yU5f2g9u1v/g0l2r8yQJEnjLOeV/u9V1XlVNdPm9wJ3VNUO4I42D3ApsKNd9gDXwuBJArgaeCVwAXD1/BOFJGl1HM/hnV3Avja9D7h8qP3GGvgqcFqSM4GLgf1VdbSqngT2A5ccx/YlScs0aegX8KUkB5LsaW1bqurxNv19YEub3go8OnTbx1rbuPZfkWRPktkks0eOHJmwPEnSJCb9RO5rqupQkt8A9if57+GFVVVJpvJrLFV1HXAdwMzMjL/wIklTNNEr/ao61K4PA59ncEz+iXbYhnZ9uHU/BJw1dPNtrW1cuyRplSwZ+klekORF89PATuB+4FZg/gyc3cAtbfpW4C3tLJ4LgafbYaDbgZ1JNrU3cHe2NknSKpnk8M4W4PNJ5vv/a1X9R5J7gJuTXAl8D3hT6/9F4DJgDvgp8FaAqjqa5H3APa3fe6vq6NRGIklaUtbzD6PPzMyU37IpScuT5MDQ6fW/wk/kSlJHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjoycegnOSnJ15N8oc2fneSuJHNJPp3kua391DY/15ZvH1rHO1v7w0kunvpoJEmLWs4r/bcDDw3Nfwj4SFW9FHgSuLK1Xwk82do/0vqR5FzgCuDlwCXAx5KcdHzlS5KWY6LQT7INeAPwz20+wOuAz7Qu+4DL2/SuNk9bflHrvwu4qap+VlXfBeaAC6YwBknShCZ9pf8PwF8Bv2jzLwGeqqpn2vxjwNY2vRV4FKAtf7r1f7Z9xG0kSatgydBP8vvA4ao6sAr1kGRPktkks0eOHFmNTUpSNyZ5pf9q4A+SPALcxOCwzj8CpyU5ufXZBhxq04eAswDa8hcDPxxuH3GbZ1XVdVU1U1UzmzdvXvaAJEnjLRn6VfXOqtpWVdsZvBH75ar6Y+BO4I2t227gljZ9a5unLf9yVVVrv6Kd3XM2sAO4e2ojkSQt6eSlu4z118BNSd4PfB24vrVfD3wiyRxwlMETBVX1QJKbgQeBZ4Crqurnx7F9SdIyZfAifH2amZmp2dnZtS5DkjaUJAeqambUMj+RK0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSR7kN/+97bnr3Mz0vSiar70Jeknhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjqyZOgneV6Su5N8I8kDSd7T2s9OcleSuSSfTvLc1n5qm59ry7cPreudrf3hJBev2KgkSSNN8kr/Z8Drqup3gPOAS5JcCHwI+EhVvRR4Eriy9b8SeLK1f6T1I8m5wBXAy4FLgI8lOWmKY5EkLWHJ0K+Bn7TZU9qlgNcBn2nt+4DL2/SuNk9bflGStPabqupnVfVdYA64YBqDkCRNZqJj+klOSnIvcBjYD3wbeKqqnmldHgO2tumtwKMAbfnTwEuG20fcZnhbe5LMJpk9cuTIsgckSRpvotCvqp9X1XnANgavzl+2UgVV1XVVNVNVM5s3b16pzUhSl5Z19k5VPQXcCbwKOC3JyW3RNuBQmz4EnAXQlr8Y+OFw+4jbSJJWwSRn72xOclqbfj7weuAhBuH/xtZtN3BLm761zdOWf7mqqrVf0c7uORvYAdw9pXFIkiZw8tJdOBPY1860eQ5wc1V9IcmDwE1J3g98Hbi+9b8e+ESSOeAogzN2qKoHktwMPAg8A1xVVT+f7nAkSYtZMvSr6j7gFSPav8OIs2+q6n+APxqzrg8AH1h+mZKkafATuZLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JElQz/JWUnuTPJgkgeSvL21n55kf5KD7XpTa0+Sa5LMJbkvyflD69rd+h9MsnvlhiVJGmWSV/rPAH9RVecCFwJXJTkX2AvcUVU7gDvaPMClwI522QNcC4MnCeBq4JXABcDV808UkqTVsWToV9XjVfW1Nv1j4CFgK7AL2Ne67QMub9O7gBtr4KvAaUnOBC4G9lfV0ap6EtgPXDLNwUiSFresY/pJtgOvAO4CtlTV423R94EtbXor8OjQzR5rbePaF25jT5LZJLNHjhxZTnmSpCVMHPpJXgh8FnhHVf1oeFlVFVDTKKiqrquqmaqa2bx58zRWKUlqJgr9JKcwCPxPVtXnWvMT7bAN7fpwaz8EnDV0822tbVy7JGmVTHL2ToDrgYeq6sNDi24F5s/A2Q3cMtT+lnYWz4XA0+0w0O3AziSb2hu4O1ubJGmVnDxBn1cDfwJ8M8m9re1vgA8CNye5Evge8Ka27IvAZcAc8FPgrQBVdTTJ+4B7Wr/3VtXRaQxCkjSZJUO/qv4LyJjFF43oX8BVY9Z1A3DDcgqUJE2Pn8iVpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSRbkN/+97b1roESVp13Ya+JPXI0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSR5YM/SQ3JDmc5P6httOT7E9ysF1vau1Jck2SuST3JTl/6Da7W/+DSXavzHAkSYuZ5JX+vwCXLGjbC9xRVTuAO9o8wKXAjnbZA1wLgycJ4GrglcAFwNXzTxSSpNWzZOhX1VeAowuadwH72vQ+4PKh9htr4KvAaUnOBC4G9lfV0ap6EtjPrz+RSJJW2LEe099SVY+36e8DW9r0VuDRoX6PtbZx7b8myZ4ks0lmjxw5cozlSZJGOe43cquqgJpCLfPru66qZqpqZvPmzdNarSSJYw/9J9phG9r14dZ+CDhrqN+21jauXZK0io419G8F5s/A2Q3cMtT+lnYWz4XA0+0w0O3AziSb2hu4O1ubJGkVnbxUhySfAn4XOCPJYwzOwvkgcHOSK4HvAW9q3b8IXAbMAT8F3gpQVUeTvA+4p/V7b1UtfHNYkrTClgz9qnrzmEUXjehbwFVj1nMDcMOyqjtO2/fexiMffAPb9972bNvCeUnqiZ/IlaSOGPqS1BFDX5I6YuhLUkdO+NA/ljdtF95m+97bRq5nvm2xbSy1/YXrPtY3mcfVuJxahtczad9R613OGI73TfVx9924/TVqbGv5xv60alnqsXisj9HjuW9W4n5dbL9Oc7uL/b0fzzrWgxM+9KdpsTAc9Ud3PKG40g+Y5T6xTbKe5YbHsQTUsZrWOlfqSWrSbY0KuEnv92MNrUle3CzXcv6ORvWZtKbl1D7Nx+M0njRWiqEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXVsF6+d6VjWYlvvdn2tZzbaMs+XOJPZv/ucVR7ZPcdpJ1HY+VWOdi2xne3rF+o+eon7Acbl94+3HbG9V31PS4+2fU9kaNcdR2FvYdNdZxYxw1loVjXarWUetYbL3L/ZK3xX5SdNxPjy51m1H7ceG6lhrHcoz74sOF+23Sx/XCdYxqP9b6lnocTlsGP2u7Ps3MzNTs7Owx336pP0Z/P1fSenU8oZ/kQFXNjFrm4R1J6oihL0kd6S70PXwjqWfdhb4k9czQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjqx66Ce5JMnDSeaS7F3t7UtSz1Y19JOcBHwUuBQ4F3hzknNXswZJ6tlqv9K/AJirqu9U1f8CNwG7VrmGZ03jxxokaSNZ7V/O2go8OjT/GPDK4Q5J9gB72uxPkjx8HNs7A/jBcdx+rW30+sExrBeOYe0tq/586Li29ZvjFqy7n0usquuA66axriSz4349ZiPY6PWDY1gvHMPaWy/1r/bhnUPAWUPz21qbJGkVrHbo3wPsSHJ2kucCVwC3rnINktStVT28U1XPJHkbcDtwEnBDVT2wgpucymGiNbTR6wfHsF44hrW3LupPVa11DZKkVeInciWpI4a+JHXkhAz9jfRVD0keSfLNJPcmmW1tpyfZn+Rgu97U2pPkmjau+5Kcv0Y135DkcJL7h9qWXXOS3a3/wSS717j+dyc51PbDvUkuG1r2zlb/w0kuHmpfs8dZkrOS3JnkwSQPJHl7a99I+2HcGDbMvkjyvCR3J/lGG8N7WvvZSe5q9Xy6nbhCklPb/Fxbvn2psU1dVZ1QFwZvEH8bOAd4LvAN4Ny1rmuReh8BzljQ9nfA3ja9F/hQm74M+HcgwIXAXWtU82uB84H7j7Vm4HTgO+16U5vetIb1vxv4yxF9z22PoVOBs9tj66S1fpwBZwLnt+kXAd9qtW6k/TBuDBtmX7T784Vt+hTgrnb/3gxc0do/DvxZm/5z4ONt+grg04uNbSVqPhFf6a+rr3o4RruAfW16H3D5UPuNNfBV4LQkZ652cVX1FeDogubl1nwxsL+qjlbVk8B+4JIVL56x9Y+zC7ipqn5WVd8F5hg8xtb0cVZVj1fV19r0j4GHGHzifSPth3FjGGfd7Yt2f/6kzZ7SLgW8DvhMa1+4H+b3z2eAi5KE8WObuhMx9Ed91cNiD6S1VsCXkhzI4CsoALZU1eNt+vvAlja9nse23JrX41je1g593DB/WIQNUH87RPAKBq8yN+R+WDAG2ED7IslJSe4FDjN40vw28FRVPTOinmdrbcufBl7CKo7hRAz9jeY1VXU+g28evSrJa4cX1uB/vw11Xu1GrBm4Fvgt4DzgceDv17SaCSV5IfBZ4B1V9aPhZRtlP4wYw4baF1X186o6j8E3DFwAvGxtK1rciRj6G+qrHqrqULs+DHyewYPmifnDNu36cOu+nse23JrX1Viq6on2x/sL4J/45b/W67b+JKcwCMtPVtXnWvOG2g+jxrAR9wVAVT0F3Am8isHhs/kPvw7X82ytbfmLgR+yimM4EUN/w3zVQ5IXJHnR/DSwE7ifQb3zZ1HsBm5p07cCb2lnYlwIPD30r/xaW27NtwM7k2xq/77vbG1rYsF7I3/IYD/AoP4r2lkXZwM7gLtZ48dZOw58PfBQVX14aNGG2Q/jxrCR9kWSzUlOa9PPB17P4L2JO4E3tm4L98P8/nkj8OX2H9m4sU3fSr+7vRYXBmcqfIvBsbV3rXU9i9R5DoN37L8BPDBfK4NjfHcAB4H/BE6vX54p8NE2rm8CM2tU96cY/Nv9fwyOPV55LDUDf8rgDas54K1rXP8nWn33MfgDPHOo/7ta/Q8Dl66HxxnwGgaHbu4D7m2XyzbYfhg3hg2zL4DfBr7ear0f+NvWfg6D0J4D/g04tbU/r83PteXnLDW2aV/8GgZJ6siJeHhHkjSGoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I68v++U8ahkEkW7AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "unique, counts = np.unique(y, return_counts=True)\n",
    "balance = dict(zip(unique, counts))\n",
    "\n",
    "plt.bar(range(0, len(counts)), counts, width=1.0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the data is quite imbalanced we need more data.\n",
    "First remove samples so that a class has maximum 10all_jis2_charsamples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "del_inds, cnt = [], 0\n",
    "for _x, _y in zip(x, y):\n",
    "    ind = np.where(unique == _y)\n",
    "    if(counts[ind] > 1000):\n",
    "            del_inds.append(cnt)\n",
    "            counts[ind] -= 1\n",
    "    cnt += 1\n",
    "\n",
    "x = np.delete(x, del_inds, axis=0)\n",
    "y = np.delete(y, del_inds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAARJ0lEQVR4nO3dfYxcV3nH8e/TOC+8NXaSlZXapusUqyhCLXFXwQiEKtyGxFR1KgUUVDVuaslSG1poisRSpIa2f5RULRGRUJCL0zoogqSGKlYNpa4ThPpHDGsIiRM3eBMwseXEm1egiELap3/M2WRYZl9m7u7Mzp7vRxrNveeee+9z5o5/O3tmdhyZiSSpDj836AIkSf1j6EtSRQx9SaqIoS9JFTH0JakiqwZdwFwuuuiiHB0dHXQZkjRUjhw58nRmjnTatqxDf3R0lImJiUGXIUlDJSJOzLbN6R1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUkXlDPyJuj4gzEXG0re2CiDgYEcfL/ZrSHhFxa0RMRsSDEbG5bZ8dpf/xiNixNMORJM1lIa/0/wm4ckbbOHAoMzcBh8o6wFXApnLbBdwGrR8SwE3Am4DLgZumf1BIkvpn3tDPzK8Az85o3g7sLct7gavb2u/IlvuB1RFxMfAO4GBmPpuZzwEH+dkfJJKkJdbrnP7azDxdlp8E1pbldcATbf1OlrbZ2n9GROyKiImImJiamuqxvPmNjh946X761t4uSStR4zdys/Vfby3af7+Vmbszcywzx0ZGOn51hCSpR72G/lNl2oZyf6a0nwI2tPVbX9pma5ck9VGvob8fmP4Ezg7gnrb268qneLYAL5RpoC8BV0TEmvIG7hWlTZLUR/N+y2ZEfAb4deCiiDhJ61M4HwXujoidwAng3aX7F4BtwCTwQ+B6gMx8NiL+Gvha6fdXmTnzzWFJ0hKbN/Qz8z2zbNraoW8CN8xynNuB27uqTpK0qPyLXEmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFVnRoT86fuCl++lbe7sk1WZFh74k6acZ+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqSKPQj4g/jYiHI+JoRHwmIs6LiI0RcTgiJiPirog4p/Q9t6xPlu2jizICSdKC9Rz6EbEO+BNgLDPfAJwFXAvcDNySma8DngN2ll12As+V9ltKP0lSHzWd3lkFvCIiVgGvBE4Dbwf2le17gavL8vayTtm+NSKi4fklSV3oOfQz8xTwd8B3aYX9C8AR4PnMfLF0OwmsK8vrgCfKvi+W/hfOPG5E7IqIiYiYmJqa6rU8SVIHTaZ31tB69b4R+AXgVcCVTQvKzN2ZOZaZYyMjI00PJ0lq02R65zeAb2fmVGb+BPg88BZgdZnuAVgPnCrLp4ANAGX7+cAzDc4vSepSk9D/LrAlIl5Z5ua3Ao8A9wHXlD47gHvK8v6yTtl+b2Zmg/NLkrrUZE7/MK03ZL8OPFSOtRv4IHBjREzSmrPfU3bZA1xY2m8ExhvULUnqwar5u8wuM28CbprR/DhweYe+PwLe1eR8kqRm/ItcSaqIoS9JFVnxoT86fqDxPqPjBzoeZ7ptrnPMd/6Zx+6l3rlq7KaW9uMstG+n43Yzhl7H22n/+R7H2cbWtIYmFquW+Z6LvT5Hmzw2S/G4znVdF/O8c/17b3KM5WDFh74k6WWGviRVxNCXpIoY+l1YrDm65TrX10k37xN0Ozfcy/xzr5bLtWs699xp/nqhj3uvc9ILee+qW/Nd+/nGtNCauql9MZ+Pi/GewFIx9CWpIoZ+l7p5hdLkFdJSvypY6CeUNDheDy0FQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0JekijQK/YhYHRH7IuK/IuJYRLw5Ii6IiIMRcbzcryl9IyJujYjJiHgwIjYvzhAkSQvV9JX+x4F/y8zXA78KHAPGgUOZuQk4VNYBrgI2ldsu4LaG55Ykdann0I+I84G3AXsAMvPHmfk8sB3YW7rtBa4uy9uBO7LlfmB1RFzc6/klSd1r8kp/IzAF/GNEfCMiPhURrwLWZubp0udJYG1ZXgc80bb/ydImSeqTJqG/CtgM3JaZlwH/zctTOQBkZgLZzUEjYldETETExNTUVIPyJEkzNQn9k8DJzDxc1vfR+iHw1PS0Tbk/U7afAja07b++tP2UzNydmWOZOTYyMtKgPEnSTD2HfmY+CTwREb9cmrYCjwD7gR2lbQdwT1neD1xXPsWzBXihbRpIktQHqxru/8fAnRFxDvA4cD2tHyR3R8RO4ATw7tL3C8A2YBL4YekrSeqjRqGfmQ8AYx02be3QN4EbmpxPktSMf5ErSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfakPRscPDLoECTD0Jakqhr4kVcTQl6SKGPqSlq3p90KW83siy7m2Tgx9SaqIoS9JFTH0Jakihr4kVWTVoAtYzkbHD/Cdj76zY/tC9l3IsZpYimPOdZ728833GMy2vf1Yndpn7j/b+Tr17bQ82+PT6XydxtjpPDP7dhrrbGPsNJaZY52v1k7HmOu4sz0+c51rrus3rb3ffPt0uo4zjzXfOLrR6TydrttCn9czj9Gpvdf65nseLrbGr/Qj4qyI+EZE/GtZ3xgRhyNiMiLuiohzSvu5ZX2ybB9teu5h10twLmSfmU/Emf8Ahu3TBpIWz2JM77wPONa2fjNwS2a+DngO2FnadwLPlfZbSj9JUh81Cv2IWA+8E/hUWQ/g7cC+0mUvcHVZ3l7WKdu3lv6SpD6JzOx954h9wN8ArwE+APw+cH95NU9EbAC+mJlviIijwJWZebJsewx4U2Y+PeOYu4BdAK997Wt/7cSJEz3XN99c68x5yLnmJSWpn5rM6UfEkcwc67St51f6EfFbwJnMPNJzZR1k5u7MHMvMsZGRkcU8tCRVr8mnd94C/HZEbAPOA34e+DiwOiJWZeaLwHrgVOl/CtgAnIyIVcD5wDMNzi9J6lLPr/Qz80OZuT4zR4FrgXsz83eB+4BrSrcdwD1leX9Zp2y/N5vMLUmSurYUf5z1QeDGiJgELgT2lPY9wIWl/UZgfAnOLUmaw6L8cVZmfhn4cll+HLi8Q58fAe9ajPNJknrj1zBIUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1Jqkh1oe/35UuqWXWhL0k1M/QlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klSRnkM/IjZExH0R8UhEPBwR7yvtF0TEwYg4Xu7XlPaIiFsjYjIiHoyIzYs1CEnSwjR5pf8i8GeZeSmwBbghIi4FxoFDmbkJOFTWAa4CNpXbLuC2BueWJPWg59DPzNOZ+fWy/H3gGLAO2A7sLd32AleX5e3AHdlyP7A6Ii7u9fySpO4typx+RIwClwGHgbWZebpsehJYW5bXAU+07XaytM081q6ImIiIiampqcUoT5JUNA79iHg18Dng/Zn5vfZtmZlAdnO8zNydmWOZOTYyMtK0PElSm0ahHxFn0wr8OzPz86X5qelpm3J/prSfAja07b6+tEmS+qTJp3cC2AMcy8yPtW3aD+woyzuAe9raryuf4tkCvNA2DSRJ6oNVDfZ9C/B7wEMR8UBp+3Pgo8DdEbETOAG8u2z7ArANmAR+CFzf4NySpB70HPqZ+Z9AzLJ5a4f+CdzQ6/kkSc35F7mSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0JekilQd+qPjB+Zcl6SVpu+hHxFXRsSjETEZEeP9Pr8k1ayvoR8RZwGfAK4CLgXeExGX9rMGSapZv1/pXw5MZubjmflj4LPA9j7XIEnVWtXn860DnmhbPwm8qb1DROwCdpXVH0TEow3OdxHwdIP9B23Y6wfHsFw4hsHrqv64udG5fnG2Df0O/Xll5m5g92IcKyImMnNsMY41CMNePziG5cIxDN5yqb/f0zungA1t6+tLmySpD/od+l8DNkXExog4B7gW2N/nGiSpWn2d3snMFyPivcCXgLOA2zPz4SU85aJMEw3QsNcPjmG5cAyDtyzqj8wcdA2SpD6p+i9yJak2hr4kVWRFhv4wfdVDRHwnIh6KiAciYqK0XRARByPieLlfU9ojIm4t43owIjYPqObbI+JMRBxta+u65ojYUfofj4gdA67/IxFxqlyHByJiW9u2D5X6H42Id7S1D+x5FhEbIuK+iHgkIh6OiPeV9mG6DrONYWiuRUScFxFfjYhvljH8ZWnfGBGHSz13lQ+uEBHnlvXJsn10vrEtusxcUTdabxA/BlwCnAN8E7h00HXNUe93gItmtP0tMF6Wx4Gby/I24ItAAFuAwwOq+W3AZuBorzUDFwCPl/s1ZXnNAOv/CPCBDn0vLc+hc4GN5bl11qCfZ8DFwOay/BrgW6XWYboOs41haK5FeTxfXZbPBg6Xx/du4NrS/kngD8vyHwGfLMvXAnfNNbalqHklvtJfCV/1sB3YW5b3Ale3td+RLfcDqyPi4n4Xl5lfAZ6d0dxtze8ADmbms5n5HHAQuHLJi2fW+mezHfhsZv5PZn4bmKT1HBvo8ywzT2fm18vy94FjtP7ifZiuw2xjmM2yuxbl8fxBWT273BJ4O7CvtM+8DtPXZx+wNSKC2ce26FZi6Hf6qoe5nkiDlsC/R8SRaH0FBcDazDxdlp8E1pbl5Ty2bmtejmN5b5n6uH16WoQhqL9MEVxG61XmUF6HGWOAIboWEXFWRDwAnKH1Q/Mx4PnMfLFDPS/VWra/AFxIH8ewEkN/2Lw1MzfT+ubRGyLibe0bs/W731B9rnYYawZuA34JeCNwGvj7gVazQBHxauBzwPsz83vt24blOnQYw1Bdi8z838x8I61vGLgceP1gK5rbSgz9ofqqh8w8Ve7PAP9C60nz1PS0Tbk/U7ov57F1W/OyGktmPlX+8f4f8A+8/Kv1sq0/Is6mFZZ3ZubnS/NQXYdOYxjGawGQmc8D9wFvpjV9Nv3Hr+31vFRr2X4+8Ax9HMNKDP2h+aqHiHhVRLxmehm4AjhKq97pT1HsAO4py/uB68onMbYAL7T9Kj9o3db8JeCKiFhTfn2/orQNxIz3Rn6H1nWAVv3Xlk9dbAQ2AV9lwM+zMg+8BziWmR9r2zQ012G2MQzTtYiIkYhYXZZfAfwmrfcm7gOuKd1mXofp63MNcG/5jWy2sS2+pX53exA3Wp9U+BatubUPD7qeOeq8hNY79t8EHp6uldYc3yHgOPAfwAX58icFPlHG9RAwNqC6P0Pr1+6f0Jp73NlLzcAf0HrDahK4fsD1f7rU9yCtf4AXt/X/cKn/UeCq5fA8A95Ka+rmQeCBcts2ZNdhtjEMzbUAfgX4Rqn1KPAXpf0SWqE9CfwzcG5pP6+sT5btl8w3tsW++TUMklSRlTi9I0mahaEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKvL/2GNUnjy/haoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "unique, counts = np.unique(y, return_counts=True)\n",
    "balance = dict(zip(unique, counts))\n",
    "\n",
    "plt.bar(range(0, len(counts)), counts, width=1.0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save etlcdb images to disk \n",
    "To use the data later with keras we save them to disk in an appropriate folder structure. <br/>\n",
    "The ETL_data_reader package provides a handy function for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5585191ed0e4f0e9c15b90be2f76c0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1630755 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "reader.save_to_file(x, y, r\"Z:\\data_sets\\dakanji_single_kanji_cnn\", name=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create samples for missing JIS-2 Kanji\n",
    "\n",
    "Because not all JIS 2 characters are in the etlcdb we need to get samples for them. <br/>\n",
    "First find the characters which are in JIS2 but not in the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "chars_to_gen = {}\n",
    "\n",
    "# add samples for the already existing classes\n",
    "for u, c in zip(unique, counts):\n",
    "    \n",
    "    if(c < 2000):\n",
    "        chars_to_gen[u] = 2000 - c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"jis2_characters.txt\", encoding=\"utf8\", mode=\"r\") as f:\n",
    "    all_jis2_chars = f.read().replace(\" \", \"\").replace(\"\\n\", \"\")\n",
    "    all_jis2_chars = list(all_jis2_chars)\n",
    "    \n",
    "missing_jis2_chars = [c for c in all_jis2_chars if c not in unique]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add samples for missing jis2 characters\n",
    "for c in missing_jis2_chars:\n",
    "    chars_to_gen[c] = 2000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy samples from DaJapanaeseDataGenerator dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "da_data_dir = r\"Z:\\data_sets\\da_japanese_data_generator\"\n",
    "\n",
    "with open(os.path.join(da_data_dir, \"encoding.txt\"), encoding=\"utf8\", mode=\"r\") as f:\n",
    "    d = eval(f.read())\n",
    "    da_data_encoding = {v : k for k, v in d.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['86', 1000]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "single_kanji_data_dir = r\"Z:\\data_sets\\dakanji_single_kanji_cnn\"\n",
    "\n",
    "with open(os.path.join(single_kanji_data_dir, \"encoding.txt\"), encoding=\"utf8\", mode=\"r\") as f:\n",
    "    single_kanji_data_encoding = eval(f.read())\n",
    "    \n",
    "    \n",
    "single_kanji_data_encoding[\"キ\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1117"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars_to_gen[\"あ\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "for char, cnt in chars_to_gen.items():\n",
    "\n",
    "    #\n",
    "    if(char not in single_kanji_data_encoding):\n",
    "        #print(char)\n",
    "        os.mkdir(os.path.join(single_kanji_data_dir, str(len(single_kanji_data_encoding))))\n",
    "        single_kanji_data_encoding[char] = [str(len(single_kanji_data_encoding)), 0]\n",
    "    \n",
    "    #\n",
    "    for i in range(cnt):\n",
    "\n",
    "        _from = os.path.join(da_data_dir, str(da_data_encoding[char]), str(i) + \".png\")\n",
    "        _to   = os.path.join(single_kanji_data_dir, single_kanji_data_encoding[char][0], str(single_kanji_data_encoding[char][1]) + \".png\")\n",
    "        #print(_from, _to)\n",
    "\n",
    "        copy(_from, _to)\n",
    "        single_kanji_data_encoding[char][1] += 1\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(single_kanji_data_dir, \"encoding.txt\"), encoding=\"utf8\", mode=\"w+\") as f:\n",
    "    f.write(str(single_kanji_data_encoding))"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "7b66e6ad73e4584ec1c4fbbc9ad8719d99baf4738cd57ba39e580b8e9994d93a"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "metadata": {
   "interpreter": {
    "hash": "7b66e6ad73e4584ec1c4fbbc9ad8719d99baf4738cd57ba39e580b8e9994d93a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}