ab-testing-trials.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"width:image width px; font-size:75%; text-align:right;\">\n",
    "    <img src=\"img/header_letizia-bordoni_unsplash.jpg\" width=\"width\" height=\"height\" style=\"padding-bottom:0.2em;\" />\n",
    "    <figcaption>Photo by Letizia Bordoni on Unsplash</figcaption>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A/B-testing trials\n",
    "**Development of the theoretical basis for statistical hypothesis testing with calculation of sample data sets**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Dennis Gluesenkamp*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "* [Introduction and motivation](#introduction)\n",
    "* [Theoretical background](#theory)\n",
    "* [Arithmetic approaches](#arithmetic)\n",
    "    * [Self-implementation](#arithmetic_selfimplementation)\n",
    "    * [SciPy implementation](#arithmetic_scipy)\n",
    "* [Study of model cases](#study)\n",
    "    * [Supermarket](#study_supermarket)\n",
    "    * [Wholesale](#study_wholesale)\n",
    "* [References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction and motivation<a class=\"anchor\" id=\"introduction\"></a>\n",
    "\n",
    "<table class=\"image styled\" style=\"margin-left: 0.5em\" cellspacing=\"0\" cellpadding=\"0\" align=\"right\">\n",
    "    <tbody><tr><td><img src=\"img/intro_jon-tyson_unsplash.jpg\" width=\"width\" height=\"height\" ></td></tr>\n",
    "        <tr><td class=\"caption\">Photo by Jon Tyson on Unsplash</td></tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "\n",
    "The comparison of two groups with respect to a specific effect is of great interest in a variety of fields and applications. Here are just a few examples:\n",
    "* The design of a webshop is checked between two alternatives in order to increase the conversion of visitors into customers. Exemplary question: *Do you want the button to buy to be red or blue?*\n",
    "* The effectiveness of a drug is checked on the basis of two study groups, in which one group is given the drug and the other a placebo. The question: *Does the drug really influence the course of the disease?*\n",
    "* Educational and career trajectories are compared in relation to socio-demographic factors. Exemplary question: *Do girls or boys achieve better results at school?*\n",
    "\n",
    "Especially in the field of marketing, website design or sales approach, this procedure is called **A/B testing**. This means that there are two groups of generally speaking consumers who are presented different versions of a certain product, presentation or similar between the groups. Consumer/customer behavior is then compared in terms of a success metric, such as signing a contract, purchasing or providing contact information. As a result, the provider or operator receives a clear decision support as to which version should be implemented for the future in order to maximize the addressed success. In the following the basics will be worked out and some simple examples will be calculated. The central question will be whether the observed difference between the groups can be described as **significant**. For this purpose, the **t-** respectively **Welch-test** will be introduced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theoretical background<a class=\"anchor\" id=\"theory\"></a>\n",
    "### Hypotheses<a class=\"anchor\" id=\"theory_hypotheses\"></a>\n",
    "If the comparison of two groups shows a difference in the measurable quantity, we want to know whether this difference is due to group membership or whether it is random. To stay with an example from the introduction: *Is a higher turnover achieved with the red button due to the color of the button or is this difference to the blue button purely coincidental.* To answer this question, hypotheses are formulated. The so-called **null hypothesis** describes the case that we want to reject on the basis of our experiment: The expected values of the measured variable are actually the same $\\mu_1 = \\mu_2 = \\mu$ because both samples are from the same population, although the sample means are different $\\bar{x}_1 \\neq \\bar{x}_2$. The statement disjoint to the null hypothesis is the **alternative hypothesis**. In this case it can be assumed that the samples are taken from different distributions with different expected values $\\mu_1 \\neq \\mu_2$ with respect to the measurand. Hence, it explains the difference between the empirical sample means $\\bar{x}_1 \\neq \\bar{x}_2$. The following chart illustrates these cases with samples from the same distribution on the left hand side and two different distributions with two different expected values on the right hand side."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<tr>\n",
    "    <td><img src=\"img/nullhypothesis.png\" style=\"width:85%;\"/></td>\n",
    "    <td><img src=\"img/alternativehypothesis.png\" style=\"width:85%;\"/></td>\n",
    "</tr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical testing<a class=\"anchor\" id=\"theory_testing\"></a>\n",
    "In order to test the null hypothesis and to make a statement whether the difference in the observed empirical mean values is significant, a statistical hypothesis test is carried out. If we now look at exactly two groups/samples with a sample mean difference in the relevant measured variable, we apply the **t-test**.\n",
    "\n",
    "> *The t-test checks whether two means differ significantly from each other.* [[1, p.103]](#schaefer2011)\n",
    "\n",
    "*Significant* means that there is sufficient statistical evidence *to reject the null hypothesis*. However, the reverse conclusion does not apply: If the difference is not significant, the null hypothesis is **not** proven.\n",
    "\n",
    "The t-test requires certain criteria to be met for appropriate use. These are [[1, p.113]](#schaefer2011):\n",
    "* The samples are independent, so the two groups do not influence each other's behavior.\n",
    "* The dependent variable, i.e. the measurand, is interval scaled.\n",
    "* The dependent variable is normally distributed.\n",
    "* The variances are (almost) equal.\n",
    "\n",
    "<a class=\"anchor\" id=\"violation\"></a>However, the t-test is a robust test - it can return meaningful results even if these criteria are violated. Especially noteworthy is the normal distribution of the dependent variable. In some publications, the use of the non-parametric **Wilcoxon-Mann-Whitney test** is directly recommended when the normal distribution is violated. However, studies like Rasch & Guiard [[2]](#rasch2004) show that this is not necessarily the case:\n",
    "\n",
    "> *On the other hand the t-test is so robust against non-normality that there is nearly no need to use the Wilcoxon test. [...]  Generally  the  results are such that in most practical cases the parametric approach for inferences about means is so robust that it can be recommended in nearly all applications.* [[2]](#rasch2004)\n",
    "\n",
    "In our case of independent/unpaired and exactly two samples, the **test statistic $t$** is calculated as follows [[3, p.86](#semendjajew2003); [4, p.8](#kanji2006)].\n",
    "\\\\[\n",
    "    t = \\frac{\\bar{x}_A-\\bar{x}_B}{\\sqrt{\\left(n_A-1\\right) s_A^2 + \\left(n_B-1\\right) s_B^2}} \\sqrt{\\frac{n_a+n_b-2}{\\frac{1}{n_A} + \\frac{1}{n_B}}} \\label{eq:tstatistic}\\tag{1}\n",
    "\\\\]\n",
    "\n",
    "Here, $A$ and $B$ are denoting the samples. In the equation above, the **sample/empirical mean $\\bar{x}$** is [[3, p.79]](#semendjajew2003)\n",
    "\\\\[\n",
    "    \\bar{x} = \\frac{1}{n} \\sum_i^n x_i\n",
    "\\\\]\n",
    "\n",
    "with $n$ as the sample size and $x_i$ as the individual sample values. The **unbiased sample variance $s^2$** is [[3, p.80]](#semendjajew2003)\n",
    "\\\\[\n",
    "    s^2 = \\frac{1}{n-1} \\sum_i^n \\left(x_i - \\bar{x} \\right).\n",
    "\\\\]\n",
    "\n",
    "In cases where the variances are different $\\sigma_1 \\neq \\sigma_2$ the **Welch test** is to be preferred [[5, p.208]](#benesch2012). The test statistic changes to [[6]](#welch1947)\n",
    "\\\\[\n",
    "    t = \\frac{\\bar{x}_A-\\bar{x}_B}{\\sqrt{\\frac{s_A^2}{n_A} + \\frac{s_B^2}{n_B}}}. \\label{eq:welchtest}\\tag{2}\n",
    "\\\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Significance<a class=\"anchor\" id=\"theory_significance\"></a>\n",
    "Finally, we define a critical value $\\alpha$ from which we can reject the null hypothesis with good reason, which is our actual goal. A significance level is defined for this purpose. If the probability of committing a type I error, i.e. rejecting the null hypothesis even though it actually applies, falls below this level, the null hypothesis is rejected and the difference in sample means is called significant.\n",
    "\n",
    "How exactly $\\alpha$ should be chosen has often been the subject of research and is still debated. It should also be considered what it means to falsely reject the null hypothesis. Typically, tests with error probabilities below 5% are considered significant. As we will look at examples with no further implications in the following, we will also adopt this value, so $\\alpha = 0.05$. This means, if the null hypothesis is true, the probability of rejecting must not exceed 5%.\n",
    "\n",
    "In order to determine whether we can describe a mean difference as significant, we must calculate a measure of the confidence of the null hypothesis and compare it with the level of significance. This measure indicates the probability that the observed value or a value even further in the direction of the alternative hypothesis (in our case, a larger difference in sample means) occurs, but that the null hypothesis corresponds to reality. It thus indicates the degree of support for the null hypothesis. If it is below our previously defined significance level, we consider the test result to be significant. This probability measure is called **p-value**.\n",
    "\\\\[\n",
    "    p = 2 \\left( 1 - F\\left(\\left|t\\right|,\\nu\\right) \\right) \\label{eq:pvalue}\\tag{3}\n",
    "\\\\]\n",
    "\n",
    "Since the test statistics follow a Student's t-distribution, its cumulative distribution function $F\\left(\\left|t\\right|,\\nu\\right)$ is used to calculate the p-value, with degrees of freedom $\\nu=n_A+n_B-2$. The two-sided distribution is used because one empirical mean value can be greater or smaller than the other sample mean by the same margin. For further information on the cumulative distribution function $F\\left(\\left|t\\right|,\\nu\\right)$ - shown in the chart below - please refer to the literature, for instance [[7]](#johnson2005).\n",
    "\n",
    "<div style=\"width:image width px; font-size:75%; text-align:right;\">\n",
    "    <img src=\"img/cdf.png\" width=\"width\" height=\"height\" style=\"padding-bottom:0.2em;\" />\n",
    "    <figcaption>Cumulative distribution function of Student's t-distribution</figcaption>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetic approaches<a class=\"anchor\" id=\"arithmetic\"></a>\n",
    "We want to try different ways of implementing the test. On the one hand, we will realize our own methods in the following, which will calculate the corresponding test statistics. Above all, this will allow an exact reconstruction of the calculation path and thus will also help to understand the technique. Furthermore, we examine calculations for the t-test, which are offered in third-party packages for Python and can thus be used as an out of the box solution for hypothesis testing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Self-implementation<a class=\"anchor\" id=\"arithmetic_selfimplementation\"></a>\n",
    "For our own implementation of the test statistic $t$ we only use NumPy [[8](#oliphant2006); [9](#walt2011)] and SciPy [[10](#virtanen2020)]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample means $\\bar{x}$ can be calculated on an array with the <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html\" target=\"_blank\">``mean()`` function</a>. The unbiased sample variance $s^2$ is implemented in NumPy's <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.var.html\" target=\"_blank\">``var()`` function</a>. The *Delta Degrees of Freedom* parameter has to be taken into account at this point. The default value is ``ddof = 0``. For an unbiased sample variance this has to be set to ``ddof = 1``. Finally, we need the <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.sqrt.html\" target=\"_blank\">``sqrt()`` function</a> which leads us to the following function to calculate the test statistic $t$ of equation ([1](#mjx-eqn-eq:tstatistic))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tValue(A, B):\n",
    "    n_A = len(A)\n",
    "    n_B = len(B)\n",
    "    m_A = A.mean()\n",
    "    m_B = B.mean()\n",
    "    s_A = A.var(ddof = 1)\n",
    "    s_B = B.var(ddof = 1)\n",
    "    \n",
    "    return (m_A-m_B)/(np.sqrt((n_A-1)*s_A + (n_B-1)*s_B)) * np.sqrt((n_A+n_B-2)/((1/n_A)+(1/n_B)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also define a function to calculate the Welch's t-test according to ([2](#mjx-eqn-eq:welchtest))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tValueWelch(A, B):\n",
    "    n_A = len(A)\n",
    "    n_B = len(B)\n",
    "    m_A = A.mean()\n",
    "    m_B = B.mean()\n",
    "    s_A = A.var(ddof = 1)\n",
    "    s_B = B.var(ddof = 1)\n",
    "    \n",
    "    return (m_A-m_B)/(np.sqrt(s_A/n_A + s_B/n_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the p-value we use the implementation of <a href=\"https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.t.html\" target=\"_blank\">Student's t distribution</a> in the module ``stats`` of the SciPy package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This makes the code to ([3](#mjx-eqn-eq:pvalue)) look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tpValue(t, nu):\n",
    "    return 2*(1-stats.t.cdf(abs(t), nu))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After all we combine the two function to a single one which gives a the test statistic and the p-value for two samples $A$ and $B$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tTest(A, B):\n",
    "    return tValue(A, B), tpValue(tValue(A, B), len(A)+len(B)-2)\n",
    "\n",
    "def WelchtTest(A, B):\n",
    "    return tValueWelch(A, B), tpValue(tValueWelch(A, B), len(A)+len(B)-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SciPy implementation<a class=\"anchor\" id=\"arithmetic_scipy\"></a>\n",
    "While the own implementation is helpful for learning purposes, there is also an out of the box solution in the SciPy package [[10]](#virtanen2020). The [function ``ttest_ind()``](#https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html) calculates test statistic and p-value in a two-sided test for the null hypothesis that two unpaired samples have identical expected values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=0.5835505589878616, pvalue=0.5601868657397915)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example: Two samples of random numbers, normally distributed\n",
    "A = stats.norm.rvs(loc = 0, scale = 1, size = 100)\n",
    "B = stats.norm.rvs(loc = 0, scale = 1, size = 100)\n",
    "stats.ttest_ind(A, B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5835505589878616, 0.5601868657397915)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the result of our own implementation\n",
    "tTest(A, B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to perform a Welch's t-test, the parameter ``equal_var`` has to be set to ``False``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result t-test:         Ttest_indResult(statistic=-0.3204080456889469, pvalue=0.7487709538034599)\n",
      "Result Welch's t-test: Ttest_indResult(statistic=-0.6885072415317205, pvalue=0.4914142529556038)\n"
     ]
    }
   ],
   "source": [
    "# Example: Two samples of random numbers, normally distributed, different variances and sizes\n",
    "A = stats.norm.rvs(loc = 10, scale = 1, size = 100)\n",
    "B = stats.norm.rvs(loc = 10, scale = 7, size = 500)\n",
    "print('Result t-test:         ' + str(stats.ttest_ind(A, B)))\n",
    "print('Result Welch\\'s t-test: ' + str(stats.ttest_ind(A, B, equal_var = False)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result t-test:         (-0.3204080456889469, 0.7487709538034599)\n",
      "Result Welch's t-test: (-0.6885072415317205, 0.49140041963810566)\n"
     ]
    }
   ],
   "source": [
    "# Check the result of our own implementation\n",
    "print('Result t-test:         ' + str(tTest(A, B)))\n",
    "print('Result Welch\\'s t-test: ' + str(WelchtTest(A, B)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Study of model cases<a class=\"anchor\" id=\"study\"></a>\n",
    "In the following, data sets will be evaluated in the sense of an A/B test. Since corresponding data is not always available, the data sets used below show at least two groups with different behavior, but which were not originally created as an A/B test. However, we pretend that an A/B test has been performed and we will now identify the favorable option. We set the significance level to $\\alpha = 0.05$ in all cases.\n",
    "\n",
    "Additionally, we make use of the packages ``pandas`` and ``matplotlib``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"study_supermarket\"></a>\n",
    "<div style=\"width:image width px; font-size:75%; text-align:right;\">\n",
    "    <img src=\"img/supermarket_neonbrand_unsplash.jpg\" width=\"width\" height=\"height\" style=\"padding-bottom:0.2em;\" />\n",
    "    <figcaption>Photo by NeONBRAND on Unsplash</figcaption>\n",
    "</div>\n",
    "\n",
    "### Supermarket\n",
    "**Pretended case:** In a supermarket, two groups of customers have received a different service (We do not define further in which way, such as personal greetings or similar). One group are <span style=\"color:#003f7e\">registered customers</span> (hereafter also marked and called as the <span style=\"color:#003f7e\">blue group</span>), i.e. members who have experienced a new service offering. The other group of <span style=\"color:#821626\">normal buyers</span> is the control group (hereafter also the <span style=\"color:#821626\">red group</span>), with the current standard of service. For the two groups, both goods and their prices are the same. The two service levels have been tested over different locations and the customers' shopping experience has been evaluated (rating).\n",
    "\n",
    "**Question: Does the changed service have a positive effect on customer satisfaction and sales figures and should it therefore be introduced across all markets?**\n",
    "\n",
    "**Source:** Pyae, A. (2019). Supermarket sales, Historical record of sales data in 3 different supermarkets. *Kaggle*. Retrieved 2020/04/07 from https://www.kaggle.com/aungpyaeap/supermarket-sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data import and column selection\n",
    "First, the data is imported as a pandas ``DataFrame`` and a selection of columns is made which is appropriate for the example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Invoice ID</th>\n",
       "      <th>Branch</th>\n",
       "      <th>City</th>\n",
       "      <th>Customer type</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Product line</th>\n",
       "      <th>Unit price</th>\n",
       "      <th>Quantity</th>\n",
       "      <th>Tax 5%</th>\n",
       "      <th>Total</th>\n",
       "      <th>Date</th>\n",
       "      <th>Time</th>\n",
       "      <th>Payment</th>\n",
       "      <th>cogs</th>\n",
       "      <th>gross margin percentage</th>\n",
       "      <th>gross income</th>\n",
       "      <th>Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>399</th>\n",
       "      <td>346-12-3257</td>\n",
       "      <td>B</td>\n",
       "      <td>Mandalay</td>\n",
       "      <td>Member</td>\n",
       "      <td>Male</td>\n",
       "      <td>Electronic accessories</td>\n",
       "      <td>19.24</td>\n",
       "      <td>9</td>\n",
       "      <td>8.658</td>\n",
       "      <td>181.818</td>\n",
       "      <td>3/4/2019</td>\n",
       "      <td>16:28</td>\n",
       "      <td>Cash</td>\n",
       "      <td>173.16</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>8.658</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>730-70-9830</td>\n",
       "      <td>C</td>\n",
       "      <td>Naypyitaw</td>\n",
       "      <td>Normal</td>\n",
       "      <td>Female</td>\n",
       "      <td>Home and lifestyle</td>\n",
       "      <td>70.11</td>\n",
       "      <td>6</td>\n",
       "      <td>21.033</td>\n",
       "      <td>441.693</td>\n",
       "      <td>3/14/2019</td>\n",
       "      <td>17:54</td>\n",
       "      <td>Ewallet</td>\n",
       "      <td>420.66</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>21.033</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>651</th>\n",
       "      <td>563-47-4072</td>\n",
       "      <td>B</td>\n",
       "      <td>Mandalay</td>\n",
       "      <td>Normal</td>\n",
       "      <td>Female</td>\n",
       "      <td>Health and beauty</td>\n",
       "      <td>55.81</td>\n",
       "      <td>6</td>\n",
       "      <td>16.743</td>\n",
       "      <td>351.603</td>\n",
       "      <td>1/22/2019</td>\n",
       "      <td>11:52</td>\n",
       "      <td>Cash</td>\n",
       "      <td>334.86</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>16.743</td>\n",
       "      <td>9.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>673</th>\n",
       "      <td>556-41-6224</td>\n",
       "      <td>C</td>\n",
       "      <td>Naypyitaw</td>\n",
       "      <td>Normal</td>\n",
       "      <td>Male</td>\n",
       "      <td>Health and beauty</td>\n",
       "      <td>33.64</td>\n",
       "      <td>8</td>\n",
       "      <td>13.456</td>\n",
       "      <td>282.576</td>\n",
       "      <td>2/15/2019</td>\n",
       "      <td>17:10</td>\n",
       "      <td>Credit card</td>\n",
       "      <td>269.12</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>13.456</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>641-96-3695</td>\n",
       "      <td>C</td>\n",
       "      <td>Naypyitaw</td>\n",
       "      <td>Member</td>\n",
       "      <td>Female</td>\n",
       "      <td>Fashion accessories</td>\n",
       "      <td>43.46</td>\n",
       "      <td>6</td>\n",
       "      <td>13.038</td>\n",
       "      <td>273.798</td>\n",
       "      <td>2/7/2019</td>\n",
       "      <td>17:55</td>\n",
       "      <td>Ewallet</td>\n",
       "      <td>260.76</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>13.038</td>\n",
       "      <td>8.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Invoice ID Branch       City Customer type  Gender  \\\n",
       "399  346-12-3257      B   Mandalay        Member    Male   \n",
       "372  730-70-9830      C  Naypyitaw        Normal  Female   \n",
       "651  563-47-4072      B   Mandalay        Normal  Female   \n",
       "673  556-41-6224      C  Naypyitaw        Normal    Male   \n",
       "619  641-96-3695      C  Naypyitaw        Member  Female   \n",
       "\n",
       "               Product line  Unit price  Quantity  Tax 5%    Total       Date  \\\n",
       "399  Electronic accessories       19.24         9   8.658  181.818   3/4/2019   \n",
       "372      Home and lifestyle       70.11         6  21.033  441.693  3/14/2019   \n",
       "651       Health and beauty       55.81         6  16.743  351.603  1/22/2019   \n",
       "673       Health and beauty       33.64         8  13.456  282.576  2/15/2019   \n",
       "619     Fashion accessories       43.46         6  13.038  273.798   2/7/2019   \n",
       "\n",
       "      Time      Payment    cogs  gross margin percentage  gross income  Rating  \n",
       "399  16:28         Cash  173.16                 4.761905         8.658     8.0  \n",
       "372  17:54      Ewallet  420.66                 4.761905        21.033     5.2  \n",
       "651  11:52         Cash  334.86                 4.761905        16.743     9.9  \n",
       "673  17:10  Credit card  269.12                 4.761905        13.456     9.3  \n",
       "619  17:55      Ewallet  260.76                 4.761905        13.038     8.5  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_supermarket = pd.read_csv('dat/supermarket/supermarket.csv')\n",
    "df_supermarket.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of central importance are the columns ``Customer type`` for the test and control group as well as the columns for the total price of the purchase ``Total`` and the customer satisfaction ``Rating``.\n",
    "\n",
    "Furthermore, we consider the following columns to make the case more vivid/realistic. These columns are therefore also included:\n",
    "* ``Branch``\n",
    "* ``Gender``\n",
    "* ``Product line``\n",
    "* ``Payment``\n",
    "* ``Rating``\n",
    "\n",
    "We will not use the ``Invoice ID`` because no purchase identified by an individual ID contains multiple products in this sample. To do this, we will check the length of the dataset with the number of unique IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDs contain only one row with one individual product. Please proceed!\n"
     ]
    }
   ],
   "source": [
    "if (len(df_supermarket['Invoice ID'].unique()) == len(df_supermarket['Invoice ID'])):\n",
    "    print('IDs contain only one row with one individual product. Please proceed!')\n",
    "else:\n",
    "    print('IDs can contain more than one individual item. Please aggregate the data.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``Customer type`` column should contain only two groups (we want to apply the t-test, not an analysis of variances) and the columns for the total purchase price ``Total`` and the customer satisfaction ``Rating`` should contain numerical values. We check that this is the case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are only two groups ['Member' 'Normal'] as demanded.\n",
      "There are 501 members in the dataset.\n",
      "There are 499 normal buyers in the dataset.\n",
      "The considered objective columns are numeric (float64). Please proceed!\n",
      "Invoice ID                 0\n",
      "Branch                     0\n",
      "City                       0\n",
      "Customer type              0\n",
      "Gender                     0\n",
      "Product line               0\n",
      "Unit price                 0\n",
      "Quantity                   0\n",
      "Tax 5%                     0\n",
      "Total                      0\n",
      "Date                       0\n",
      "Time                       0\n",
      "Payment                    0\n",
      "cogs                       0\n",
      "gross margin percentage    0\n",
      "gross income               0\n",
      "Rating                     0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "if (len(df_supermarket['Customer type'].unique()) == 2):\n",
    "    print('There are only two groups ' + str(df_supermarket['Customer type'].unique()) + ' as demanded.')\n",
    "else:\n",
    "    print('There are not exactly two groups! Please check the data.')\n",
    "    \n",
    "print('There are ' + str(len(df_supermarket[df_supermarket['Customer type'] == 'Member'])) + ' members in the dataset.')\n",
    "print('There are ' + str(len(df_supermarket[df_supermarket['Customer type'] == 'Normal'])) + ' normal buyers in the dataset.')\n",
    "\n",
    "if ((df_supermarket['Total'].dtype == 'float64') & (df_supermarket['Rating'].dtype == 'float64')):\n",
    "    print('The considered objective columns are numeric (float64). Please proceed!')\n",
    "else:\n",
    "    print('Check the type of the objective columns.')\n",
    "    \n",
    "print(df_supermarket.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, there are only the test and control group. The sizes of the group are almost equal. Furthermore, the measurands are floats and there is no missing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Total purchase price\n",
    "We start by looking at the distribution of the total purchase price as a histogram for the two groups. For the sake of simplicity, we inspect here purely from a visual point of view using the histogram and do not further quantify the similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA68AAAIpCAYAAABNBg6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7TtZV0v/vdHNl4ICQk0QREvZF5GKu5Ms7QjXagkPGdkWBpY1j5dLPXYMbPycsqyM8qysstOTTimYuZR6W6QkB0zN7ukSP2ByhYUAREE3GqCn98f87t1sdhrsxasNeezNq/XGHvM+b0+n+9cz2DxXs/z/c7q7gAAAMDI7rDoAgAAAOCWCK8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwBWpaouqKpvWXQdi1RV/7WqLqmq66vqEQto/51V9SPzbncvdXRVPWDRdaxFVf1BVf3iousA4NYTXgFIVV1cVd+6bN3Tq+pde5a7+yHd/c5bOM8xU7DZskGlLtqvJ3lmdx/c3f+yfONaQ91mDIGbVXf/WHf/0qLrAODWE14B2DQGCMX3SXLBgmvYcAN8zuuqqg5YdA0A3HbCKwCrsnR0tqoeVVU7quraqrq8ql4+7Xbu9HrNNLX2MVV1h6r6haraVVVXVNXpVfWVS857yrTtqqr6xWXtvLiq3lxVr6uqa5M8fWr73VV1TVVdVlW/W1V3XHK+rqqfqKoLq+q6qvqlqrr/dMy1VfWmpfsvu8a91lpVd6qq65MckOR9VfWhvRy759rfN137ydP6H62qi6rqU1X19qo6cqX9q+puVfXnVXVlVV09vb/XKn8+ez6rM6br3llVD1v2uTxgyfJrq+qXp/ffUlWXVtXPVtUnkvxxVR1QVS+oqg9N5zuvqu69pMlvnT7jq6vqlVVV07nuX1VnTz/PT1bVn1TVoUva/dmq+th0zg9W1fFLPvvnT+1dNf2cDlvhWvfU+4KpjYur6qnLru33q+ovq+ozSf7L0uud9jmpqv516hMfqqoTpvVfWVWvnvrWx6rql2sKv1X1gKo6p6o+PbV7xmp+NgCsD+EVgFvjFUle0d2HJLl/kjdN6x83vR46Ta19d5KnT//+S5L7JTk4ye8mSVU9OMnvJXlqknsm+cokRy1r66Qkb05yaJI/SXJjkuckOTzJY5Icn+Qnlh1zQpJHJnl0kucl2T61ce8kD03y/Stc115r7e7Pd/fB0z4P6+77Lz+wux+3ZPvB3X1GVT0hya8m+b7p+nYleeNK+2f2e/mPMxvhPTrJZ/d8Vqt0UpI/TXJYktcneWtVHbjKY796Ou4+SbYl+R+ZfU7fleSQJD+cZPeS/Z+Y5OuTPGy6vu+Y1td0zUcmeVBmn/mLk6SqHpjkmUm+vrvvOh1z8XTcTyd5UpLHT8deneSVt1Dv4Zn1l1OTbJ/Ov8cPJHlpkrsmedfSA6vqUUlOT/I/M+tXj1tSx2lJbkjygCSPSPLtSfbcZ/xLSf42yd2S3CvJ7+yjPgDWmfAKwB5vnUYzr6mqazILlSv5QpIHVNXh3X19d//TPvZ9apKXd/eHu/v6JD+X5Ck1m5r6vUnO7O53dfd/Jnlhkl52/Lu7+63d/cXu/mx3n9fd/9TdN3T3xUn+MLPAs9Svdfe13X1Bkn9P8rdT+59O8leZhZK11nprPDXJa7p7Z3d/fjrfY6rqmL3t3N1Xdfefdffu7r4us/C1/Nr25bzufnN3fyHJy5PcObMAvxpfTPKiKah/NrPA9gvd/cGeeV93X7Vk/5d19zXd/dEkf5/k4dM1XNTd75jOc+VUx55ruDHJnZI8uKoO7O6Lu3vPKPZ/T/Lz3X3p9Fm9OMn33sJn/4tTO+ck+YvMQvQeb+vuf5z6zeeWHfeMzH4u75i2f6y7P1BV90jynUme3d2f6e4rkvxmkqdMx30hs3B/ZHd/rrvfFQDmRngFYI8ndfehe/7l5qOZSz0jydck+UBVvbeqnriPfY/MbMRxj11JtiS5x7Ttkj0bunt3kqtyU5csXaiqr5mm036iZlOJfyWzEbilLl/y/rN7WT44e7evWm+Nm5xvCsRX5eajy0mSqjqoqv5wmrZ8bWbTsA+t1d+zufSz/GKSS6caVuPKZSHv3kluNj16iU8seb8702daVXevqjdOU26vTfK6TD+f7r4oybMzC6ZXTPvtqe8+Sf7vkj+evD+zsLvSZ391d39myfKu3PRaL8nKVrq2+yQ5MMllS+r4wyR3n7Y/L7OR5X+u2dO3f3gfbQCwzoRXANasuy/s7u/P7H/qfy3Jm6vqK3LzUdMk+XhmoWCPozOblnl5kssym36ZJKmquyT5quXNLVv+/SQfSHLsNG35BZkFivWwr1pv8/mmz+irknxshf2fm+SBSb5hurY9U4tXe31fuie1qu6Q2Wf78WnV7iQHLdn3q5cdu/xzviSzKeFr9avTub5uuoanZUn93f367v6mzD6Xzqz/7GnvO5f+AaW779zdK31Wd5s+zz2OzpevdW/Xs9RK13ZJks8nOXxJDYd090Om2j/R3T/a3UdmNlL8e+Vp0QBzI7wCsGZV9bSqOmIa3btmWn1jkiszm356vyW7vyHJc6rqvlV1cGYjpWd09w2Z3ct6YlV9Y80eovSS3HJQu2uSa5NcX1Vfm+TH1+3C9l3ralyem17765P8UFU9vKruNJ3vPdN0573tf9fMRoavmR5W9KI11v/Iqvpv01TbZ2cWxPZM6f7XJD8wPYjphNzydORXJfmlqjq2Zr6uqpb/YWFv7prk+ukajsrsvtIks3teq+oJ02fxucyu9cZp8x8keWlV3Wfa94iqOukW2npJVd2xqr45s3tw/3QV9SXJqzP7uRw/PSjqqKr62u6+LLN7Wn+jqg6Ztt2/qh4/1fTk+vIDtK7OLCDfuPcmAFhvwisAt8YJSS6o2RN4X5HkKdM9gLszu0/zH6dpl49O8pok/yezKbAfySy0/FSSTPek/lRmDzG6LMl1Sa7ILHSt5GcyexjPdUn+KMl6PvF1xVpX6cVJTpuu/fu6+6wkv5jkzzK7vvvny/dP3mz/JL+V5C5JPplZ6PzrNdb/tiQnZxasfjDJf5vuf02SZyU5MbM/Njw1yVtv4Vwvz+xBXH+b2R8LXj3VdktekuS4JJ/O7D7UtyzZdqckL8vs+j6R2cj9C6Ztr0jy9iR/W1XXZXb937CPdj6R2XV+PLMHef1Yd39gFfWlu/85yQ9ldj/rp5Ocky+PkJ+S5I5J/mM6/5sze9hWMntA1Xumfv/2JM/q7o+spk0Abrvq3tesGgCYn2m085rMpgQLBWtQVS9O8oDuftqia9loVfUtSV7X3av6GiEA9g9GXgFYqKo6cXpQ0Vck+fUk/5Yvf20JAEAS4RWAxTsps6mfH09ybGZTkE0LAgBuwrRhAAAAhmfkFQAAgOEJrwAAAAxvy6ILWKvDDz+8jznmmEWXAQAAwAY477zzPtndRyxfv+nC6zHHHJMdO3YsugwAAAA2QFXt2tt604YBAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOFtWXQBbB7bz9wxt7a2nbh1bm0BAADjM/IKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8LYsugBum+1n7lh0CQAAABvOyCsAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8LYsugA2l93nnD2XdnZe/aG5tLPHcaecPNf2AACAtTHyCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhbVl0AXB7s/P0M+bW1nGnnDy3tgAAYCMZeQUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhjfX8FpVB1TVv1TVn0/Lh1XVO6rqwun1bvOsBwAAgM1h3iOvz0ry/iXLz09yVncfm+SsaRkAAABuYm7htaruleS7k7xqyeqTkpw2vT8tyZPmVQ8AAACbxzxHXn8ryfOSfHHJunt092VJMr3efY71AAAAsEnMJbxW1ROTXNHd593K47dV1Y6q2nHllVeuc3UAAACMbl4jr49N8j1VdXGSNyZ5QlW9LsnlVXXPJJler9jbwd29vbu3dvfWI444Yk4lAwAAMIq5hNfu/rnuvld3H5PkKUnO7u6nJXl7klOn3U5N8rZ51AMAAMDmsujveX1Zkm+rqguTfNu0DAAAADexZd4Ndvc7k7xzen9VkuPnXQMAAACby6JHXgEAAOAWCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGN7cnzbM+tt9ztmLLgEAAGBDGXkFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeFsWXQCMYOfpZyy6BAAAYB+MvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPC2LLoAWLRzz9+16BI2xOO+7j6LLgEAANaNkVcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwvC2LLgDY/LafuWNubW07cevc2gIAYBxGXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGt2XRBeyvdp5+xlza2X3+rrm0AwAAsEhGXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADG8u4bWq7lxV/1xV76uqC6rqJdP6w6rqHVV14fR6t3nUAwAAwOYyr5HXzyd5Qnc/LMnDk5xQVY9O8vwkZ3X3sUnOmpYBAADgJuYSXnvm+mnxwOlfJzkpyWnT+tOSPGke9QAAALC5bJlXQ1V1QJLzkjwgySu7+z1VdY/uvixJuvuyqrr7CsduS7ItSY4++uh5lXyrbT9zR3afv2vRZQAAAOw35vbApu6+sbsfnuReSR5VVQ9dw7Hbu3trd2894ogjNq5IAAAAhjT3pw139zVJ3pnkhCSXV9U9k2R6vWLe9QAAADC+eT1t+IiqOnR6f5ck35rkA0nenuTUabdTk7xtHvUAAACwuczrntd7Jjltuu/1Dkne1N1/XlXvTvKmqnpGko8mefKc6gEAAGATmUt47e7zkzxiL+uvSnL8PGoAAABg85r7Pa8AAACwVsIrAAAAwxNeAQAAGJ7wCgAAwPDm9bRhgHWx/cwdc2tr24lb59YWAAD7ZuQVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4W1ZdAGwN+eev2vRJQAAAAMx8goAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMNbdXitqievsP57168cAAAAuLm1jLy+eoX129ejEAAAAFjJllvaoaruN729Q1XdN0kt2Xy/JJ/biMIAAABgj1sMr0kuStKZhdYPLdv2iSQvXueaAAAA4CZuMbx29x2SpKrO6e7Hb3xJAAAAcFOrvudVcAUAAGBRVjNtOEky3e/60iQPT3Lw0m3dffQ61wUAAABfsurwmuT1md3z+twkuzemHAAAALi5tYTXhyR5bHd/caOKAQAAgL1Zy/e8npvkERtVCAAAAKxkLSOvFyf5m6p6S2ZfkfMl3f3C9SwKAAAAllpLeP2KJGcmOTDJvTemHAAAALi5VYfX7v6hjSwEAAAAVrKWr8q530rbuvvD61MOAAAA3Nxapg1flKST1JJ1Pb0esG4VAQAAwDJrmTZ8kycTV9VXJ3lRkn9Y76IAAABgqbV8Vc5NdPcnkjw7ya+uXzkAAABwc2uZNrw3D0xy0HoUAjCanaefMbe2jjvl5Lm1BQCwGa3lgU3/kC/f45rMQutDkvyv9S4KAAAAllrLyOurli1/Jsn7uvvCdawHAAAAbmYtD2w6bSMLAQAAgJWs+oFNVXVgVb2kqj5cVZ+bXl9SVXfcyAIBAABgLdOG/3eSRyX5sSS7ktwnyS8mOSTJc9a/NAAAAJhZS3h9cpKHdfdV0/IHq2pnkvdFeAUAAGADreV7XmuN6wEAAGBdrCW8/mmSM6vqO6rqQVV1QpK3TusBAABgw6xl2vDzkvxCklcmOTLJx5K8Ickvb0BdAAAA8CW3GF6r6rFJvqe7fzbJC6d/e7b9WpLjkvzThlUI3Go7Tz9jLu3sPn9XDnr8E+bSFgAAt0+rmTb8giTnrrDt75P8/PqVAwAAADe3mvD68CR/vcK2v0vyyPUrBwAAAG5uNeH1kCR3XGHbgUnuun7lAAAAwM2tJrx+IMm3r7Dt26ftAAAAsGFW87Th30zyh1V1QJK3dvcXq+oOSZ6U2ZOH/8dGFggAAAC3GF67+/VV9dVJTktyp6r6ZJLDk3wuyYu6+w0bXCMAAAC3c6v6ntfufnlVvSrJY5J8VZKrkry7u6/dyOIAAAAgWWV4TZIpqP7NBtYCAAAAe7WaBzYBAADAQq165BVgX3afc/Zc2jno8U+YSzsAAIzFyCsAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4cwmvVXXvqvr7qnp/VV1QVc+a1h9WVe+oqgun17vNox4AAAA2l3mNvN6Q5Lnd/aAkj07yk1X14CTPT3JWdx+b5KxpGQAAAG5iLuG1uy/r7p3T++uSvD/JUUlOSnLatNtpSZ40j3oAAADYXLbMu8GqOibJI5K8J8k9uvuyZBZwq+ruKxyzLcm2JDn66KPnUyjAfmrn6WfMra3jTjl5bm0BAPu3uT6wqaoOTvJnSZ7d3deu9rju3t7dW7t76xFHHLFxBQIAADCkuYXXqjows+D6J939lmn15VV1z2n7PZNcMa96AAAA2Dzm9bThSvLqJO/v7pcv2fT2JKdO709N8rZ51AMAAMDmMq97Xh+b5AeT/FtV/eu07gVJXpbkTVX1jCQfTfLkOdUDAADAJjKX8Nrd70pSK2w+fh41AAAAsHnN9YFNAAAAcGsIrwAAAAxPeAUAAGB4wisAAADDm9fThgEYxLnn75pbW8fNrSUAYH9n5BUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhbVl0AQBrsfucs+fW1rlzayk5bo5tAQBsRkZeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPC2LLoAYGOce/6uRZcAc7Xz9DPm1taOu91/bm1tO3Hr3NoCgJEZeQUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4WxZdAAD7r52nn7HoEgCA/YSRVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIa3ZdEFAJBsP3PHokvYEOeev2vRJWyIgx5//0WXAAC3O0ZeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGN6WRRcAQLL7nLMXXQIAwNCMvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADC8LYsuAABY2c7Tz5hbW8edcvLc2tpfr2t/tv3MHXNra9uJW+fWFrB5GHkFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeFsWXQAAsLJzz981t7aOm1tL872uHWfuyLYTt86tPQA2hpFXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhrdl0QUAwGaz+5yzF10CA9t+5o65tbXtxK1zawtg0Yy8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAY3lzCa1W9pqquqKp/X7LusKp6R1VdOL3ebR61AAAAsPnMa+T1tUlOWLbu+UnO6u5jk5w1LQMAAMDNzCW8dve5ST61bPVJSU6b3p+W5EnzqAUAAIDNZ5H3vN6juy9Lkun17gusBQAAgIFtWXQBq1FV25JsS5Kjjz56wdUAANx+7Dz9jCTJ7vN3bXhbBz3+CRveBrB5LXLk9fKqumeSTK9XrLRjd2/v7q3dvfWII46YW4EAAACMYZHh9e1JTp3en5rkbQusBQAAgIHN66ty3pDk3UkeWFWXVtUzkrwsybdV1YVJvm1aBgAAgJuZyz2v3f39K2w6fh7tAwAAsLktctowAAAArIrwCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhzeVpwwDA+LafuWPRJcDc7Dz9jLm1ddwpJ8+tLdifGXkFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeFsWXQAAMIbd55y96BJYpT0/q51Xf2jBlayvRVzXuefv2vA2dpy5I0my7cStG94W7M+MvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADC8LYsuAABgo20/c8eiS9gQ556/a9ElsAbz7IfbTtw6t7ZgXoy8AgAAMDzhFQAAgOEJrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMLwtiy4AAGAj7T7n7Lm0c9DjnzCXdmA028/cMbe2tp24dW5tMR4jrwAAAAxPeAUAAGB4wisAAADDE14BAAAYnvAKAADA8IRXAAAAhie8AgAAMDzhFQAAgOEJrwAAAAxvy6ILAADYH+w+5+xFl7DfOPf8XYsugUFtP3PH3NraduLWubXF6hh5BQAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHhbFl0AAACwPnafc3aSZOfVH9rwto475eQNb2N/t/P0M+bW1v7w8zLyCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPC2LLoAAADYn+0+5+xFl7Ahdp5+RpJk9/m7FlzJ+jro8U+Ye5vnzuEz3HHmjiTJthO3bnhbG8XIKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCAAAwPOEVAACA4QmvAAAADE94BQAAYHjCKwAAAMMTXgEAABjelkUXAAAAMIrd55ydJNl59YcWXAnLGXkFAABgeMIrAAAAwxNeAQAAGJ7wCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeFsWXQAAALC+zj1/16JLgHVn5BUAAIDhCa8AAAAMT3gFAABgeMIrAAAAwxNeAQAAGN7Cw2tVnVBVH6yqi6rq+YuuBwAAgPEsNLxW1QFJXpnkO5M8OMn3V9WDF1kTAAAA41n0yOujklzU3R/u7v9M8sYkJy24JgAAAAaz6PB6VJJLlixfOq0DAACAL9my4PZrL+v6ZjtVbUuybVq8vqo+uKFV7dvhST65wPa5/dHnWAT9jnnT55g3fY5FWFy/+43Zy39fSONrdp+9rVx0eL00yb2XLN8ryceX79Td25Nsn1dR+1JVO7p766Lr4PZDn2MR9DvmTZ9j3vQ5FkG/u20WPW34vUmOrar7VtUdkzwlydsXXBMAAACDWejIa3ffUFXPTPI3SQ5I8pruvmCRNQEAADCeRU8bTnf/ZZK/XHQdazDE9GVuV/Q5FkG/Y970OeZNn2MR9LvboLpv9nwkAAAAGMqi73kFAACAWyS8rkFVnVBVH6yqi6rq+Yuuh/1DVd27qv6+qt5fVRdU1bOm9YdV1Tuq6sLp9W5Ljvm5qR9+sKq+Y3HVs5lV1QFV9S9V9efTsj7HhqmqQ6vqzVX1gem/d4/R59hIVfWc6ffqv1fVG6rqzvoc662qXlNVV1TVvy9Zt+Z+Vrs5xDcAAAmKSURBVFWPrKp/m7b9dlXt7StFb/eE11WqqgOSvDLJdyZ5cJLvr6oHL7Yq9hM3JHludz8oyaOT/OTUt56f5KzuPjbJWdNypm1PSfKQJCck+b2pf8JaPSvJ+5cs63NspFck+evu/tokD8us7+lzbIiqOirJTyfZ2t0PzezBoE+JPsf6e21mfWapW9PPfj/JtiTHTv+Wn5MIr2vxqCQXdfeHu/s/k7wxyUkLron9QHdf1t07p/fXZfY/dEdl1r9Om3Y7LcmTpvcnJXljd3++uz+S5KLM+iesWlXdK8l3J3nVktX6HBuiqg5J8rgkr06S7v7P7r4m+hwba0uSu1TVliQHJfl49DnWWXefm+RTy1avqZ9V1T2THNLd7+7ZA4lOX3IMSwivq3dUkkuWLF86rYN1U1XHJHlEkvckuUd3X5bMAm6Su0+76Yush99K8rwkX1yyTp9jo9wvyZVJ/niaqv6qqvqK6HNskO7+WJJfT/LRJJcl+XR3/230OeZjrf3sqOn98vUsI7yu3t7mnXtUM+umqg5O8mdJnt3d1+5r172s0xdZtap6YpIruvu81R6yl3X6HGuxJclxSX6/ux+R5DOZptGtQJ/jNpnuMTwpyX2THJnkK6rqafs6ZC/r9DnW20r9TP9bJeF19S5Ncu8ly/fKbPoJ3GZVdWBmwfVPuvst0+rLp2kkmV6vmNbri9xWj03yPVV1cWa3QDyhql4XfY6Nc2mSS7v7PdPymzMLs/ocG+Vbk3yku6/s7i8keUuSb4w+x3ystZ9dOr1fvp5lhNfVe2+SY6vqvlV1x8xutn77gmtiPzA9Te7VSd7f3S9fsuntSU6d3p+a5G1L1j+lqu5UVffN7Kb+f55XvWx+3f1z3X2v7j4ms/+Wnd3dT4s+xwbp7k8kuaSqHjitOj7Jf0SfY+N8NMmjq+qg6ffs8Zk9U0KfYx7W1M+mqcXXVdWjp/56ypJjWGLLogvYLLr7hqp6ZpK/yeyJda/p7gsWXBb7h8cm+cEk/1ZV/zqte0GSlyV5U1U9I7Nfwk9Oku6+oKrelNn/+N2Q5Ce7+8b5l81+SJ9jI/1Ukj+Z/gD84SQ/lNkf0fU51l13v6eq3pxkZ2Z96F+SbE9ycPQ51lFVvSHJtyQ5vKouTfKi3Lrfpz+e2ZOL75Lkr6Z/LFOzB1oBAADAuEwbBgAAYHjCKwAAAMMTXgEAABie8AoAAMDwhFcAAACGJ7wCwKSqjqmqrqpN9VVyVfVXVXXqLe85rqp6cVW9btF1ADAu4RWATaGqLq6qz1bV9VV1eVX9cVUdvOi6RtDd39ndp63nOavqtVX19Kp6wfSZX19Vn6uqG5csr/h951X1zqr6kVvZ9jFVdfGtLh6A/ZLwCsBmcmJ3H5zkuCRfn+QX1nqCzTaqui81s6G/y7v7V7r74Olz/7Ek796z3N0P2ci2AWAp4RWATae7P5bkr5I8dG9TfZeO+k2jh/9YVb9ZVZ9K8uKquktV/UZV7aqqT1fVu6rqLkuaeGpVfbSqPllVP7/kvI+qqndX1TVVdVlV/W5V3XHaVlMbV0znPL+qHjptu1NV/fp0zsur6g+WtZclbeyp93em83ygqo5fdm0vrap/TLI7yf2Wj3JW1Y9W1fur6rqq+o+qOm5af2RV/VlVXVlVH6mqn74tP4eq+saqeu9U53ur6hun9S9N8s1Jfncaof3daf0rquqSqrq2qs6rqm++Le0DcPsivAKw6VTVvZN8V5J/WeUh35Dkw0nunuSlSX49ySOTfGOSw5I8L8kXl+z/TUkemOT4JC+sqgdN629M8pwkhyd5zLT9J6Zt357kcUm+JsmhSU5OctW07dem9Q9P8oAkRyV54SrqPTzJi5K8paoOW7L9B5NsS3LXJLuWHlhVT07y4iSnJDkkyfckuWoaoT0zyfum9o9P8uyq+o69FdDdT+/u165U4FTPXyT57SRfleTlSf6iqr6qu38+yT8keeY0QvvM6bD3Tp/BYUlen+RPq+rOe2n74u4+ZqW2Abh9El4B2EzeWlXXJHlXknOS/Moqj/t4d/9Od9+Q5PNJfjjJs7r7Y919Y3f/v+7+/JL9X9Ldn+3u92UW9h6WJN19Xnf/U3ff0N0XJ/nDJI+fjvlCZmHya5NUd7+/uy+rqkryo0me092f6u7rprqfso96r0jyW939he4+I8kHk3z3ku2v7e4Lpjq+sOzYH0nyv7v7vT1zUXfvymya9RHd/b+6+z+7+8NJ/ugW6tiX705yYXf/n6mONyT5QJITVzqgu1/X3VdN+/9Gkjtl9kcCALhF+819PwDcLjypu/9u6YpZNrxFlyx5f3iSOyf50D72/8SS97uTHDy19TWZjTBuTXJQZr9Hz0uS7j57mh77yiRHV9X/TfIzU1sHJTlvSa2V5IB9tP+x7u4ly7uSHLnC9Sx37xWu7T5JjpzC/x4HZDZCemscmWWjvtPyUSsdUFXPzSxcH5mkMxsZPvxWtg/A7YyRVwA2u89MrwctWffVy/ZZGgQ/meRzSe5/K9r6/cxGF4/t7kOSvCCzIDprpPu3u/uRSR6S2TTh/zm199kkD+nuQ6d/Xzk9AGklR9VNU/nRST6+wvUsd0n2fm2XJPnIkhoO7e67dvd37eNc+/LxzALxUkcn+djeapzub/3ZJN+X5G7dfWiST2fJ5wcA+yK8ArCpdfeVmQWmp1XVAVX1w9lHMO3uLyZ5TZKXTw8wOqCqHlNVd1pFc3dNcm2S66vqa5P8+J4NVfX1VfUNVXVgZoH6c0lunNr7oyS/WVV3n/Y9aqV7TSd3T/LTVXXgdA/rg5L85SrqS5JXJfmZqnrk9BCpB1TVfZL8c5Jrq+pnpwdWHVBVD62qr1/leZf7yyRfU1U/UFVbqurkJA9O8ufT9suT3G/J/ndNckOSK5NsqaoXZjbyCgCrIrwCsD/40cxGOa/KbNTz/93C/j+T5N8ye4DQpzJ7oNJqfif+TJIfSHJdZoH0jCXbDpnWXZ3Z9NmrMnswVDIbcbwoyT9V1bVJ/i77vtfzPUmOzWzU9qVJvre7r9rH/l/S3X86HfP6qc63Jjmsu2/M7H7Uhyf5yHTuVyX5ytWcdy/tXJXkiUmem9m1Pi/JE7v7k9Mur0jyvVV1dVX9dpK/yewJ0f9fZp/P57Lv6c8AcBN101tqAIBFqqqnJ/mR7v6mRdcCACMx8goAAMDwhFcAAACGZ9owAAAAwzPyCgAAwPCEVwAAAIYnvAIAADA84RUAAIDhCa8AAAAMT3gFAABgeP8/iULc94Z4KsUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 9))\n",
    "plt.hist(df_supermarket[df_supermarket['Customer type'] == 'Member']['Total'],\n",
    "         alpha = 0.4, bins = 30, color = '#003f7e')                                 # members = blue group\n",
    "plt.hist(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Total'],\n",
    "         alpha = 0.4, bins = 30, color = '#821626')                                 # members = red group\n",
    "plt.xlabel(\"Purchase price 'Total'\", size = 12)\n",
    "plt.ylabel(\"Count\", size = 12)\n",
    "plt.title(\"Histogram of total purchase prices\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean of members: 327.7913053892216\n",
      "Mean of normals: 318.122855711423\n"
     ]
    }
   ],
   "source": [
    "print('Mean of members: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Member']['Total'].mean()))\n",
    "print('Mean of normals: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Total'].mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the distributions of the dependent variable ``Total`` are not normally distributed, rather left-skewed, and therefore violate one of the preconditions. However, as described [earlier](#violation), we rely on the robustness of the t-test and proceed. The means differ a bit, so we can check whether this difference is due to the group and therefore service level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance of members: 61500.91580485452\n",
      "Variance of normals: 59488.5760661809\n"
     ]
    }
   ],
   "source": [
    "print('Variance of members: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Member']['Total'].var()))\n",
    "print('Variance of normals: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Total'].var()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variances are quite similar, so we stick with the t-test. The hypotheses are:\n",
    "* **Null hypothesis $H_0$: There is no difference between members and normal buyers in terms of total purchase price.**\n",
    "* **Alternative hypothesis $H_1$: The difference in the total purchase price sample means results from a different behavior of the two groups and is not random.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=0.6215268724928228, pvalue=0.5343949596143621)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.ttest_ind(df_supermarket[df_supermarket['Customer type'] == 'Member']['Total'],\n",
    "                df_supermarket[df_supermarket['Customer type'] == 'Normal']['Total'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a probability value of $p \\approx 0.534$, our chosen significance level $\\alpha = 0.05$ has been clearly exceeded. The null hypothesis can therefore not be rejected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rating\n",
    "Our second dependent variable is customer satisfaction, the ``Rating`` attribute. Again, we look at the distribution of ratings across the two groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA68AAAIpCAYAAABNBg6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7RlZXkm+ueVQo3inRJBQbzFjtoBSYVoTBsTjY0eEe0RO5qo6DGpdFqTYPSgsRPFtDHK8JrOZaS8nIBXxHgjx2skQsxR2xIFIejBC4iCgCgiIkbgPX+sWbopahe7qnat9W34/cbYY60151zze9aqNXbVU9+cc1V3BwAAAEZ2s0UHAAAAgBuivAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAbhRqaqzquphi86xSFX1+Ko6v6quqKoH7qYxPlBVR+6OfQPAtpTveQVgraiqc5P8dnf/05JlT5uW/dIO7OfAJF9Nsmd3X726KRevqr6c5I+6+72rtL9jkty7u5+8GvsDgJ1h5hUAVllVrVtwhLsnOWslGw6QFQBWRHkF4Ealqs6tqkdM9w+tqs1VdXlVXVRVr5o2O3W6vWw6tPbBVXWzqvqTqjqvqi6uquOr6nZL9vvUad2lVfWnW41zTFW9s6reXFWXJ3naNPYnquqyqrqwqv6qqm6+ZH9dVf+9qs6pqu9V1f+sqntNz7m8qt6xdPutXuM2s1bVLarqiiR7JDl9moHd1vO7qp5ZVeckOWda9trpUOPLq+ozVfWfpuWHJXlBkt+Y3qvTp+Ufq6rfnu4/rao+XlWvqKrvVNVXq+pRS8a7R1WdOr3Of6qqv66qN0/rbjm9b5dO79Wnq2qfHftTB+CmQHkF4MbstUle2923TXKvJO+Ylj90ur19d+/V3Z9I8rTp51eS3DPJXkn+Kkmq6n5J/ibJbyXZN8ntktx1q7GOSPLOJLdP8pYk1yR5dpK9kzw4ycOT/PetnnNYkp9L8qAkRyfZNI2xf5IHJHnSMq9rm1m7+4fdvde0zUHdfa/l35o8LskvJLnf9PjTSQ5Ocsckb01yYlXdsrs/mOSlSU6Y3quDltnfLyT54vR6j03yhqqqad1bk/zvJHdKckySpyx53pGZvZ/7T+v/W5IfbCc3ADdRyisAa817phm6y6rqssxK5XJ+lOTeVbV3d1/R3Z/czra/leRV3f2V7r4iyR8neeJ0WO2vJzmpuz/e3f+e5IVJtr5oxCe6+z3dfW13/6C7P9Pdn+zuq7v73CR/l+SXt3rOy7v78u4+K8mZST48jf/dJB9IstzFlraXdaX+oru/3d0/SJLufnN3XzrlfWWSWyS57w7s77zufl13X5PkuMxK/j5VdUCSn0/ywu7+9+7+eJL3LXnejzIrrffu7mum9+3yHRgXgJsI5RWAteZx3X37LT+5/mzmUs9I8tNJvjAdjvqY7Wy7X5Lzljw+L8m6JPtM687fsqK7r0xy6VbPP3/pg6r66ar6x6r65nQo8Uszm5Vc6qIl93+wjcd7Zdu2l3Wlts77nKo6u6q+O/2nwO22kXd7vrnlzvT+JLP8+yX59pJlW4/9piQfSvL2qrqgqo6tqj13YFwAbiKUVwButLr7nO5+UpI7J3l5kndW1a1z/VnTJLkgswsdbXFAkqszK5QXJrnblhVV9VOZzRZeZ7itHv9tki8kuc902PILklRWx/ayrtSP807ntz4vyX9NcofpPwW+m5/k3ZWvJrgwyR2r6lZLlu3/4xDdP+ruF3f3/ZL8YpLHJHnqLowHwI2U8grAjVZVPbmq1nf3tUkumxZfk+SSJNdmdr7oFm9L8uzp4kJ75SfneV6d2bmsh1fVL04XUXpxbriI3ibJ5UmuqKr/kOT3Vu2FbT/rzrhNZuX3kiTrquqFSW67ZP1FSQ6sqh3+d0N3n5dkc5JjqurmVfXgJIdvWV9Vv1JV/7Gq9sjs/fpRZn9GAHAdyisAN2aHJTlrugLva5M8sbuvmg5h/fMk/zqdO/ugJG/M7BDWUzP7Dtirkvx+kkznpP5+krdnNpP4vSQXJ/nhdsZ+bpLfnLZ9XZITVvF1LZt1J30os3Ns/7/MDkG+Ktc9tPfE6fbSqjptJ/b/W5ldtOrSJC/J7L3Y8t7dJbP/HLg8ydlJTkny5p0YA4AbuerelSOBAOCmZ5rtvCyzQ4K/uug8a01VnZDkC939okVnAWDtMPMKACtQVYdX1a2mc2ZfkeTzSc5dbKq1oap+fvoO25tN3xt7RJL3LDoXAGuL8goAK3NEZhdKuiDJfTI7BNnhSytzlyQfS3JFkr9M8nvd/dmFJgJgzXHYMAAAAMMz8woAAMDwlFcAAACGt27RAXbU3nvv3QceeOCiYwAAALAbfOYzn/lWd6/fevmaK68HHnhgNm/evOgYAAAA7AZVdd62ljtsGAAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAY3rpFBwAAgKU2nbR5bmNtPHzD3MYCdo2ZVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIY3l/JaVbesqv9dVadX1VlV9eJp+R2r6iNVdc50e4d55AEAAGBtmdfM6w+T/Gp3H5Tk4CSHVdWDkjw/yUe7+z5JPjo9BgAAgOuYS3ntmSumh3tOP53kiCTHTcuPS/K4eeQBAABgbZnbOa9VtUdVfS7JxUk+0t2fSrJPd1+YJNPtnZd57saq2lxVmy+55JJ5RQYAAGAQcyuv3X1Ndx+c5G5JDq2qB+zAczd194bu3rB+/frdFxIAAIAhzf1qw919WZKPJTksyUVVtW+STLcXzzsPAAAA45vX1YbXV9Xtp/s/leQRSb6Q5H1Jjpw2OzLJe+eRBwAAgLVl3ZzG2TfJcVW1R2aF+R3d/Y9V9Ykk76iqZyT5WpInzCkPAAAAa8hcymt3n5HkgdtYfmmSh88jAwAAAGvX3M95BQAAgB2lvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIa3btEBAGA1bDpp89zG2nj4hrmNBQDMmHkFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIa3btEBAAB2t9OOP2FuYx3y1N+Y21gANyVmXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADG/dogMAwGq48pST5zfY4RvmNxYAP7bppM1zG2uj3/XDMfMKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAY3lzKa1XtX1X/XFVnV9VZVfWH0/JjquobVfW56efR88gDAADA2rJuTuNcneQ53X1aVd0myWeq6iPTuld39yvmlAMAAIA1aC7ltbsvTHLhdP97VXV2krvOY2wAAADWvrmf81pVByZ5YJJPTYueVVVnVNUbq+oO884DAADA+OZ12HCSpKr2SvIPSY7q7sur6m+T/M8kPd2+Msn/uY3nbUyyMUkOOOCA+QUGuBE67fgT5jbWIU/9jbmNBdtz6hnnzW2sQ+Y2EsBNy9xmXqtqz8yK61u6+11J0t0Xdfc13X1tktclOXRbz+3uTd29obs3rF+/fl6RAQAAGMS8rjZcSd6Q5OzuftWS5fsu2ezxSc6cRx4AAADWlnkdNvyQJE9J8vmq+ty07AVJnlRVB2d22PC5SX53TnkAAABYQ+Z1teGPJ6ltrHr/PMYHAABgbZv71YYBAABgRymvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4a1bdAAAAICVuPKUk+c32OEb5jcWK2LmFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDW7foAOyaTSdtnttYGw/fMLexAICbritPOXl+g/n3DawZZl4BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxv3aIDwE3Ja5577NzGOuoVR89tLHbdacefsOgIu8WN9XUB8BN+1zMvZl4BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8OZSXqtq/6r656o6u6rOqqo/nJbfsao+UlXnTLd3mEceAAAA1pZ5zbxeneQ53f0zSR6U5JlVdb8kz0/y0e6+T5KPTo8BAADgOuZSXrv7wu4+bbr/vSRnJ7lrkiOSHDdtdlySx80jDwAAAGvL3M95raoDkzwwyaeS7NPdFyazgpvkzvPOAwAAwPjWzXOwqtoryT8kOaq7L6+qlT5vY5KNSXLAAQfsvoAAMJhNJ22e21gbD98wt7EA+InXPPfYuY111CuOnttYq21uM69VtWdmxfUt3f2uafFFVbXvtH7fJBdv67ndvam7N3T3hvXr188nMAAAAMOY19WGK8kbkpzd3a9asup9SY6c7h+Z5L3zyAMAAMDaMq/Dhh+S5ClJPl9Vn5uWvSDJy5K8o6qekeRrSZ4wpzwAAACsIXMpr9398STLneD68HlkAAAAYO2a+9WGAQAAYEcprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOGtW3QAAJJTzzhvbmM99GfvPrexgN1r00mb5zbWxsM3zG0sgG0x8woAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjrFh2AXXPlKSfPb7DDN8xvLADm7jXPPXZuYx31iqPnNhYANw5mXgEAABjeistrVT1hmeW/vnpxAAAA4Pp2ZOb1Dcss37QaQQAAAGA5N3jOa1Xdc7p7s6q6R5JasvqeSa7aHcEAAABgi5VcsOlLSTqz0vrlrdZ9M8kxq5wJAAAAruMGy2t33yxJquqU7v7l3R8JAAAArmvF57wqrgAAACzKir/ndTrf9c+THJxkr6XruvuAVc4FAAAAP7bi8prkrZmd8/qcJFfunjgAAABwfTtSXu+f5CHdfe3uCgMAAADbsiPf83pqkgfuriAAAACwnB2ZeT03yYeq6l2ZfUXOj3X3C1czFAAAACy1I+X11klOSrJnkv13TxwAAAC4vhWX1+5++u4Mwvhe89xj5zbWUa84em5jsetOO/6EuY21+Q73mttYGw/fMLexALjx23TS5rmN5e8wbox25Kty7rncuu7+yurEAQAAgOvbkcOGv5Skk9SSZT3d7rFqiQAAAGArO3LY8HWuTFxVd0nyoiT/stqhAAAAYKkd+aqc6+jubyY5KslfrF4cAAAAuL6dLq+T+ya51WoEAQAAgOXsyAWb/iU/Occ1mZXW+yf5s9UOBQAAAEvtyAWbXr/V4+8nOb27z1nFPAAAAHA9O3LBpuN2ZxAAAABYzorPea2qPavqxVX1laq6arp9cVXdfHcGBAAAgB05bPjYJIcm+W9Jzkty9yR/muS2SZ69+tEAAABgZkfK6xOSHNTdl06Pv1hVpyU5PcorAAAAu9GOfFVO7eByAAAAWBU7Ul5PTHJSVf3nqvqZqjosyXum5QAAALDb7Mhhw0cn+ZMkf51kvyTfSPK2JC/ZDbkAAADgx26wvFbVQ5I8trufl+SF08+WdS9PckiST+62hGvQppM2LzoCO8Cf16479Yzz5jbWrX75XnMby2dj183zs3FjdeUpJy86wm5x2vEnLDoCAGvMSg4bfkGSU5dZ989J/sfqxQEAAIDrW0l5PTjJB5dZ909Jfm714gAAAMD1raS83jbJzZdZt2eS26xeHAAAALi+lZTXLyR55DLrHjmtBwAAgN1mJVcbfnWSv6uqPZK8p7uvraqbJXlcZlce/qPdGRAAAABusLx291ur6i5Jjktyi6r6VpK9k1yV5EXd/bbdnBEAAICbuBV9z2t3v6qqXp/kwUnulOTSJJ/o7st3ZzgAAABIVlhek2Qqqh/ajVkAAABgm1ZywSYAAABYKOUVAACA4c2lvFbVG6vq4qo6c8myY6rqG1X1uenn0fPIAgAAwNozr5nXv09y2DaWv7q7D55+3j+nLAAAAKwxcymv3X1qkm/PYywAAABufFZ8teHd5FlV9dQkm5M8p7u/s62Nqmpjko1JcsABB8wxHjcFV55y8qIjMCifDWB08/w9ddp3vjy3sdh1px1/wqIjwKpb5AWb/jbJvZIcnOTCJK9cbsPu3tTdG7p7w/r16+eVDwAAgEEsrLx290XdfU13X5vkdUkOXVQWAAAAxraw8lpV+y55+PgkZy63LQAAADdtcznntareluRhSfauqq8neVGSh1XVwUk6yblJfnceWQAAAFh75lJeu/tJ21j8hnmMDQAAwNq3yAs2AQAAwIoorwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOGtW3QA2JZNJ21edAQAgDXr1DPOm9tYD/3Zu89tLG7azLwCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABjeukUHAHaP1zz32EVH2C2uPOXkRUcAVsGpZ5y36Agwd/4Og11j5hUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGN66RQe4MbrylJMXHQEAAOBGxcwrAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhrVt0AADm69Qzzlt0BLhRO+34ExYdAebK3yvMi5lXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMLy5lNeqemNVXVxVZy5Zdseq+khVnTPd3mEeWQAAAFh75jXz+vdJDttq2fOTfLS775Pko9NjAAAAuJ65lNfuPjXJt7dafESS46b7xyV53DyyAAAAsPasW+DY+3T3hUnS3RdW1Z2X27CqNibZmCQHHHDAnOIBALDFqWect+gIu8VrnnvsoiMAK7QmLtjU3Zu6e0N3b1i/fv2i4wAAADBniyyvF1XVvkky3V68wCwAAAAMbJHl9X1JjpzuH5nkvQvMAgAAwMDm9VU5b0vyiST3raqvV9Uzkrwsya9V1TlJfm16DAAAANczlws2dfeTlln18HmMDwAAwNq2Ji7YBAAAwE2b8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjKKwAAAMNTXgEAABie8goAAMDwlFcAAACGp7wCAAAwPOUVAACA4SmvAAAADE95BQAAYHjrFh0AANaa044/YdERANjNNp20edER2IqZVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1u36AAAsNacesZ5i47AwHw+AHYPM68AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPDWLToAbMuVp5y86AgAANyE+ffoeMy8AgAAMDzlFQAAgOEprwAAAAxPeQUAAGB4yisAAADDU14BAAAYnvIKAADA8JRXAAAAhqe8AgAAMDzlFQAAgOEprwAAAAxv3aIDVNW5Sb6X5JokV3f3hsUmAgAAYDQLL6+TX+nuby06BAAAAGNy2DAAAADDG6G8dpIPV9Vnqmrjtjaoqo1VtbmqNl9yySVzjgcAAMCijVBeH9LdhyR5VJJnVtVDt96guzd194bu3rB+/fr5JwQAAGChFl5eu/uC6fbiJO9OcuhiEwEAADCahZbXqrp1Vd1my/0kj0xy5iIzAQAAMJ5FX214nyTvrqotWd7a3R9cbCQAAABGs9Dy2t1fSXLQIjMAAAAwvoWf8woAAAA3RHkFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADA85RUAAIDhKa8AAAAMT3kFAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADC8hZfXqjqsqr5YVV+qqucvOg8AAADjWWh5rao9kvx1kkcluV+SJ1XV/RaZCQAAgPEseub10CRf6u6vdPe/J3l7kiMWnAkAAIDBLLq83jXJ+Usef31aBgAAAD+2bsHj1zaW9fU2qtqYZOP08Iqq+uJuTbXr9k7yrUWHYEg+GyzHZ4Pl+GywPT4fLMdng2169iuftxY+G3ff1sJFl9evJ9l/yeO7Jblg6426e1OSTfMKtauqanN3b1h0Dsbjs8FyfDZYjs8G2+PzwXJ8NljOWv5sLPqw4U8nuU9V3aOqbp7kiUnet+BMAAAADGahM6/dfXVVPSvJh5LskeSN3X3WIjMBAAAwnkUfNpzufn+S9y86xypbM4c4M3c+GyzHZ4Pl+GywPT4fLMdng+Ws2c9GdV/v+kgAAAAwlEWf8woAAAA3SHldZVW1R1V9tqr+cdFZGEdVnVtVn6+qz1XV5kXnYSxVdfuqemdVfaGqzq6qBy86E4tXVfedfmds+bm8qo5adC7GUFXPrqqzqurMqnpbVd1y0ZkYQ1X94fS5OMvvDKrqjVV1cVWduWTZHavqI1V1znR7h0Vm3BHK6+r7wyRnLzoEQ/qV7j54rV6anN3qtUk+2N3/IclB8TuEJN39xel3xsFJfi7JlUneveBYDKCq7prkD5Js6O4HZHbRyycuNhUjqKoHJPmdJIdm9vfJY6rqPotNxYL9fZLDtlr2/CQf7e77JPno9HhNUF5XUVXdLcn/keT1i84CrA1VddskD03yhiTp7n/v7ssWm4oBPTzJl7v7vEUHYRjrkvxUVa1LcqskFyw4D2P4mSSf7O4ru/vqJKckefyCM7FA3X1qkm9vtfiIJMdN949L8ri5htoFyuvqek2So5Ncu+ggDKeTfLiqPlNVGxcdhqHcM8klSf7v6ZSD11fVrRcdiuE8McnbFh2CMXT3N5K8IsnXklyY5Lvd/eHFpmIQZyZ5aFXdqapuleTRSfZfcCbGs093X5gk0+2dF5xnxZTXVVJVj0lycXd/ZtFZGNJDuvuQJI9K8syqeuiiAzGMdUkOSfK33f3AJN/PGjp8h92vqm6e5LFJTlx0FsYwnZ92RJJ7JNkvya2r6smLTcUIuvvsJC9P8pEkH0xyepKrFxoKVpHyunoekuSxVXVukrcn+dWqevNiIzGK7r5gur04s3PWDl1sIgby9SRf7+5PTY/fmVmZhS0eleS07r5o0UEYxiOSfLW7L+nuHyV5V5JfXHAmBtHdb+juQ7r7oZkdLnrOojMxnIuqat8kmW4vXnCeFVNeV0l3/3F33627D8zs8K6Tu9v/gpKqunVV3WbL/SSPzOywHkh3fzPJ+VV132nRw5P82wIjMZ4nxSHDXNfXkjyoqm5VVZXZ7w0XeiNJUlV3nm4PSPJf4vcH1/e+JEdO949M8t4FZtkh6xYdAG4C9kny7j9ONrsAAAicSURBVNm/L7IuyVu7+4OLjcRgfj/JW6bDQ7+S5OkLzsMgpnPWfi3J7y46C+Po7k9V1TuTnJbZIaGfTbJpsakYyD9U1Z2S/CjJM7v7O4sOxOJU1duSPCzJ3lX19SQvSvKyJO+oqmdk9p9hT1hcwh1T3b3oDAAAALBdDhsGAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvALAAlXVWVX1sBVs9/iqOr+qrqiqB67i+AdM+9xjtfY5D1X1gqp6/aJzADA/yisAu6SqfrOqNk8F6MKq+kBV/dIu7vOYqnrzamUcRVX9fVW9ZOmy7r5/d39sBU9/RZJndfde3f3ZXchwblU9Ysn4X5v2ec3O7nM7Y/WS+x+rqqumz8m3qupdVbXvCvfzsOn7CX+su1/a3b+9SjnPraoDV2NfAOw+yisAO62q/ijJa5K8NMk+SQ5I8jdJjlhkrtVUVesWnWFy9yRnLTrELnpWd++V5N5J9sqskAPAiiivAOyUqrpdkj9L8szufld3f7+7f9TdJ3X3/zVtc52Zxq1n0KrqeVX1jar6XlV9saoeXlWHJXlBkt+YZulOn7bdr6reV1XfrqovVdXvLNnPMVV1YlW9edrX56vqp6vqj6vq4ulw20cuzV5Vb5hmir9RVS/ZcthsVT2tqv61ql5dVd9Ocsw2Xvuh02zz5VV1UVW9asm6E6vqm1X13ao6taruPy3fmOS3khw9va6TpuU/ngnd1n6r6hZVdUWSPZKcXlVfnrZ9flV9eXq9/1ZVj98q4+9U1dlL1h9SVW/K7D8YTpoyHF1VB1ZVbynpK3if31FVx0/7PauqNqz8UzPT3ZcleU+Sg5fs++lL8n6lqn53Wn7rJB9Ist+U+Yop449n55e8hiOr6mvTzO7/WLLvn6qq46rqO9MYR289kwvA+JRXAHbWg5PcMsm7d+bJVXXfJM9K8vPdfZsk/znJud39wcxmck+YDmc9aHrK25J8Pcl+SX49yUur6uFLdnl4kjcluUOSzyb5UGZ/z901s5L9d0u2PS7J1ZnNAD4wySOTLD0E9ReSfCXJnZP8+TbivzbJa7v7tknuleQdS9Z9IMl9pueeluQtSdLdm6b7x06v6/CV7Le7fzjNVibJQd19r+n+l5P8pyS3S/LiJG/echhuVT0hs9L91CS3TfLYJJd291OSfC3J4VOGY7eR4Ybe58cmeXuS2yd5X5K/2sY+Mr3m2tbyqrpTkv+S5EtLFl+c5DFT3qcneXVVHdLd30/yqCQXTJn36u4Llhnyl5LcN8nDk7ywqn5mWv6iJAcmuWeSX0vy5K1yHtjd5y73OgAYg/IKwM66U5JvdffVO/n8a5LcIsn9qmrP7j63u7+8rQ2rav/Misnzuvuq7v5cktcnecqSzf6luz805TkxyfokL+vuH2VWtg6sqttX1T6ZlaGjptnii5O8OskTl+zrgu7+X919dXf/YBuRfpTk3lW1d3df0d2f3LKiu9/Y3d/r7h9mViAPmmapV2LZ/W6tu0/s7gu6+9ruPiHJOUkOnVb/dmYl+dM986XuPu+GBl/h+/zx7n7/dI7sm5IctI1dLecvq+q7Sb6VZO8kv7/k9fw/3f3lKe8pST6cWTnfES/u7h909+lJTl+S7b8meWl3f6e7v57kL3dwvwAMQHkFYGddmmTv2slzQrv7S0mOyqzgXVxVb6+q/ZbZfL8k3+7u7y1Zdl5ms6pbXLTk/g8yK9bXLHmczM6zvHuSPZNcWFWXVdVlmc3K3nnJ88+/gfjPSPLTSb5QVZ+uqsckSVXtUVUvmw7nvTzJudP2e9/A/ra7322pqqdW1eeWvIYHLBln/8xmZnfUSt7nby65f2WSW+7AZ+APuvt2SX42sxnyu21ZUVWPqqpPTocrX5bk0Vn5+7Zcti0z1vvlun+mN/TnC8CAlFcAdtYnklyV5HHb2eb7SW615PFdlq7s7rd29y9lVig7ycu3rNpqPxckuWNV3WbJsgOSfGMncp+f5IdJ9u7u208/t+3u+y+Ntr0ddPc53f2kzArvy5O8czo38zczu1jVIzI7nPfA6SlbDp/d2f1eR1XdPcnrMjvs+k7dffskZy4Z5/zMDjve5jDbibCa7/OyuvvzSV6S5K9r5hZJ/iGzCzjtM72e92eF79sKXJglRTmzcg/AGqO8ArBTuvu7SV6YWQF5XFXdqqr2nGbQtpxL+bkkj66qO1bVXTKbaU0yO+e1qn51Ki5XZTY7umWm9KLMDvO92TTW+Un+3yR/UVW3rKqfzWyW8i07kfvCzA5JfWVV3baqblZV96qqX17pPqrqyVW1vruvTXLZtPiaJLfJrBhfmllpf+lWT70os/Mud3S/W7t1ZoXukul5T89s5nWL1yd5blX93FQO7z0V3u1mWM33eQWOy6ykPzbJzTM7hPySJFdX1aMyOw95i4uS3GkHDr/e2juS/HFV3aGq7ppZ6QdgjVFeAdhp3f2qJH+U5E8yKx7nZ1YM3jNt8qbMzj08N7PCeMKSp98iycsyO//xm5kVmRdM606cbi+tqtOm+0/KbCbzgswuEvWi7v7ITkZ/amaF6d+SfCfJO5Os6DtHJ4clOatmVwF+bZIndvdVSY7P7DDbb0z73vqc1Tdkdo7vZVX1nlzfcvu9ju7+tySvzGz2+6Ik/zHJvy5Zf2JmF5p6a5LvZfbnccdp9V8k+ZMpw3O3kWE13+dldfe/Z3bu6Z9Ohyn/QWYl8zuZzWC/b8m2X8jsQlJfmXIvd3j5cv4ss4tQfTXJP2X25/3DXX4RAMxVde/qkTgAAGtHVf1eZv8xsOLZdgAWz8wrAHCjVlX7VtVDpkPE75vkOdnJr3gCYHF26gqRAABryM0zu6L0PTI7l/jtSf5moYkA2GEOGwYAAGB4DhsGAABgeMorAAAAw1NeAQAAGJ7yCgAAwPCUVwAAAIanvAIAADC8/x8HmfheysRazQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 9))\n",
    "plt.hist(df_supermarket[df_supermarket['Customer type'] == 'Member']['Rating'],\n",
    "         alpha = 0.4, bins = 30, color = '#003f7e')                                 # members = blue group\n",
    "plt.hist(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Rating'],\n",
    "         alpha = 0.4, bins = 30, color = '#821626')                                 # members = red group\n",
    "plt.xlabel(\"Customer satisfaction 'Rating'\", size = 12)\n",
    "plt.ylabel(\"Count\", size = 12)\n",
    "plt.title(\"Histogram of ratings\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean of members: 6.940319361277447\n",
      "Mean of normals: 7.005210420841684\n",
      "Variance of members: 3.060331097804394\n",
      "Variance of normals: 2.850093278927331\n"
     ]
    }
   ],
   "source": [
    "print('Mean of members: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Member']['Rating'].mean()))\n",
    "print('Mean of normals: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Rating'].mean()))\n",
    "print('Variance of members: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Member']['Rating'].var()))\n",
    "print('Variance of normals: ' + str(df_supermarket[df_supermarket['Customer type'] == 'Normal']['Rating'].var()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, the distributions do not follow the normal distribution, but we proceed again as above already explained. The average values only differs to a limited extent - and on top with a lower satisfaction value for members. Nevertheless, we check for significance here also. The difference in variance does not cause us to drop t-test. The hypothesis are:\n",
    "* **Null hypothesis $H_0$: There is no difference between members and normal buyers in terms of customer satisfaction.**\n",
    "* **Alternative hypothesis $H_1$: The difference in the sample means of the satisfaction rating results from a different customer experience of the two groups and is not random.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=-0.5968211265254104, pvalue=0.5507621727856713)\n"
     ]
    }
   ],
   "source": [
    "print(stats.ttest_ind(df_supermarket[df_supermarket['Customer type'] == 'Member']['Rating'],\n",
    "                      df_supermarket[df_supermarket['Customer type'] == 'Normal']['Rating']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, with $p = 0.551$, the selected significance level is surpassed. The null hypothesis cannot be rejected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion\n",
    "If we look at our initial question (*Does the changed service have a positive effect on customer satisfaction and sales figures and should it therefore be introduced across all markets?*) with regard to the tests just completed, we can state that in the (pretended) case no significant change/improvement could be observed due to the newly introduced service level. Therefore, at least on the basis of this data, we cannot recommend a launch across all markets, because this would imply increased costs.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"study_wholesale\"></a>\n",
    "<div style=\"width:image width px; font-size:75%; text-align:right;\">\n",
    "    <img src=\"img/wholesale_michael-jasmund_unsplash.jpg\" width=\"width\" height=\"height\" style=\"padding-bottom:0.2em;\" />\n",
    "    <figcaption>Photo by Michael Jasmund on Unsplash</figcaption>\n",
    "</div>\n",
    "\n",
    "### Wholesale\n",
    "**Pretended case:** In a wholesale company there are two groups of customers: professional HoReCa which are hotels, restaurant owners, and caterers (hereafter also marked and called as the <span style=\"color:#003f7e\">blue group</span>) and retail customers (hereafter also the <span style=\"color:#821626\">red group</span>). The respective group is assigned a separate entrance to the stores. All other conditions are the same, such as prices, service levels etc. It should be checked whether the different entrances lead to different sales figures in the departments.\n",
    "\n",
    "**Question: Does the different entrance to the market cause a different buying behavior and if so, which positioning of the entrance is sales-promoting?**\n",
    "\n",
    "**Source:** Shekhar, R. (2018). Wholesale customers Data Set, Annual spending in monetary units of clients of a wholesale distributor. *Kaggle*. Retrieved 2020/04/07 from https://www.kaggle.com/binovi/wholesale-customers-data-set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data import and column selection\n",
    "First, the data is imported as a pandas ``DataFrame`` and a selection of columns is made that is appropriate for the pretense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Channel</th>\n",
       "      <th>Fresh</th>\n",
       "      <th>Milk</th>\n",
       "      <th>Grocery</th>\n",
       "      <th>Frozen</th>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <th>Delicassen</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>1</td>\n",
       "      <td>4625</td>\n",
       "      <td>8579</td>\n",
       "      <td>7030</td>\n",
       "      <td>4575</td>\n",
       "      <td>2447</td>\n",
       "      <td>1542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>1</td>\n",
       "      <td>38793</td>\n",
       "      <td>3154</td>\n",
       "      <td>2648</td>\n",
       "      <td>1034</td>\n",
       "      <td>96</td>\n",
       "      <td>1242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>1</td>\n",
       "      <td>25203</td>\n",
       "      <td>11487</td>\n",
       "      <td>9490</td>\n",
       "      <td>5065</td>\n",
       "      <td>284</td>\n",
       "      <td>6854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>2</td>\n",
       "      <td>15076</td>\n",
       "      <td>6257</td>\n",
       "      <td>7398</td>\n",
       "      <td>1504</td>\n",
       "      <td>1916</td>\n",
       "      <td>3113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>1</td>\n",
       "      <td>22096</td>\n",
       "      <td>3575</td>\n",
       "      <td>7041</td>\n",
       "      <td>11422</td>\n",
       "      <td>343</td>\n",
       "      <td>2564</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Channel  Fresh   Milk  Grocery  Frozen  Detergents_Paper  Delicassen\n",
       "136        1   4625   8579     7030    4575              2447        1542\n",
       "377        1  38793   3154     2648    1034                96        1242\n",
       "202        1  25203  11487     9490    5065               284        6854\n",
       "373        2  15076   6257     7398    1504              1916        3113\n",
       "240        1  22096   3575     7041   11422               343        2564"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_wholesale = pd.read_csv('dat/wholesale/wholesale.csv')\n",
    "df_wholesale.drop(['Region'], axis = 1, inplace = True)\n",
    "df_wholesale.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dependent variables are ``Fresh`` to ``Delicassen`` [sic, author's note: We write \"delicatessen\" in the text and keep the attribute name] and the independet test and control group is assigned as ``Channel`` - which is here the different entrance - with <span style=\"color:#003f7e\">HoReCa = 1</span> and <span style=\"color:#821626\">Retail = 2</span>. The attribute ``Region`` is not of interest and is deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are only two groups [2 1] as demanded.\n",
      "There are 298 HoReCa customers in the dataset.\n",
      "There are 142 retail customers in the dataset.\n",
      "The considered objective columns are numeric (int64). Please proceed!\n",
      "Channel             0\n",
      "Fresh               0\n",
      "Milk                0\n",
      "Grocery             0\n",
      "Frozen              0\n",
      "Detergents_Paper    0\n",
      "Delicassen          0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "if (len(df_wholesale['Channel'].unique()) == 2):\n",
    "    print('There are only two groups ' + str(df_wholesale['Channel'].unique()) + ' as demanded.')\n",
    "else:\n",
    "    print('There are not exactly two groups! Please check the data.')\n",
    "    \n",
    "print('There are ' + str(len(df_wholesale[df_wholesale['Channel'] == 1])) + ' HoReCa customers in the dataset.')\n",
    "print('There are ' + str(len(df_wholesale[df_wholesale['Channel'] == 2])) + ' retail customers in the dataset.')\n",
    "\n",
    "if ((df_wholesale['Fresh'].dtype == 'int64') &\n",
    "    (df_wholesale['Milk'].dtype == 'int64') &\n",
    "    (df_wholesale['Grocery'].dtype == 'int64') &\n",
    "    (df_wholesale['Frozen'].dtype == 'int64') &\n",
    "    (df_wholesale['Detergents_Paper'].dtype == 'int64') &\n",
    "    (df_wholesale['Delicassen'].dtype == 'int64')\n",
    "   ):\n",
    "    print('The considered objective columns are numeric (int64). Please proceed!')\n",
    "else:\n",
    "    print('Check the type of the objective columns.')\n",
    "    \n",
    "print(df_wholesale.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the test and control group are the only channels, albeit in varying numbers. Moreover, the measurands are integers and there is no missing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample means, variances and distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Channel</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fresh</th>\n",
       "      <td>13475.560403</td>\n",
       "      <td>8904.323944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Milk</th>\n",
       "      <td>3451.724832</td>\n",
       "      <td>10716.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grocery</th>\n",
       "      <td>3962.137584</td>\n",
       "      <td>16322.852113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frozen</th>\n",
       "      <td>3748.251678</td>\n",
       "      <td>1652.612676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <td>790.560403</td>\n",
       "      <td>7269.507042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Delicassen</th>\n",
       "      <td>1415.956376</td>\n",
       "      <td>1753.436620</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             0             1\n",
       "Channel               1.000000      2.000000\n",
       "Fresh             13475.560403   8904.323944\n",
       "Milk               3451.724832  10716.500000\n",
       "Grocery            3962.137584  16322.852113\n",
       "Frozen             3748.251678   1652.612676\n",
       "Detergents_Paper    790.560403   7269.507042\n",
       "Delicassen         1415.956376   1753.436620"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df_wholesale[df_wholesale.Channel == 1].mean(),\n",
    "           df_wholesale[df_wholesale.Channel == 2].mean()], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Channel</th>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fresh</th>\n",
       "      <td>1.913156e+08</td>\n",
       "      <td>8.077902e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Milk</th>\n",
       "      <td>1.894135e+07</td>\n",
       "      <td>9.369526e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grocery</th>\n",
       "      <td>1.257067e+07</td>\n",
       "      <td>1.504871e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frozen</th>\n",
       "      <td>3.185375e+07</td>\n",
       "      <td>3.286257e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Detergents_Paper</th>\n",
       "      <td>1.219023e+06</td>\n",
       "      <td>3.957781e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Delicassen</th>\n",
       "      <td>9.906296e+06</td>\n",
       "      <td>3.817323e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             0             1\n",
       "Channel           0.000000e+00  0.000000e+00\n",
       "Fresh             1.913156e+08  8.077902e+07\n",
       "Milk              1.894135e+07  9.369526e+07\n",
       "Grocery           1.257067e+07  1.504871e+08\n",
       "Frozen            3.185375e+07  3.286257e+06\n",
       "Detergents_Paper  1.219023e+06  3.957781e+07\n",
       "Delicassen        9.906296e+06  3.817323e+06"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df_wholesale[df_wholesale.Channel == 1].var(),\n",
    "           df_wholesale[df_wholesale.Channel == 2].var()], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8EAAAImCAYAAABzfoiOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde7ilZ10f/O+PhFOASGJ2Yg6EQZqiYDl1i6dWaCKClpC8XiLBFxkhOlJBocVC0Kuv9LVpUwtaKh46CpKIQiIKSawH0iDwUgTcAUQg0KAkJDAkw0kOQWji7/1jPQOLYc/Mzuy91tqzns/nuta11nNcv7V35s767vt+7qe6OwAAADAGd1p0AQAAADAvQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBDMaVfWCqnrFousAOBxV9RtV9e+G14+qqpumtl1fVd+zuOoA4MghBLOtDF/kvlBVn5t6nLLougBmaWj7vlRVJ+y3/l1V1VW1o7uf3t2/sKgaAQ6kqs6rqrdV1eer6pbh9U9WVS26NliPEMx2dHZ333Pq8dF9G6rq6EUWBjBDH0rypH0LVfVPktx9ceUAHFpVPSfJi5P8lyTfkOSkJE9P8l1J7rLO/kfNoaaqKjmHA/IfB9ve0AvyjKq6Lsl1w7rHDT0kn66qt1TVg6f2f15VfaSqPltVH6iqs6ZOd5equmTY9t6qWp335wE4gN9J8pSp5Z1JLtm3UFUvr6r/cKiTVNU3VdWHquq8GdQI8GVV9XVJ/t8kP9ndr+7uz/bEO7v7/+7uLw5t169X1R9X1eeT/Iuq+uaqesPwPe69VfX4qXPevapeVFU3VNXfVdWbq+ruw7ZvH773fbqq/qqqHjV13Buq6sKq+l9Jbk3ynKq6Zr96n1NVr53Hz4btTQjmSHFukm9L8sCqeniSlyX5iSRfn+S/J7miqu5aVQ9I8swk39rd90rymCTXT53n8UleleTeSa5I8pK5fQKAg3trkmOHL4dHJXlikjs0j8HQPr4uyU9196tmUCPAtO9Ictcklx9ivx9OcmGSeyV5W5IrM2mrTkzyU0l+d/gOlyQvTPJPk3xnkuOTPDfJP1TVqUn+R5L/MKz/mSR/UFUrU+/zI0l2De/z35Lcr6q+eWr7kzP5gyMjJwSzHb12+Avfp6f+WvefuvuT3f2FJD+e5L9399u6+/buvjjJF5N8e5LbM2mMH1hVd+7u67v7b6bO/ebu/uPuvj2TRvAhc/xcAIeyrzf40Unen+Qjd+DYf57JH/d2dvcfzaA2gP2dkOTj3X3bvhVTPbVfqKrvHlZf3t3/q7v/IclDk9wzyUXd/aXufn2SP0rypGEI89OSPKu7PzJ8z3tLd38xkwD7x8P3uH/o7quSrCX5/ql6Xt7d7+3u24ZjLh2OS1U9KMmO4b0YOSGY7ejc7r738Dh3WHfj1Pb7ZjLEZV9Q/nSS+yQ5pbs/mOTZSV6Q5JaqetV+E2t9bOr1rUnu5jpjYBv5nUx6TH40U0OhN+jpSd7S3X++1UUBHMAnkpww/V2qu7+zu+89bNuXNaa/x52S5MYhEO9zQ5JTMwnVd0sy3YGxz32TPGG/73//LMnJU/vcuN8xFyf54WGCrh9JctkQjhk5IZgjRU+9vjHJhVNB+d7dfUx3vzJJuvv3uvufZdJYdpL/vIB6Ae6w7r4hkwmyvj/JH97Bw5+e5PSq+uUtLwxgfX+RyWi8cw6x3/T3uI8muc9+E1ednsnIl48n+fsk91/nHDcm+Z39vv/do7svOsD7pLvfmuRLmYyU+eEYCs1ACOZI9JtJnl5V3zbM/nePqvqXVXWvqnpAVZ1ZVXfNpBH9QiZDpAGOFOcnObO7P38Hj/tskscm+e6quuhQOwNsVnd/Osm/T/JrVfWDVXXPqrpTVT00yT0OcNjbknw+yXOr6s7D5FZnJ3nV0Dv8siS/VFWnVNVRVfUdw/e6VyQ5u6oeM6y/W03umX7aIcq8JJM5YG7r7jdv/lOzDIRgjjjdvZbJdcEvSfKpJB/MZOhgMrke+KJM/pL4sUwmXPjZ+VcJcHi6+2+Gdu5wjv10JtcTf19VuacwMHPd/YtJ/k0mE1jdkuTmTCYtfV6St6yz/5cymaj0+zL5vvZrSZ7S3e8fdvmZJH+d5C+TfDKTEX136u4bM+lx/tkkezPpGf63OXSe+Z0k3xK9wEyp7j70XgAAAEeY4fZKtyR5eHdft+h62B70BAMAAMvqXyX5SwGYaWbFBQAAlk5VXZ+kkpx7iF0ZGcOhAQAAGA3DoQEAABgNIRgAAIDROKKvCT7hhBN6x44diy4D2Gauueaaj3f3yqLr2CraOmA9y9bWJdo74GvNoq07okPwjh07srZ2WLdSBJZYVd2w6Bq2krYOWM+ytXWJ9g74WrNo6wyHBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEbj6EUXME+7rzz4zdd3nb06p0oAZme9tk77Biy7fW2f9g44FD3BAAAAjIYQDAAAwGjMPQRX1QOq6l1Tj89U1bOr6viquqqqrhuej5t3bQAAACy3uYfg7v5Adz+0ux+a5J8muTXJa5JckOTq7j4jydXDMgAAAGyZRQ+HPivJ33T3DUnOSXLxsP7iJOcurCoAAACW0qJD8HlJXjm8Pqm79yTJ8HziegdU1a6qWquqtb17986pTAAAAJbBwkJwVd0lyeOT/P4dOa67d3f3anevrqyszKY4AAAAltIie4K/L8k7uvvmYfnmqjo5SYbnWxZWGQAAAEtpkSH4SfnKUOgkuSLJzuH1ziSXz70iAAAAltpCQnBVHZPk0Un+cGr1RUkeXVXXDdsuWkRtAAAALK+jF/Gm3X1rkq/fb90nMpktGgAAAGZi0bNDA7AOM+EDAMyGEAywDZkJHwBgNoRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAWpqp2VdVaVa3t3bt30eUAIyAEAwCwMN29u7tXu3t1ZWVl0eUAIyAEAwAAMBpCMAAAAKMhBAMAADAaQjAAAACjsZAQXFX3rqpXV9X7q+raqvqOqjq+qq6qquuG5+MWURsAAADLa1E9wS9O8qfd/U1JHpLk2iQXJLm6u89IcvWwDAAAAFtm7iG4qo5N8t1JXpok3f2l7v50knOSXDzsdnGSc+ddGwAAAMttET3B35hkb5Lfrqp3VtVvVdU9kpzU3XuSZHg+cb2D3VAdAACAw7WIEHx0kocn+fXufliSz+cODH12Q3UAAAAO1yJC8E1Jburutw3Lr84kFN9cVScnyfB8ywJqAwAAYInNPQR398eS3FhVDxhWnZXkfUmuSLJzWLczyeXzrg0AAIDldvSC3venkvxuVd0lyd8meWomgfyyqjo/yYeTPGFBtQEAALCkFhKCu/tdSVbX2XTWvGsBAABgPBZ1n2AAANhyu69cy+4r1xZdBrCNCcEAAACMhhAMAADAaAjBAAAAjIYQDAAAwGgIwQAAAIyGEAwAAMBoCMEAAACMhhAMAADAaAjBAAAAjIYQDAAAwGgIwQDbUFXtqqq1qlrbu3fvossBAFgaQjDANtTdu7t7tbtXV1ZWFl0OAMDSEIIBAAAYDSEYAACA0Th60QVsJ7uvXDvo9l1nr86pEgAAAGZBTzAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKOxkNmhq+r6JJ9NcnuS27p7taqOT3Jpkh1Jrk/yQ939qUXUB7Bs1pv93oz3AMAYLbIn+F9090O7e9+3sAuSXN3dZyS5elgGAGCJVdWuqlqrqrW9e/cuuhxgBLbTcOhzklw8vL44ybkLrAUAgDno7t3dvdrdqysrK4suBxiBRYXgTvK6qrqmqnYN607q7j1JMjyfuN6B/loIAADA4VrINcFJvqu7P1pVJya5qqrev9EDu3t3kt1Jsrq62rMqEAAAgOWzkJ7g7v7o8HxLktckeUSSm6vq5CQZnm9ZRG0AAAAsr7mH4Kq6R1Xda9/rJN+b5D1Jrkiyc9htZ5LL510bAAAAy20Rw6FPSvKaqtr3/r/X3X9aVX+Z5LKqOj/Jh5M8YQG1AQAAsMTmHoK7+2+TPGSd9Z9Icta86wEAAGA8ttMtkgAAAGCmhGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEbj6EUXcCTZfeXaQbfvOnt1TpUAAABwOPQEAwAAMBpCMAAAAKMhBAMAsHQOdRkbMF5CMAAAAKMhBAMAADAamwrBVXX1RtYBAADAdnBYt0iqqrslOSbJCVV1XJIaNh2b5JQtqg0AAAC21OHeJ/gnkjw7k8B7Tb4Sgj+T5Fc3coKqOirJWpKPdPfjqur4JJcm2ZHk+iQ/1N2fOsz6AAAA4Gsc1nDo7n5xd98vyc909zd29/2Gx0O6+yUbPM2zklw7tXxBkqu7+4wkVw/LAKNUVbuqaq2q1vbu3bvocgAAlsbh9gQnSbr7V6rqOzPpvT16av0lBzuuqk5L8i+TXJjk3wyrz0nyqOH1xUnekOR5m6kP4EjV3buT7E6S1dXVXnA5AKPyjksuPej2hz/liXOqBJiFTYXgqvqdJPdP8q4ktw+rO8lBQ3CS/5rkuUnuNbXupO7ekyTdvaeqTjzAe+5KsitJTj/99MMvHgAAgNHZVAhOsprkgd294V6Kqnpcklu6+5qqetQdfUO9IwAAAByuzYbg9yT5hiR77sAx35Xk8VX1/UnuluTYqnpFkpur6uShF/jkJLdssjYAAAD4Kpu6T3CSE5K8r6r+rKqu2Pc42AHd/fzuPq27dyQ5L8nru/vJSa5IsnPYbWeSyzdZGwAAAHyVzfYEv2ArihhclOSyqjo/yYeTPGELzw0AAACbnh36jZs8/g2ZzAKd7v5EkrM2cz4AAAA4mM3ODv3ZTGaDTpK7JLlzks9397GbLQwAgOXnzh/AvG22J3j6FkepqnOTPGJTFQEAMBru/AHM22Ynxvoq3f3aJGdu5TkBAABgq2x2OPQPTC3eKZP7BvsLHgAAANvSZmeHPnvq9W1Jrk9yzibPCQAAADOx2WuCn7pVhQAAAMCsbeqa4Ko6rapeU1W3VNXNVfUHVXXaVhUHAAAAW2mzE2P9dpIrkpyS5NQkVw7rAAAAYNvZ7DXBK909HXpfXlXP3uQ5AQBgQ3ZfubboEoAjzGZ7gj9eVU+uqqOGx5OTfGIrCgMAAICtttme4KcleUmSX87k1khvSWKyLIAjwHq9J7vOXl1AJQAA87PZEPwLSXZ296eSpKqOT/LCTMIxAAAAbCubHQ794H0BOEm6+5NJHrbJcwIAAMBMbDYE36mqjtu3MPQEb7Z3GQAAAGZis4H1RUneUlWvzuSa4B9KcuGmqwIAAIAZ2FQI7u5LqmotyZlJKskPdPf7tqQyAAAA2GKbHro8hF7BFwAAgG1vs9cEAwAAwBFDCAYAAGA0hGAAAABGY+4huKruVlVvr6q/qqr3VtW/H9YfX1VXVdV1w/NxhzoXAAAA3BGLuKfvF5Oc2d2fq6o7J3lzVf1Jkh9IcnV3X1RVFyS5IMnzFlDfYdt95dpBt+86e3VOlQAAALCeufcE98TnhsU7D49Ock6Si4f1Fyc5d961AQAAsNwWck1wVR1VVe9KckuSq7r7bUlO6u49STI8n7iI2gAAAFheCwnB3X17dz80yWlJHlFV37LRY6tqV1WtVdXa3r17Z1ckAAAAS2ehs0N396eTvCHJY5PcXFUnJ8nwfMsBjtnd3avdvbqysjK3WgEAADjyLWJ26JWquvfw+u5JvifJ+5NckWTnsNvOJJfPuzYAAACW2yJmhz45ycVVdVQmIfyy7v6jqvqLJJdV1flJPpzkCVv9xre+8fVffn3MI8/c6tMDAACwzc09BHf3u5M8bJ31n0hy1rzrAQAAYDwWek0wAAAAzJMQDAAAwGgIwQAAAIyGEAywDbknOgDAbAjBANuQe6IDAMzGIm6RBAAAC/OOSy6d2fEPf8oTN3VuYPaE4DnafeXaQbfvOnt1TpUAAACMkxAMwAEdqrdEjwcAcKRxTTAAAACjIQQDALAwZsMH5k0IBgBgYcyGD8ybEAwAAMBomBgLAIAjzqHuujG9z747cOxbdj8OGDc9wQAAAIyGEAwAAMBoCMEAAACMhhAMAADAaAjBAAAAjIYQDAAAwGgIwQAAAIyGEAwAAMBozD0EV9V9qurPq+raqnpvVT1rWH98VV1VVdcNz8fNuzYAAACW2yJ6gm9L8pzu/uYk357kGVX1wCQXJLm6u89IcvWwDAAAAFvm6Hm/YXfvSbJneP3Zqro2yalJzknyqGG3i5O8Icnz5l0fAADLZfeVa4suAdhG5h6Cp1XVjiQPS/K2JCcNATndvaeqTjzAMbuS7EqS008/fT6FzsmhGuhdZ6/OqRIAAIDltLCJsarqnkn+IMmzu/szGz2uu3d392p3r66srMyuQAAAAJbOQkJwVd05kwD8u939h8Pqm6vq5GH7yUluWURtAAAALK9FzA5dSV6a5Nru/qWpTVck2Tm83pnk8nnXBgAAwHJbxDXB35XkR5L8dVW9a1j3s0kuSnJZVZ2f5MNJnrCA2gAAAFhii5gd+s1J6gCbz5pnLQAAAIzLwibGAgAAgHkTggEAABgNIRgAAIDRWMTEWNvCrW98/VctH/PIMxdUCcD2sfvKta9aXl1QHQAAs6InGAAAgNEYbU8wAF87KuZrPPi+8ykEYEm845JLD7r94U954pwqAQ5ETzAAAACjIQQDAAAwGoZDAwDAnBguDYunJxgAAIDREIIBAAAYDcOhAZgZw/4AgO1GTzAAAKP0pnffkDe9+4ZFlwHMmZ5ggG2oqnYl2ZUkp59++oKrObBD9fQCAGw3eoIBtqHu3t3dq929urKysuhyAACWhp7gddz6xtd/+fUxjzxzgZUAAACwlYRgAAAWZlaXf0x3anyNB993y95nq5lQEGbPcGgAABbG5R/AvOkJPoLsvnLtoNt3nb06p0oAAACOTHqCAQAAGI2FhOCqellV3VJV75lad3xVXVVV1w3Pxy2iNgAAxsW9gmFcFjUc+uVJXpLkkql1FyS5ursvqqoLhuXnzaugg06esM4+Zo0GAAA48iykJ7i735Tkk/utPifJxcPri5OcO9eiAAAAWHrb6Zrgk7p7T5IMzycuuB4AAACWzBE3O/Ss7iW3DMweDQAAcHDbKQTfXFUnd/eeqjo5yS3r7dTdu5PsTpLV1dWeZ4EAY7PeZDHf/eD7LqASAICtsZ2GQ1+RZOfwemeSyxdYCwAAAEtoIT3BVfXKJI9KckJV3ZTk55NclOSyqjo/yYeTPGERtQEAsNzWG+Wyb53RLrD8FhKCu/tJB9h01lwLAQAAYFS20zXB29JG7h98sGO20/2ETZwFAACM3Xa6JhgAAABmSk8wwBI7nNEsAByZ3nHJpQfd/vCnPHFOlcD2JgRvgWUaMg0AALDMhGAA7hD3DgYAjmRCMAAAHCEONeQZODQh+DC5zg4AAODIY3ZoAAAARkMIBgAAYDQMh56j7T6EeveVawfdvuvs1TlVAgCwvbzp3TeYBBCWhBAMwMIcbIIX97MEAGbBcGgAAABGQ0/wNrb/8OljHnnmgirZGMOpAQCA7U4IBgCAwZvefcNXLS/TdcCHusewy1AYCyEYgG1ps1/WfNkDANYjBB9BpodHL2Jo9KGGOy+a4dgAAMChCMEAzMQyDykEAI5cQjAAANwB+/7It++Pe/svH2jddue2dYyFELzN7D8j9GaO326zSRuuDMtr/15fAIDtyn2CAQAAGA09wQBse+v1NK/tN7rkjo4mOdTs0Ztl6CAsh+n2Z/+26FDLB1q3/xDpI2EOhc22mZuZ0X+z7elmap/ley/r/yeOhLszbLsQXFWPTfLiJEcl+a3uvmjBJW3KRoY3H84Q6HkMmz7QPgd77wPtt5Gh2dt59unNDuU2FPzg/HwAAJiXbRWCq+qoJL+a5NFJbkryl1V1RXe/b7GVAbBZG71ueCt7QA6nB2YzNUyfa19P9eH8EWfWPS7J+n982lfrIv+KfyT0IByuZf5sAEeS7XZN8COSfLC7/7a7v5TkVUnOWXBNAAAALInq7kXX8GVV9YNJHtvdPzYs/0iSb+vuZ07tsyvJrmHxAUk+cAfe4oQkH9+icmftSKn1SKkzUessbNc679vdK4suYjNG1NbNip+Bn0Gy/D+DI76tSzbV3i3b73eZPs8yfZZkuT7PkfhZtryt224h+AlJHrNfCH5Ed//UFp1/rbuPiIsLj5Raj5Q6E7XOwpFS59j4vfgZJH4GiZ/Bslu23+8yfZ5l+izJcn2eZfosm7HdhkPflOQ+U8unJfnogmoBAABgyWy3EPyXSc6oqvtV1V2SnJfkigXXBAAAwJLYVrNDd/dtVfXMJH+WyS2SXtbd793Ct9i9heeatSOl1iOlzkSts3Ck1Dk2fi9+BomfQeJnsOyW7fe7TJ9nmT5LslyfZ5k+y2HbVtcEAwAAwCxtt+HQAAAAMDNCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAPBlVfWjVfXmRdcBbH9V9aiqumlq+b1V9agFlgQbIgSzUFV1fVV9oao+N/U4ZdF1AdvPVHvx2ar6dFW9paqeXlUb+n/Z/l/WtpuqekNV/dii6wDGZbNt67TuflB3v2GT9bygql6xmXPAoQjBbAdnd/c9px4f3behqo5eZGHAtnN2d98ryX2TXJTkeUleOo831h5tH34XsOUW1rbCIgjBbDtV1VX1jKq6Lsl1w7ofr6oPVtUnq+qKfb3FVfXc/XqR/09VvXzY9nVV9dKq2lNVH6mq/1BVRw3bfrSq3lxVL6yqT1XVh6rq+xb1mYE7prv/rruvSPLEJDur6luSpKruOvy7/nBV3VxVv1FVd6+qeyT5kySnTI86qao7VdUFVfU3VfWJqrqsqo4fzrVjaI/Or6oPJ3l9VR1VVS+qqo8P7cYzh32OHo45rHanqi5M8s+TvGSo7SU18ctVdUtV/V1VvXvf59xfVT21qq4denL+tqp+Ymrbo6rqpqp6znCuPVX11KntXz+0q5+pqrcnuf+Bfu5TP5NdVfXR4VzPmdr+iKr6i6E3ac/wOe4ytb2r6qeHGj9eVf9lurepqp42fI5PVdWfVdV99zv2q/7fAGyt9drWA7Wr6x1fk17l7xleH1VVPzu0r5+tqmuq6j7DthdX1Y1Du3NNVf3zYf1jk/xskicObeFfDesP1rb+o6p649BOfryqLh3WH7ANPdhnOlSbyXIQgtmuzk3ybUkeWFVnJvlPSX4oyclJbkjyqiTp7l/c14Oc5JuT7E1y2XCOi5PcluQfJXlYku9NMj3U8NuSfCDJCUl+MclLq6pm/LmALdTdb09yUyYBMkn+c5J/nOShmfzbPzXJ/9Pdn0/yfUk+ut+ok5/OpL15ZJJTknwqya/u9zaPzKR9eUySHx/O89AkDx+OnXZY7U53/1yS/y/JM4fanjkc+93D57l3Jl9KP3GAH8UtSR6X5NgkT03yy1X18Knt35Dk64afx/lJfrWqjhu2/WqSv8+kfX3a8DiUf5HkjKHGC/Z96U1ye5J/PXy+70hyVpKf3O/Y/yvJaiY/v3P2vV9VnZvJl98fSLIy/Dxeud+xX/5/wwZqBA7Tfm3ruu3qBk7zb5I8Kcn3Z9I2PS3JrcO2vxzOd3yS30vy+1V1t+7+0yT/McmlQ1v4kGH/g7Wtv5DkdUmOS3Jakl8Z1h+sDT3UZzpYm8ky6G4Pj4U9klyf5HNJPj08Xpukk5w5tc9Lk/zi1PI9k/yfJDum1t09yTVJnjcsn5Tki0nuPrXPk5L8+fD6R5N8cGrbMcP7fsOifyYeHh7rP4b24nvWWf/WJD+XpJJ8Psn9p7Z9R5IPDa8fleSm/Y69NslZU8snD+3L0Ul2DO3CN05tf32Sn5ha/p5hn6M32+4keUOSH5vafmaS/53k25Pc6Q7+rF6b5FlTn/sLSY6e2n7LcN6jhs/7TVPb/mOSNx/gvPt+JtP7/2KSlx5g/2cnec3Ucid57NTyTya5enj9J0nOn9p2p0y+MN936tgzD/SZPTw8Du+xgbZ1w+3q9Lky+YPfORus4VNJHjK8fkGSV0xtO1TbekmS3UlO2++c67ah2dj/K9ZtMxf9u/LYuodratgOzu3u/7lvoao6yY1T209J8o59C939uar6RCZ/nbt+WP3SJB/o7v88LN83yZ2T7Jnq3L3Tfuf92NQ5bx32u+cWfB5gvk5N8slMeg+PSXLN1L/7yiToHch9k7ymqv5hat3tmXzp2mf/9ujGA2zb0nanu19fVS/JpKf29Kp6TZKf6e7P7L9vTYZV/3wmPRt3yuTn8NdTu3yiu2+bWr51eN+VTAL8dI03rFfPfvbf/58MdfzjJL+USU/vMcO5rznEsfsmQ7xvkhdX1YumP1omv98b1jkWmK1TM/k3fEfb1X3uk+Rv1tswXEbxY5n8++9MeopPOMB5DtW2PjeT3uC3V9Wnkryou192oDY0yd028JkO1GayJAyHZrvqqdcfzaQBTJLU5Nq+r0/ykWH5giQPyGS4yj43ZvJXwxO6+97D49juftDMKwfmpqq+NZMvam9O8vFM/nr/oKl/91/Xk8slkq9uV/a5Mcn3Te1/7+6+W3d/ZGqf6eP2ZDLcbp/77HeuzbQ7X1Nfd/+37v6nSR6UScD9t/vvU1V3TfIHSV6Y5KTuvneSP87kS92h7M1kiOH05zh9A8ftv/++CQ1/Pcn7k5zR3cdmMrx5/zoOdOyNmfSyT/8u7t7db5naf73fIbDFptrW1+bg7erB3Jh15vQW0H8AAB/BSURBVBgYrv99XiaXuR03tFl/l6+0Ffv/Oz9o29rdH+vuH+/uU5L8RJJfq6p/NGxbrw091P8rGAEhmCPB7yV5alU9dPiy9x+TvK27rx96P346k97kL+w7oLv3ZHJ9yIuq6tiaTH5z/6p65EI+AbClhn/Xj8tkfoBXdPdfd/c/JPnNTK6HPXHY79Sqesxw2M1Jvr6qvm7qVL+R5MJ9EzBV1UpVnXOQt74sybOG8947ky9ySbak3bk5yTdOfcZvrapvq6o7ZzJ07+8z6aXe312S3DVDoB3axe/dyBt29+1J/jDJC6rqmKp6YJKdGzj03w37PyiTa5AvHdbfK8lnknyuqr4pyb9a59h/W1XHDRPkPGvq2N9I8vzhnPsmwnnCRj4HsDXWaVv/KgdvVw/mt5L8QlWdMUxS9eCq+vpM2onbMmmzjq6q/yeTnuB9bk6yo4ZJ8w7VtlbVE6pq3x8nP5VJiL79QG3oBv5fwQgIwWx73X11kn+XSU/Hnkz+qnjesPmJmQznu7a+MuPrbwzbnpLJl8P3ZdIovjqT6/2AI9eVVfXZTHoGfi6TobfTs3Y+L8kHk7y1qj6T5H9mMlIk3f3+TCZa+tuazF58SpIXJ7kiyeuG8741k4mXDuQ3M/ky9u4k78ykx/W2fCWcbqbdeXGSH6zJzMj/LZMvhb85nOeGTCZ0eeH+B3X3ZzP5Y+Blw74/PHymjXpmJsP8Ppbk5Ul+ewPHvDGTn/PVSV7Y3a8b1v/M8P6fHWq/dJ1jL89kiPS7kvyPDLdh6e7XZDJZzauG3917MpmEDJi9g7WtB2xXD+GXMmmXXpfJH8demskcLn+WyRwA/zuTtu3v89WXOvz+8PyJqtp3OdzB2tZvTfK2qvpcJm3fs7r7Qzl4G3q4n4klUd1GFgHA4Rh6XX+ju+97yJ2XQFXtSPKhJHfe73q5jR7fmQyV/uAWlwYAG6YnGAA2qCb3HP7+qjq6qk7NZDKq1yy6LgBg44RgANi4SvLvMxle985MbrG0kftlAgDbhOHQAAAAjIaeYAAAAEbj6EUXsBknnHBC79ixY9FlANvMNddc8/HuXll0HVtFWwesZ9naukR7B3ytWbR1R3QI3rFjR9bW1hZdBrDNVNUNi65hK2nrgPUsW1uXaO+ArzWLts5waAAAAEZDCAYAAGA0hGCAbaiqdlXVWlWt7d27d9HlAAAsDSEYYBvq7t3dvdrdqysrSzXvDQDAQgnBAAAAjIYQDAAAwGgIwQAAAIyGEAwAAMBoCMEAAACMxtGLLmCedl+5dtDtu85enVMlALOzr63TpgHLzPc64HDpCQYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGY2YhuKpeVlW3VNV71tn2M1XVVXXC1LrnV9UHq+oDVfWYWdUFAADAeM2yJ/jlSR67/8qquk+SRyf58NS6ByY5L8mDhmN+raqOmmFtAAAAjNDMQnB3vynJJ9fZ9MtJnpukp9adk+RV3f3F7v5Qkg8mecSsagMAAGCc5npNcFU9PslHuvuv9tt0apIbp5ZvGtYBAADAljl6Xm9UVcck+bkk37ve5nXW9TrrUlW7kuxKktNPP33L6gMAAGD5zbMn+P5J7pfkr6rq+iSnJXlHVX1DJj2/95na97QkH13vJN29u7tXu3t1ZWVlxiUDAACwTOYWgrv7r7v7xO7e0d07Mgm+D+/ujyW5Isl5VXXXqrpfkjOSvH1etQEAADAOs7xF0iuT/EWSB1TVTVV1/oH27e73JrksyfuS/GmSZ3T37bOqDQAAgHGa2TXB3f2kQ2zfsd/yhUkunFU9AAAAMNfZoQEAAGCRhGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAYGGqaldVrVXV2t69exddDjACQjAAAAvT3bu7e7W7V1dWVhZdDjACQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBANsQ1W1q6rWqmpt7969iy4HAGBpzCwEV9XLquqWqnrP1Lr/UlXvr6p3V9VrqureU9ueX1UfrKoPVNVjZlUXwJGgu3d392p3r66srCy6HACApTHLnuCXJ3nsfuuuSvIt3f3gJP87yfOTpKoemOS8JA8ajvm1qjpqhrUBAAAwQjMLwd39piSf3G/d67r7tmHxrUlOG16fk+RV3f3F7v5Qkg8mecSsagMAAGCcFnlN8NOS/Mnw+tQkN05tu2lY9zVcJwcAAMDhWkgIrqqfS3Jbkt/dt2qd3Xq9Y10nBwAAwOE6et5vWFU7kzwuyVndvS/o3pTkPlO7nZbko/OuDQAAgOU2157gqnpskucleXx33zq16Yok51XVXavqfknOSPL2edYGAADA8ptZT3BVvTLJo5KcUFU3Jfn5TGaDvmuSq6oqSd7a3U/v7vdW1WVJ3pfJMOlndPfts6oNAACAcZpZCO7uJ62z+qUH2f/CJBfOqh4AAABY5OzQAAAAMFdCMAAAAKMhBAMAADAaQjAAAACjIQQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMsqd1Xri26BACAbUcIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGY2YhuKpeVlW3VNV7ptYdX1VXVdV1w/NxU9ueX1UfrKoPVNVjZlUXAAAA4zXLnuCXJ3nsfusuSHJ1d5+R5OphOVX1wCTnJXnQcMyvVdVRM6wNAACAEZpZCO7uNyX55H6rz0ly8fD64iTnTq1/VXd/sbs/lOSDSR4xq9oAAAAYp3lfE3xSd+9JkuH5xGH9qUlunNrvpmEdAAAAbJntMjFWrbOu192xaldVrVXV2t69e2dcFgAAAMtk3iH45qo6OUmG51uG9Tcluc/Ufqcl+eh6J+ju3d292t2rKysrMy0WAIDZ0sEBzNu8Q/AVSXYOr3cmuXxq/XlVddequl+SM5K8fc61AQAwZzo4gHk7elYnrqpXJnlUkhOq6qYkP5/koiSXVdX5ST6c5AlJ0t3vrarLkrwvyW1JntHdt8+qNgAAAMZpZiG4u590gE1nHWD/C5NcOKt6AAAAYLtMjAUAAAAzJwQDAAAwGkIwAAAAoyEEAwAAMBpCMAAAAKMhBAMAADAaQjDANlRVu6pqrarW9u7du+hyAACWhhAMsA119+7uXu3u1ZWVlUWXAwCwNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNDYUgqvq6o2sAwAAgO3s6INtrKq7JTkmyQlVdVySGjYdm+SUGdcGAAAAW+qgITjJTyR5diaB95p8JQR/JsmvzrAuAAAA2HIHDcHd/eIkL66qn+ruX5lTTQAAADATh+oJTpJ0969U1Xcm2TF9THdfMqO6AAAAYMttKARX1e8kuX+SdyW5fVjdSYRgAAAAjhgbCsFJVpM8sLt7K960qv51kh/LJEj/dZKnZjIB16WZ9DZfn+SHuvtTW/F+AAAAkGz8PsHvSfINW/GGVXVqkp9Ostrd35LkqCTnJbkgydXdfUaSq4dlAAAA2DIb7Qk+Icn7qurtSb64b2V3P34T73v3qvo/mfQAfzTJ85M8ath+cZI3JHneYZ4fAAAAvsZGQ/ALtuoNu/sjVfXCJB9O8oUkr+vu11XVSd29Z9hnT1WduN7xVbUrya4kOf3007eqLAAAAEZgo7NDv3Gr3rCqjktyTpL7Jfl0kt+vqidv9Pju3p1kd5Ksrq5uyTXKAAAAjMNGZ4f+bCaTWCXJXZLcOcnnu/vYw3jP70nyoe7eO5z7D5N8Z5Kbq+rkoRf45CS3HMa5AQAA4IA22hN8r+nlqjo3ySMO8z0/nOTbq+qYTIZDn5VkLcnnk+xMctHwfPlhnh8AAADWtdFrgr9Kd7+2qg5r9ubufltVvTrJO5LcluSdmQxvvmeSy6rq/EyC8hMO5/wAAABwIBsdDv0DU4t3yuS+wYd9PW53/3ySn99v9Rcz6RUGAACAmdhoT/DZU69vS3J9JpNbAQAAwBFjo9cEP3XWhQAAAMCs3WkjO1XVaVX1mqq6papurqo/qKrTZl0cAAAAbKUNheAkv53kiiSnJDk1yZXDOgAAADhibDQEr3T3b3f3bcPj5UlWZlgXAAAAbLmNhuCPV9WTq+qo4fHkJJ+YZWEAAACw1TYagp+W5IeSfCzJniQ/mMRkWQAAABxRNnqLpF9IsrO7P5UkVXV8khdmEo4BAADgiLDRnuAH7wvASdLdn0zysNmUBAAAALOx0RB8p6o6bt/C0BO80V5kAAAA2BY2GmRflOQtVfXqJJ3J9cEXzqwqAAAAmIENheDuvqSq1pKcmaSS/EB3v2+mlQEAAMAW2/CQ5iH0Cr4AAAAcsTZ6TTAAAAAc8UxuNWX3lWsH3b7r7NU5VQIAMA5VtSvJriQ5/fTTF1wNMAZ6ggEAWJju3t3dq929urKysuhygBEQggEAABgNIRgAAIDREIIBAAAYjYWE4Kq6d1W9uqreX1XXVtV3VNXxVXVVVV03PB+3iNoAAABYXovqCX5xkj/t7m9K8pAk1ya5IMnV3X1GkquHZYBRqqpdVbVWVWt79+5ddDkAAEtj7iG4qo5N8t1JXpok3f2l7v50knOSXDzsdnGSc+ddG8B2YbZUAIDZWERP8Dcm2Zvkt6vqnVX1W1V1jyQndfeeJBmeT1xAbQAAACyxRYTgo5M8PMmvd/fDknw+d2DosyGCAAAAHK5FhOCbktzU3W8bll+dSSi+uapOTpLh+Zb1DjZEEAAAgMM19xDc3R9LcmNVPWBYdVaS9yW5IsnOYd3OJJfPuzYAAACW29ELet+fSvK7VXWXJH+b5KmZBPLLqur8JB9O8oQF1QYAAMCSWkgI7u53JVldZ9NZ864FAACA8VjUfYIBAABg7oRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEZDCAYAAGA0hGAAAABGQwgGAABgNIRgAAAARkMIBgAAYDSEYAAAAEbj6EUXAMDs7L5y7cuvd529usBKAAC2Bz3BAAAAjIYQDAAAwGgIwQAAAIzGwkJwVR1VVe+sqj8alo+vqquq6rrh+bhF1QYAAMByWuTEWM9Kcm2SY4flC5Jc3d0XVdUFw/LzFlUcwNi945JLD7r94U954pwqAQDYOgvpCa6q05L8yyS/NbX6nCQXD68vTnLuvOsCAABguS1qOPR/TfLcJP8wte6k7t6TJMPziesdWFW7qmqtqtb27t07+0oBAABYGnMPwVX1uCS3dPc1h3N8d+/u7tXuXl1ZWdni6gAAAFhmi7gm+LuSPL6qvj/J3ZIcW1WvSHJzVZ3c3Xuq6uQktyygNgAAAJbY3HuCu/v53X1ad+9Icl6S13f3k5NckWTnsNvOJJfPuzYAAACW23a6T/BFSR5dVdclefSwDAAAAFtmkbdISne/IckbhtefSHLWIusBAABguW2nnmAAAACYqYX2BAMAwCzsvnLtoNt3nb06p0qA7UZPMAAAAKOhJxiAmXjHJZcedPvDn/LEOVUCAPAVeoIBAFiYqtpVVWtVtbZ3795FlwOMwKhC8K1vfH1ufePrF10GAACD7t7d3avdvbqysrLocoARGFUIBgAAYNxcEwywDVXVriS7kuT0009fcDXrO9Q1vwAA29Foe4INjQa2M8MDAQBmY7QhGAAAgPERggEAABgNIRgAAIDRMDHWHbD7yrWDbt919uqcKgEAAOBw6AkGAABgNPQEA4yUWxwBAGOkJxgAAIDREIIBAAAYDSEYAACA0RCCAUbiUDPcAxxJbn3j67/8ALgj5h6Cq+o+VfXnVXVtVb23qp41rD++qq6qquuG5+PmXRsAAADLbRE9wbcleU53f3OSb0/yjKp6YJILklzd3WckuXpYBgAAgC0z9xDc3Xu6+x3D688muTbJqUnOSXLxsNvFSc6dd20AAAAst4VeE1xVO5I8LMnbkpzU3XuSSVBOcuIBjtlVVWtVtbZ37955lQoAAMASWFgIrqp7JvmDJM/u7s9s9Lju3t3dq929urKyMrsCAQAAWDpHL+JNq+rOmQTg3+3u/7+9e4+VtK7vOP7+lNtyseCVbIG6kBDtaiiXLbKBAAXLRUWTVlNMUVovG1Nb2jStgRoppmmkTWMtUVtXRa1awVrLTVskLiwxlctyW5bLFlAoG6hbUqHSbazgr3/M78DsYc+es7szZ37zPO9XMjnP/Gbmeb6/OU8+M7/nNl+vzT9IsrSU8niSpcDmSdS2K+a78uqqs1YsUiWSJEmSpG1Z9EFwkgCfBe4rpXx06KGrgHOBi+vfKxe7tnFzkCxp0lZfvc6skSRJvTaJPcHHA+8A7k5yZ237YwaD368meTfw78DbJlCbJEmSJKnDFn0QXEr5DpA5Hj51MWuRJE3O7X93+XYfP/qdv75IlUiadlvWrtlm+z4nnbLIlUiaBhO9OvSkzBWUkiRJ6o4ta9f4vU/SC/RyECxJkiRJ6icHwZIkSZKk3pjITyS1ZuYwGc8bkdQHM1eqb/0a0Z4zLEmSxqH3e4I9T0SSJEmS+sM9wZLUUfNu5DvilYtTiCRJUkN6vydYkiRJktQfDoIlSZIkSb3hIFiSJEmS1BsOgiVJkiRJveGFsSRJnePPK0mSpLm4J3iWLWvX+LNJkiRJktRR7gmWpJ66cf0jnNjTn0lyT7EkSf3lIHgOW9auYZ+TTpl0GZIkSRqhmSP+PjbPd71VZ61YrJIkLTIPh5YkAYM9wzvzmCRJ0jRxT7AkSZI6zeu9SBrmILhHVl+9bruPe9iPpBnTcL7wfOf1SpIkbYuD4CGtbyV0ECtJkiRJu8ZBsCT12HznAc+1N3hn9xTPLG/a9zLPd/Xoce+l3pWrV3tlbGlhVl+9bqsdJLMvouXOB2l6NTcITnIG8NfAbsBnSikXT6qWmeDb56RTtpoel/n29I779ZIkSZLUdU1dHTrJbsAngDOB5cDbkyyfbFVbm33I9Ja1axZ8GHXrh1tLkiT1ycz3uB39jrZl7RqvSyBNsdb2BB8LPFhK+R5AksuAtwD3TrKobQ1853v+8B7jnR38Lsbe52F9P+d4V/akd/29Gbe+r3stmX149PD97T02Y/gw52m4uNbO6vOX3z4fTj3uw+S7/N5JUktaGwQfBDw6dH8T8LoJ1SJJkqSemWvnxez2G9c/wro5NuJu71ziGbu6gXdXT4Nb8cOHnpt2A4xGaRp2bqSUMukanpPkbcDppZT31PvvAI4tpfzu0HNWAavq3VcBG3dgES8DnhhRuaPWcm3Qdn3WtnNarg12rb5XllJePspiFluHs27U+tLXvvQT+tPXUfRz6rMOdinvurKu2I+22I92zPRh5FnX2iB4JXBRKeX0ev8CgFLKR0Y0/3WllMlvetiGlmuDtuuztp3Tcm3Qfn0t69N715e+9qWf0J++9qWf49SV99B+tMV+tGOcfWjqwljArcDhSQ5NsidwNnDVhGuSJEmSJHVEU+cEl1KeSfI7wLUMfiLp0lLKPRMuS5IkSZLUEU0NggFKKd8Evjmm2a8e03xHoeXaoO36rG3ntFwbtF9fy/r03vWlr33pJ/Snr33p5zh15T20H22xH+0YWx+aOidYkiRJkqRxau2cYEmSJEmSxqY3g+AkZyTZmOTBJOePcTmXJtmcZMNQ20uSXJfkgfr3xUOPXVBr2pjk9KH2Y5LcXR+7JElq+15JLq/tNydZtsC6DklyfZL7ktyT5Pdaqa2+dkmSW5LcVev7cEv11dfvluSOJNe0VFuSh+s870yyrrHaDkjytST313VvZSu1dVUWKetGKY3m5qil8RwepUxBpo9SGv186LLWsq4rOdaVnOpSBnUhX9Lid9VSSudvDC6y9RBwGLAncBewfEzLOhE4Gtgw1PYXwPl1+nzgz+v08lrLXsChtcbd6mO3ACuBAP8MnFnbfxv42zp9NnD5AutaChxdp18E/Ftd/sRrq88PsF+d3gO4GTiulfrqa/4A+Hvgmlb+r/X5DwMvm9XWSm1fAN5Tp/cEDmilti7eWMSsG3HdTebmGPrZdA6PuK/NZ/qI+9vk50NXby1mXVdyrCs51aUM6kK+0OB31YmFxWLe6pt17dD9C4ALxri8ZWwdghuBpXV6KbBxW3UwuCr2yvqc+4fa3w58avg5dXp3Bj8gnZ2o8UrgVxqtbR/gduB1rdQHHAx8GziF50Ooldoe5oXBMvHagJ8Fvj/7uS3U1tUbi5x1I659GY3n5hj63GwOj7ifzWX6iPvX7OdDV2+tZl0Xc6wLOTXNGdSVfKHB76p9ORz6IODRofubattiObCU8jhA/fuKeeo6qE7Pbt/qNaWUZ4CngJfuSDH1EIGjGGwVa6a2erjHncBm4LpSSkv1fQz4APDTobZWaivAt5LclmRVQ7UdBvwn8Ll6GM9nkuzbSG1dNemsG6VOryet5vAoNZ7po9Ty50NXTUvWTfV6MO051ZEM6kq+NPddtS+D4GyjrSx6FS80V13bq3eX+pJkP+Afgd8vpfx3S7WVUp4tpRzJYKvXsUle20J9Sd4EbC6l3Dbfcxe7tur4UsrRwJnA+5Oc2EhtuzM4NOxvSilHAf/D4HCXFmrrqj68H1O/nrScw6PUaqaP0hR8PnTVtL8nza8HXcipac+gjuVLc99V+zII3gQcMnT/YOCxRVz+D5IsBah/N89T16Y6Pbt9q9ck2R3YH/ivhRSRZA8GgfblUsrXW6ptWCnlSeAG4IxG6jseeHOSh4HLgFOSfKmR2iilPFb/bgb+CTi2kdo2AZvqlleArzEYFLdQW1dNOutGqZPrybTk8Cg1mOmj1PTnQ4dNS9ZN5XrQtZya4gzqTL60+F21L4PgW4HDkxyaZE8GJ0xftYjLvwo4t06fy+D8ipn2s+sVzQ4FDgduqYcE/CjJcfWqZ++c9ZqZeb0VWFPqAfDbU+fzWeC+UspHW6qt1vfyJAfU6b2B1wP3t1BfKeWCUsrBpZRlDNadNaWUc1qoLcm+SV40Mw2cBmxoobZSyn8AjyZ5VW06Fbi3hdo6bNJZN0qdW09az+FRajnTR6nlz4eOm5asm7r1oCs51YUM6kq+NPtdtYzpJPTWbsAbGFzh7iHgg2NczleAx4GfMNgq8W4Gx6R/G3ig/n3J0PM/WGvaSL3CWW1fUVeQh4CPU0/uBpYA/wA8yOAKaYctsK4TGBwWsB64s97e0EJt9bVHAHfU+jYAF9b2JuobmvfJPH9hgonXxuC827vq7Z6ZdbuF2uprjwTW1f/rFcCLW6mtqzcWKetGXHOTuTmGfjadwyPu61Rk+oj7fDINfT50/dZa1nUlx7qSU13LoGnOFxr9rjrzQkmSJEmSOq8vh0NLkiRJkuQgWJIkSZLUHw6CJUmSJEm94SBYkiRJktQbDoIlSZIkSb3hIFjNS/L5JG+dcA3Lkmyo0yuSXDLJeiR1j1knqQ/MOrVg90kXIE2bUso6Br99K0mdZdZJ6gOzrp/cE6wFSXJFktuS3JNk1VD700n+LMldSW5KcmBt/3ySS5L8a5LvzWzxS3JykmuGXv/xJL9Zpy9McmuSDUlWJ8k8NZ2X5N4k65NcVtsuSvLFJGuSPJDkvUPP/6M6//VJPlzbliW5L8mna9++lWTv+tgxtV/fBd4/NJ/n+lCXd2mSG2o/zxt63oeS3J/kuiRfSfKHO/8fkLQYzDqzTuoDs86s6zsHwVqod5VSjgFWAOcleWlt3xe4qZTyi8CNwHuHXrMUOAF4E3DxApbx8VLKL5VSXgvsXV+3PecDR5VSjgDeN9R+BPBGYCVwYZKfS3IacDhwLHAkcEySE+vzDwc+UUp5DfAk8Gu1/XPAeaWUlfPU8Wrg9DrvP0myR5IVdT5HAb/K4H2T1D6zbm5mndQdZt3czLoecBCshTovyV3ATcAhDAIG4P+AmS2AtwHLhl5zRSnlp6WUe4EDF7CMX05yc5K7gVOA18zz/PXAl5OcAzwz1H5lKeV/SylPANczCLHT6u0O4HYGATfTh++XUu4c7kOS/YEDSilra/sXt1PHN0opP67L21z7esJQHT8Crp6395JaYNbNzayTusOsm5tZ1wOeE6x5JTkZeD2wspSyJckNwJL68E9KKaVOP8vW69SPh2dT/z7D1htfltRlLAE+CawopTya5KKhZczljcCJwJuBDyWZCdcy63mlLv8jpZRPzerbsll1Pstga2W2MZ+5zH797jzfX0lTwqybl1kndYBZNy+zrgfcE6yF2B/4YQ3KVwPH7cK8HgGWJ9mrbpU7tbbPBOMTSfYDtnvVwCQ/AxxSSrke+ABwALBfffgtSZbUQ3tOBm4FrgXeVedNkoOSvGKu+ZdSngSeSnJCbfqNHeznd4Czah37MQh2SW0z68w6qQ/MOrOu99wTrIX4F+B9SdYDGxkcOrNT6tbArzI45OUBBoexUEp5MsmngbuBhxkE3PbsBnypBm6Av6rzALgF+Abw88CfllIeAx5L8gvAd+tzngbOYbCFby6/BVyaZAuDsN2Rft6a5CrgLgYfEOuAp3ZkHpIWnVln1kl9YNaZdb2X5494kKZfPdzm6VLKXzZQy36llKeT7MPg4hKrSim3T7ouSdPPrJPUB2adxsU9wdL4rE6ynMEhQV8wKCV1lFknqQ/Mug5xT7AkSZIkqTe8MJYkSZIkqTccBEuSJEmSesNBsCRJkiSpNxwES5IkSZJ6w0GwJEmSJKk3HARLkiRJknrj/wH6JixFSJtszwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x648 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(2, 3, figsize = (16, 9))\n",
    "axs[0, 0].hist(df_wholesale[df_wholesale.Channel == 1]['Fresh'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[0, 0].hist(df_wholesale[df_wholesale.Channel == 2]['Fresh'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[0, 0].set_title('Fresh')\n",
    "axs[0, 1].hist(df_wholesale[df_wholesale.Channel == 1]['Milk'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[0, 1].hist(df_wholesale[df_wholesale.Channel == 2]['Milk'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[0, 1].set_title('Milk')\n",
    "axs[0, 2].hist(df_wholesale[df_wholesale.Channel == 1]['Grocery'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[0, 2].hist(df_wholesale[df_wholesale.Channel == 2]['Grocery'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[0, 2].set_title('Grocery')\n",
    "axs[1, 0].hist(df_wholesale[df_wholesale.Channel == 1]['Frozen'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[1, 0].hist(df_wholesale[df_wholesale.Channel == 2]['Frozen'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[1, 0].set_title('Frozen')\n",
    "axs[1, 1].hist(df_wholesale[df_wholesale.Channel == 1]['Detergents_Paper'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[1, 1].hist(df_wholesale[df_wholesale.Channel == 2]['Detergents_Paper'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[1, 1].set_title('Detergents and paper')\n",
    "axs[1, 2].hist(df_wholesale[df_wholesale.Channel == 1]['Delicassen'],\n",
    "                   alpha = 0.4, bins = 30, color = '#003f7e')\n",
    "axs[1, 2].hist(df_wholesale[df_wholesale.Channel == 2]['Delicassen'],\n",
    "                   alpha = 0.4, bins = 30, color = '#821626')\n",
    "axs[1, 2].set_title('Delicatessen')\n",
    "\n",
    "for ax in axs.flat:\n",
    "    ax.set(xlabel='annual spending', ylabel='count')\n",
    "\n",
    "# Hide x labels and tick labels for top plots and y ticks for right plots.\n",
    "for ax in axs.flat:\n",
    "    ax.label_outer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample means differ in some departments more and in some less. Nevertheless, there is a difference across all products. In addition, the variances are quite substantial and the distributions are not normally distributed. Due to the different sample sizes and the different variances we choose Welch's t-test to check the significance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hypothesis testing\n",
    "The hypotheses are:\n",
    "* **Null hypothesis $H_0$: There is no difference in the resulting sales with regard to a different market entry in the mentioned products.**\n",
    "* **Alternative hypothesis $H_1$: The difference in the sample means of the sales turnover results from different market entrances for the two groups and is not random.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fresh:\n",
      "Ttest_indResult(statistic=4.154185080193192, pvalue=3.997523791114204e-05)\n",
      "- - -\n",
      "Milk:\n",
      "Ttest_indResult(statistic=-8.541550659835881, pvalue=7.513432722470988e-15)\n",
      "- - -\n",
      "Grocery:\n",
      "Ttest_indResult(statistic=-11.775037432327998, pvalue=3.623878362357236e-23)\n",
      "- - -\n",
      "Frozen:\n",
      "Ttest_indResult(statistic=5.811487334401299, pvalue=1.2659274764585105e-08)\n",
      "- - -\n",
      "Detergents and paper:\n",
      "Ttest_indResult(statistic=-12.183128559103928, pvalue=5.8093151114456274e-24)\n",
      "- - -\n",
      "Delicatessen:\n",
      "Ttest_indResult(statistic=-1.3763225647039357, pvalue=0.16947545890793167)\n"
     ]
    }
   ],
   "source": [
    "print('Fresh:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Fresh'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Fresh'],\n",
    "                      equal_var = False))\n",
    "print('- - -')\n",
    "print('Milk:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Milk'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Milk'],\n",
    "                      equal_var = False))\n",
    "print('- - -')\n",
    "print('Grocery:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Grocery'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Grocery'],\n",
    "                      equal_var = False))\n",
    "print('- - -')\n",
    "print('Frozen:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Frozen'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Frozen'],\n",
    "                      equal_var = False))\n",
    "print('- - -')\n",
    "print('Detergents and paper:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Detergents_Paper'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Detergents_Paper'],\n",
    "                      equal_var = False))\n",
    "print('- - -')\n",
    "print('Delicatessen:')\n",
    "print(stats.ttest_ind(df_wholesale[df_wholesale.Channel == 1]['Delicassen'],\n",
    "                      df_wholesale[df_wholesale.Channel == 2]['Delicassen'],\n",
    "                      equal_var = False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the exception of the delicatessen sector, the p-value for all other products is below the significance level of $\\alpha = 0.05$. We therefore reject the null hypothesis for these segments and note that the argument that the difference in sales is due to the change in market access has been supported. For delicatessen, $p \\approx 0.169$ is above our defined $\\alpha$, so we cannot reject the null hypothesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion\n",
    "Our initial question (*Does the different entrance to the market cause a different buying behavior and if so, which positioning of the entrance is sales-promoting?*) has to be answered in different ways for the different products:\n",
    "* For Fresh and Frozen, the market entry of the <span style=\"color:#003f7e\">blue channel-1 group</span> seems to be sales-promoting, because there is statistical evidence to reject the null hypothesis.\n",
    "* For Milk, Grocery as well as Detergents and paper the entrance of the <span style=\"color:#821626\">red channel-2 group</span> is likely to be boosting sales, due to a justified rejection of the null hypothesis.\n",
    "* For the Delicatessen segment there are no indications that the specific entrance is of relevance.\n",
    "\n",
    "The store managers must now be challenged with these results. Depending on the strategic orientation (sales targets in the segments, etc.) a profound decision regarding market access can be made."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References<a class=\"anchor\" id=\"references\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1]<a class=\"anchor\" id=\"schaefer2011\"></a> Schäfer, T. (2011). *Statistik II*. VS Verlag für Sozialwissenschaften."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[2]<a class=\"anchor\" id=\"rasch2004\"></a> Rasch, D., & Guiard, V. (2004). The robustness of parametric statistical methods. *Psychology Science, 46*, 175-208."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[3]<a class=\"anchor\" id=\"semendjajew2003\"></a> Semendjajew, K.A., Grosche, G., Bronštejn I.N., & Zeidler, E. (Eds.). (2003). *Teubner-Taschenbuch der Mathematik* (Vol. 2). Springer DE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[4]<a class=\"anchor\" id=\"kanji2006\"></a> Kanji, G. K. (2006). 100 statistical tests. Sage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[5]<a class=\"anchor\" id=\"benesch2012\"></a> Benesch, T. (2012). *Schlüsselkonzepte zur Statistik: die wichtigsten Methoden, Verteilungen, Tests anschaulich erklärt*. Springer-Verlag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[6]<a class=\"anchor\" id=\"welch1947\"></a> Welch, B. L. (1947). The generalization ofstudent's' problem when several different population variances are involved. Biometrika, 34(1/2), 28-35."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[7]<a class=\"anchor\" id=\"johnson2005\"></a> Johnson, N. L., Kemp, A. W., & Kotz, S. (2005). *Univariate discrete distributions* (Vol. 444). John Wiley & Sons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[8]<a class=\"anchor\" id=\"oliphant2006\"></a> Oliphant, T. E. (2006). *A guide to NumPy* (Vol. 1, p. 85). USA: Trelgol Publishing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[9]<a class=\"anchor\" id=\"walt2011\"></a> Walt, S. V. D., Colbert, S. C., & Varoquaux, G. (2011). The NumPy array: a structure for efficient numerical computation. *Computing in Science & Engineering*, 13(2), 22-30."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[10]<a class=\"anchor\" id=\"virtanen2020\"></a> Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., ... & van der Walt, S. J. (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. *Nature methods*, 17(3), 261-272."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}