02.1.LinearRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression线性回归原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 问题引入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.1 一元线性回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFBCAYAAAA/sXjgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XeYluWB7/HvPTC0AaQjCoiigIhSHEosEXvX3WSziTkxrnHjJtm4JpoYC3uyu6DYe4kYu7EkajZl09y0XRNnhqFXQVEQpBfpZWbu88cz5xzjWgacee+3fD/X5cUMjNf8vEbly/M+cz8hxogkSZJyqyz1AEmSpFJkhEmSJCVghEmSJCVghEmSJCVghEmSJCVghEmSJCXQYhEWQngkhLAmhDD3XT/XLYTwUghhceOPXVvq80uSJOWzlrwS9hhwxnt+7mrgtzHGw4DfNr4vSZJUckJLHtYaQhgA/DzGOKzx/VeB8THGlSGEPsAfYoyDW2yAJElSnsr1PWG9Y4wrG99eBfTO8eeXJEnKC61TfeIYYwwhfOBluBDCpcClABUVFUcPGTIkZ9skSVIzqt8F21bBjnVAgPbdoWJ/aNU29bIWMW3atHUxxp4f9XG5jrDVIYQ+73o5cs0HfWCMcQowBaCysjLW1tbmaqMkSWoO6xdCzQ2w4Gkoaw1Hfh1Gfxs690+9rEWFEJY25eNyHWE/BS4Cbmz88Sc5/vySJKmlrZ0DVZNg0Y+gdXsYdTlUfgs69km9LK+0WISFEJ4BxgM9QgjLge+SxdcPQwiXAEuBv22pzy9JknJs9TR4ZSK8/hNo0wnGXA1HfxM6fOQrcyWpxSIsxnjBB/zSyS31OSVJUgJvvwJVE+GNX0LbLvCJ78LIf4L23VIvy2vJbsyXJEkFLEZY/scsvpb9Dtr3gONugBH/CG07p15XEIwwSZLUdDHC0t9k93yteBk69IYTboXhX4HyitTrCooRJkmSPlqMsOTnWXytqoGOfeGke2DYJVDePvW6gmSESZKkDxYbYPGLWXytnQX7HQynToEjLoJWbVKvK2hGmCRJ+p8a6uDV56DqetiwALoOgjMegyGfh1blqdcVBSNMkiT9f/V7YMFTUH0DbHoNuh8BZz8Dgz4DZa1SrysqRpgkSYK6XTDvUai5ETYvhV4j4bwX4dDzIeT6UdOlwQiTJKmU7dkOc74PU2+GrSugz1g4+T44+CwIIfW6omaESZJUinZvhVkPQO2tsH0N9P1kds9X/5ONrxwxwiRJKiW73oEZ98K0O2DnejjoVBg3IYsw5ZQRJklSKdixHqbfBTPuzkLskHNg7HVwwLjUy0qWESZJUjHbvgZqb4OZ98OerXDYp2DsBOg9MvWykmeESZJUjLasyO73mv0g1O+CwZ+FsddCj2Gpl6mRESZJUjHZvBRqboK5D0NDPQz9Aoy5FroNSr1M72GESZJUDDa+BjWTYf4TQIBhF8Po70CXQ1Iv0wcwwiRJKmTrF2Sn2y98GsrK4aivwOiroHO/1Mv0EYwwSZIK0drZ2UO1Fz0PrdvDqG9C5ZXQsU/qZWoiI0ySpEKyqjaLr9d/Am06wdhrYNQ3oEPP1Mu0l4wwSZIKwYo/Q9VEePNX0LYLfOJfYNQ/QbuuqZdpHxlhkiTlqxjhrT9k8fXW76F9DzhuMoz4GrTtnHqdPiYjTJKkPLFs/XYueXwqS9Zu5W97zOO7vf6ddmtegYr94YTbYPg/QHlF6plqJkaYJEl54pLHahiw+Xfcst+zjGAxa9b0ot1J98KRl0DrdqnnqZkZYZIkpRYbYNEL3F33HQ7v/AZL6/fnO1su4ye7TmbhyPNTr1MLMcIkSUqloQ5efQ6qrocNC6ho3Y8rt3yTf985nhhaMbBnx9QL1YKMMEmScq1+D8x/MjvhftNr2fMcz34Wup/F7Cenw65tDOxZwcMXjU69VC3ICJMkKVfqdsG8R6HmxuwZj71GwXk/hkPPg1BGf+ClK05IvVI5YoRJktTS9myHOQ/B1Jth69vQZxycfD8cfCaEkHqdEjHCJElqKbu3wMwHYNptsH0N9D0BzngC+p9kfMkIkySp2e3cBDPvhWl3wM4NcNBpMG4C9D0+9TLlESNMkqTmsmM9TL8Tpt8NuzfDIedk8dVnbOplykNGmCRJH9e21VB7G8y6H/Zsg8M+DWOvg94jUy9THjPCJEnaV1tWQO0tMHsK1O+CwZ/N4qvHEamXqQAYYZIk7a3NS7NjJuY+Ag31MPRCGHMNdBuUepkKiBEmSVJTbXwNqm+ABU8CAYZ9CcZ8B/Y7OPUyFSAjTJKkj7J+AVRfDwufgVZtYPhXYfRV0Klv6mUqYEaYJEkfZM0sqJ4Ei16A8g5w9BVQeSVU7J96mYqAESZJ0nutmgpVk+D1n0KbTjD2Ghj1TejQI/UyFREjTJKk/2vFn6BqIrz5a2jXFY75Vxh5Wfa21MyMMElSaYsR3vp9Fl9v/QHa94Tjb8zu+2rbOfU6FTEjTJJUmmLMrnhVTYS3/wwVfWD87XDUpVBekXqdSoARJkkqLbEBXv9Zds/X6lro1A9Ovi87bqJ1u9TrVEKMMElSaWioh8UvZEdNrJ0N+x0Cp30/O2i1VZvU61SCjDBJUnFrqIOFz2bxtWEhdB0MZz4BQy6AMn8bVDr+2ydJKk71u2H+k1AzGTa9Dj2OhHOeyx6uXdYq9TrJCJMkFZm6nTD30ezZjluWQe+j4bwfw6HnQShLvU76f4wwSVJx2LMdZk+B2ltg69vQ5xNw6vdgwBkQQup10v+QJMJCCN8E/h6IwBzg4hjjzhRbJEkFbvcWmHk/1N4GO9ZCv/Fw5pPQ70TjS3kt5xEWQjgQ+CdgaIxxRwjhh8DngMdyvUWSVMB2boIZ98D0O2HnBhhwOoydAH2PS71MapJUL0e2BtqHEPYAHYC3E+2QJBWa7euy8JpxD+zeDIecC+MmQJ8xqZdJeyXnERZjXBFCuBVYBuwAfhNj/M17Py6EcClwKUD//v1zO1KSlH+2rcpecpz1QHb/16BPw9jroNeI1MukfZLi5ciuwPnAwcAm4EchhC/EGJ9698fFGKcAUwAqKytjrndKkvLElhUw9WaYMyU7dmLw52DcddB9aOpl0seS4uXIU4A3YoxrAUIILwLHAE996N8lSSot77yZHTMx79HsUUOHXwhjr4Guh6VeJjWLFBG2DBgXQuhA9nLkyUBtgh2SpHy0cTFUT4YFT2bneh1xMYy5GvYbkHqZ1KxS3BNWHUJ4HpgO1AEzaHzZUZJUwtbPh6rr4dVns2c5Dv8ajP42dOqbepnUIpJ8d2SM8bvAd1N8bklSnlkzE6omweIXobwDHH0lVF4JFb1TL5NalCfmS5LSWFmTxdeSn0GbzjD2Whj1DejQI/UyKSeMMElSbi1/GaomwtLfQLtucMy/wcjLoF2X1MuknDLCJEktL0ZY9rssvpb/Edr3hONvghFfhTadUq+TkjDCJEktJ0Z481fwykRY+QpU9IHxd8BRl2b3f0klzAiTJDW/2ACv/RSqJ8HqadCpP5x8Pwy7GFq3S71OygtGmCSp+TTUw+IXshvu182BLgPhtO/D0AuzYyck/T9GmCTp42uog4XPQPUNsGEhdBsCZz4JQz4HZf5WI70f/8uQJO27+t0w7wmomQzvLIGeR8E5P4TDPgVlrVKvk/KaESZJ2nt1O2HuI1BzE2xZBr0rYfztMPDc7FFDkj6SESZJaro922H2gzD1Fti2Eg44Bk59EAacDiGkXicVFCNMkvTRdm+BGffBtNthx1rodyKc9VT2o/El7RMjTJL0wXZughl3w/Q7YefG7IrXuH+GA49NvUwqeEaYJOl/2r4Opt8BM+6F3Zth4HkwbgLsPzr1MqloGGGSpP9v2yqovQ1mPZDd/zXo0zB2AvQannqZVHSMMEkSbFkOU2+GOQ9lx04MuQDGXgvdh6ZeJhUtI0ySStk7b0DNjTD3USDC0C/CmGug66Gpl0lFzwiTpFK0YVF2wOr8J7NDVY+8BEZ/B/YbkHqZVDKMMEkqJevmQfX18Opz2bMcR34dKr8NnQ5MvUwqOUaYJJWC1TOgehIsfhHKK6DyW3D0FVDRO/UyqWQZYZJUzFbWQNVEWPJzaNM5O2Zi1DegfffUy6SSZ4RJUjFa/t9ZfC19Cdp1g2MnwoivQ7suqZdJamSESVKxiBGW/S6Lr+V/hA694PibYMRXoU2n1OskvYcRJkmFLkZ445dZfK2sgo4HwIl3wpFfhvIOqddJ+gBGmCQVqtgAr/0EqibBmunQ+SA45QE44mJo3Tb1OkkfwQiTpELTUA+Lns++23HdXOgyEE57GIZeCK3KU6+T1ERGmCQVioY6WPA0VN8AG1+FbofDWU/B4M9Cmf87lwqN/9VKUr6r3w3zHs8eL/TOEug5HM79ERz2KQhlqddJ2kdGmCTlq7qdMOdhmHoTbHkLelfC+Dtg4LkQQup1kj4mI0yS8s2ebTDrQai9BbatggOOhdMegoNOM76kImKESVK+2LUZZt4P026DHeug/0lw1tPQb7zxJRUhI0ySUtu5EabfDTPuyt4ecAaM+2c48JjUyyS1ICNMklLZvham3QEz74XdW2Dg+dmzHfevTL1MUg4YYZKUa1tXQu1tMOsBqNsBg/4mi6+eR6VeJimHjDBJypXNb8HUm2HOQ9CwB4Z8HsZeC90PT71MUgJGmCS1tHfegOrJMO8xIMLQi2DM1dD10NTLJCVkhElSS9mwCGpugPlPQVkrOPLvYcx3smc8Sip5RpgkNbd1c6Hqelj0Q2jVFkZeBqO/DR0PSL1MUh4xwiSpuayekT1Ue/GLUN4RKr8NlVdAh16pl0nKQ0aYJH1cK6uhaiIs+Q9ou192xteoy6F999TLJOUxI0yS9tXy/4JXJsKy/4R23eDYSTDy61mISdJHMMIkaW/ECMt+m135Wv5f2UuNn7wZhn8V2nRMvU5SATHCJKkpYoQ3fpHF18pq6HggnHgXHPllKG+fep2kAmSESdKHiQ3w2r9D1SRYMyM7XuKU78ERfwet26ZeJ6mAGWGS9H4a6mHRj6D6+uzIiS6HwumPwOFfgFblqddJKgJGmCS9W/0eWPg0VN8AGxdB96Fw1g9g8N9Cmf/LlNR8kvwfJYTQBfg+MAyIwJdijK+k2CJJANTvhnmPQ83k7DFDPYfDuT+Cwz4FoSz1OklFKNUf6+4CfhVj/JsQQhugQ6Idkkrdnh0w92GouQm2Lof9R2c33B9yDoSQep2kIpbzCAsh7Ad8Evg7gBjjbmB3rndIKnF7tsGs70HtrbBtFRx4HJz+MBx0qvElKSdSXAk7GFgLPBpCGA5MAy6PMW5LsEVSqdm1GWbeB9Nuhx3roP9JcPYz0PcE40tSTqW40aE1MAp4IMY4EtgGXP3eDwohXBpCqA0h1K5duzbXGyUVm50b4c//Ag8dBC9fm73s+Lk/wWd+C/3GG2CSci7FlbDlwPIYY3Xj+8/zPhEWY5wCTAGorKyMuZsnqahsX5td9Zp5H+zeAof+FYy9DvavTL1MUonLeYTFGFeFEN4KIQyOMb4KnAzMz/UOSUVu68rsfq9Z34O6HTDoMzDuOuh5VOplkgSk++7Iy4AfNH5n5BLg4kQ7JBWbzW/B1JtgzvehoQ4O/zyMuRa6D0m9TJL+QpIIizHOBHwtQFLz2bQEam6EeY8BMXus0JirocvAxMMk6f15/LOkwrbh1ex0+wU/yE60P/LLMOY70Ll/6mWS9KGMMEmFad3c7KHar/4QWreDUf8Eld+CjgekXiZJTWKESSosq6dn8fXaj6G8I4y+CiqvgA69Ui+TpL1ihEkqDG9XQdVEeOMX0HY/GPe/YdTl0L5b6mWStE+MMEn57a0/Zle+lv0ntOsOx10PI/4xCzFJKmBGmKT8EyMs/c/syteK/4YOveGTt8Dwr0CbjqnXSVKzMMIk5Y8YYcl/QPUkWFkNHQ+EE++GI/8eytunXidJzcoIk5RebIDFP85edlw7EzoPgFMfhKEXQeu2qddJUoswwiSl01CfHTFRfT2snwddD4PTH4XD/xe0Kk+9TpJalBEmKffq92SHq9bcABsXQ/ehcNbTMPhvoaxV6nWSlBNGmKTcqdsF8x+H6smw+U3oOQLOfR4O+2sIZanXSVJOGWGSWt6eHdkDtafeDFuXw/5j4KR74JCzIYTU6yQpCSNMUsvZvRVmfQ9qb4Xtq+HA4+H0R+CgU4wvSSXPCJPU/HZthpn3Qu3tsHM99D8Zxj0H/U5IvUyS8oYRJqn57NgA0++CGXfDrk1w8FkwbgIc8InUyyQp7xhhkj6+7Wtg2h0w417YsxUO/WsYdx30Pjr1MknKW0aYpH23dSXU3pLd91W3MztiYux10PPI1MskKe8ZYZL23uZlUHMTzH0YGuqyw1XHXgvdBqdeJkkFwwiT1HSblkDNZJj3ePb+ERfBmKuhy8C0uySpABlhkj7a+oXZ6fYLnoay1nDUpTD6KujcP/UySSpYHxlhIYTLgKdijBtzsEdSHli2fjuXPD6V1hvmcVXX5xnPHwit28Ooy6HyW9CxT+qJklTwmnIlrDcwNYQwHXgE+HWMMbbsLEkp3fjYU3xr16Oc3rWKrQ3tebbsAi748p3QoWfqaZJUND7yYW0xxgnAYcDDwN8Bi0MIN4QQvAlEKjZvvwIvns39/APjyudw57YLOHbDI0xY978MMElqZk26JyzGGEMIq4BVQB3QFXg+hPBSjPGqlhwoqYXFCMv/CFWTYNlvoX0PHubL3L3xFN5pqKAswMCeFalXSlLRaco9YZcDXwTWAd8Hvh1j3BNCKAMWA0aYVIhihKUvQdVEWPEydOgNJ9wKw7/CqZsDzz4+la1rt3FIzwoevmh06rWSVHSaciWsG/CpGOPSd/9kjLEhhHBOy8yS1GJihCU/z658raqBjn3hpHtg2CVQ3h6A/t3hpSt8zqMktaSPjLAY43c/5NcWNO8cSS0mNsDiH2fxtXYmdB4Apz4IQy+C1m1Tr5OkkuM5YVKxa6iHV5+D6uth/XzoOgjOeAyGfB5aladeJ0klywiTilX9Hljwg+yQ1Y2LofsRcPYzMOgzUNYq9TpJKnlGmFRs6nbBvMeg5kbY/Cb0GgnnvQCH/hWEjzyVRpKUI0aYVCz27IA5D8HUm2HrCugzFk6+Fw4+C0JIvU6S9B5GmFTodm+FWd+D2lth+2o48Hg4/VE46BTjS5LymBEmFapd78CMe2HaHbBzPfQ/BT7xQ+j7ydTLJElNYIRJhWbHBph+J8y4OwuxQ86GsRPggHGpl0mS9oIRJhWK7Wug9naYeR/s2QqH/jWMmwC9R6VeJknaB0aYlO+2vg1Tb4HZD0LdThj8WRh3HfQYlnqZJOljMMKkfLV5KdTcDHMfhoY6GPoFGHMNdBucepkkqRkYYVK+2fQ6VE+G+Y8DAY74OxhzNXQ5JPUySVIzMsKkfLF+YfZooYVPQ1k5HPUVGH0VdO6XepkkqQUYYVJqa2dD1fWw6EfQuj2M+iZUXgkd+6ReJklqQUaYlMrqafDKRHj9J9CmU/aS49HfhA49Uy+TJOWAESbl2oo/Q/UkeOOX0LYLfOJfYORl0L5b6mWSpBwywqRciBGW/xGqJsKy30H7HnDcZBjxNWjbOfU6SVICRpjUkmKEpb+Bqkmw4mWo2B9OuA2G/wOUV6ReJ0lKyAiTWkKMsOTn2ZWvVVOhUz846V4Y9iUob596nSQpDySLsBBCK6AWWBFjPCfVDqlZxQZY/GJ25WvtLNjvYDh1ChxxEbRqk3qdJCmPpLwSdjmwAPCGGBW+hjp49bnsqIkNC6DrIDjjcRhyAbQqT71OkpSHkkRYCKEvcDZwPXBFig1Ss6jfAwueguobYNNr0P0IOPsZGPQZKGuVep0kKY+luhJ2J3AV0CnR55c+nrpdMO9RqLkxe8Zjr5Fw3otw6PkQylKvkyQVgJxHWAjhHGBNjHFaCGH8h3zcpcClAP3798/ROukj7NkOcx6CqTfD1rehzzg4+X44+EwIIfU6SVIBSXEl7FjgvBDCWUA7oHMI4akY4xfe/UExxinAFIDKysqY+5nSu+zeCrMegNpbYfsa6PvJ7J6v/icbX5KkfZLzCIsxXgNcA9B4Jexb7w0wKW/segdm3APT7oCdG+CgU2HchCzCJEn6GDwnTHo/O9bD9Ltgxt1ZiB1yThZffcamXiZJKhJJIyzG+AfgDyk3SH9h+xqovQ1m3g97tsJhn4KxE6D3yNTLJElFxithEsCWFVB7C8yeAvW7YPBnYex10OOI1MskSUXKCFNp27wUam6CuQ9DQz0MvRDGXAPdBqVeJkkqckaYStPG16BmMsx/Aggw7GIYc3X2mCFJknLACFNpWb8gO91+4dPZsxyHfxVGXwWd+qZeJkkqMUaYSsPa2dlDtRc9D63bw9FXQOWVULF/6mWSpBJlhKm4rarN4uv1n0CbTjD2Ghj1TejQI/UySVKJM8JUnFb8KYuvN38F7brCMf8KIy/L3pYkKQ8YYSoeMcJbf4CqifDW76F9DzhuMoz4GrTtnHqdJEl/wQhT4YsR3vx1duXr7T9BRR8YfzscdSmUV6ReJ0nS+zLCVLhihNd/ll35Wl0LnfrByffBsC9B63ap10mS9KGMMBWe2ACLXoDqSdl3Pe53CJz6EBzxxezYCUmSCoARpsLRUAcLn83O+dqwALoOhjOfgCEXQJn/KkuSCou/cyn/1e+G+U9lJ9xveg16DIOzn4VBfwNlrVKvkyRpnxhhyl91u2DuI1BzI2xZBr1GwXk/hkPPg1CWep0kSR+LEab8s2c7zHkIpt4MW9+GPuPglAfg4DMhhNTrJElqFkaY8sfuLTDzAZh2G2xfA31PgDOegP4nGV+SpKJjhCm9nZtgxj0w/U7YuQEOOg3GTYC+x6deJklSizHClM6O9Vl4Tb8bdm+GQ86FcddBn7Gpl0mS1OKMMOXettVQexvMuh/2bIPDPp1d+eo1IvUySZJyxghT7mxZAbW3wOwpUL8LBn8Oxl4LPY5IvUySpJwzwtTyNi/NjpmY+0h22v3hF8KYq6HboNTLJElKxghTy9n4Wna6/YIngZA903HM1bDfgNTLJElKzghT81s/P4uvhc9kz3Ic/jUY/W3o1Df1MkmS8oYRpuazZlb2UO1FL0B5Bzj6Cqi8Eir2T71MkqS8Y4Tp41s1Faomwes/hTads5vtR30DOvRIvUySpLxlhGnfrfgTVE2EN38N7brCMf8GIy+Ddl1SL5MkKe8ZYdo7McJbv8/i660/QPuecPyNMOJr0KZT6nWSJBUMI0xNEyO8+avsZce3/wwVfWD8HXDUl6G8IvU6SZIKjhGmDxcb4PWfZfG1uhY69YOT78uOm2jdLvU6SZIKlhGm99dQD4tfgOrrYe1s2O8QOO37MPTC7NgJSZL0sRhh+ksNddn5XtU3wIaF0HUwnPkEDLkAyvzXRZKk5uLvqsrU74b5T0LNZNj0OvQ4Es55Lnu4dlmr1OskSSo6Rlipq9uZPdOx5ibYsgx6Hw3n/zsMPBdCWep1kiQVLSOsVO3ZDrOnQO0tsPVt6PMJOPV7MOAMCCH1OkmSip4RVmp2b4GZ90PtbbBjLfQbD2c+Cf1ONL4kScohI6xU7NwEM+6B6XfCzg0w4HQYOwH6Hpd6mSRJJckIK3bb12XhNeMe2L0ZBp4HY6+DPmNSL5MkqaQZYcVq26rsJcdZD2T3fw36dBZfvUakXiZJkjDCis+W5TD1FpgzJTt2YsgFMPZa6D409TJJkvQuRlixeOdNqLkR5j2aPWpo6BdhzNXQ9bDUyyRJ0vswwgrdxsVQPRkWPJmd6zXsSzD6O7DfgNTLJEnShzDCCtX6+VB1Pbz6bPYsxxH/CJXfgk59Uy+TJElNYIQVmjUzoWoSLH4RyjvA0VdC5ZVQ0Tv1MkmStBeMsEKxsiaLryU/gzadYdx1MPJy6NAj9TJJkrQPjLB8t/xlqJoIS38D7brBMf8GIy+Ddl1SL5MkSR+DEZaPYoRlv8via/kfoUMvOP4mGPFVaNMp9TpJktQMch5hIYR+wBNAbyACU2KMd+V6R16KEd74Zfay48pXoKIPjL8Djro0u/9LkiQVjRRXwuqAK2OM00MInYBpIYSXYozzE2zJD7EBXvspVE+C1dOgU384+X4YdjG0bpd6nSRJagE5j7AY40pgZePbW0IIC4ADgdKLsIZ6WPQ8VF8P6+ZAl4Fw2sMw9AvZsROSJKloJb0nLIQwABgJVL/Pr10KXArQv3//nO5qcQ11sPAZqL4BNiyEbkPgzCdhyOegzNv0JEkqBcl+xw8hdAReAL4RY9z83l+PMU4BpgBUVlbGHM9rGfW7Yd4TUDMZ3lkCPY+Cc34Ih30KylqlXidJknIoSYSFEMrJAuwHMcYXU2zIqbqdMPcRqLkJtiyD3pXZDfcDz8keNSRJkkpOiu+ODMDDwIIY4+25/vw5tWcbzJ4CU2+BbSvhgGPg1AdhwOkQQup1kiQpoRRXwo4FLgTmhBBmNv7ctTHGXyTY0jJ2b4EZ98G022HHWuh3Ipz1A+g33viSJElAmu+OfBkozhLZuRFm3APT78zeHnAGjJsABx6bepkkScozfitec9i+DqbfATPuhd2bYeD52bMd9x+depkkScpTRtjHsW0VTL0VZj0AdTtg0N/A2Oug1/DUyyRJUp4zwvbFluUw9WaY81B27MSQz8PYa6D70NTLJElSgTDC9sY7b0DNjTD3USDC0C/CmGug66Gpl0mSpAJjhDXFhkXZAavzn8wOVT3y72H0VbDfgNTLJElSgTLCPsy6edlzHV99LnuW48ivQ+W3odOBqZdJkqQCZ4S9n9UzoHoSLH4Ryiug8ltw9BVQ0Tv1MkmSVCSMsHdbWQ1Vk2DJz6HtfjDun2HU5dC+e+plkiSpyBhhAMv/G6omwtKXoF03OHYijPg6tOuSepkkSSpSpRthMcKy32bxtfy/oEMv+OTNMPyr0KZj6nWSJKnIlV6ExQhv/DKLr5VV0PEAOPFOOPLLUN4h9TpJklQiSifCYgO89pPsnq8106HzQXDKA3DExdC6bep1kiSpxBR/hDXUw6Lns+92XDcXuhwKpz8Ch38BWpWnXieO22/KAAAF9klEQVRJkkpU8UZYQx0seBqqb4CNr0K3w+Gsp2DwZ6GseP+xJUlSYSi+GqnfDfMez064f+cN6Dkczv0RHPYpCGWp10mSJAHFFGF1O2HOwzD1JtjyFuw/Gk68Cw45B0JIvU6SJOkvFH6E7dkGsx6E2ltg2yo44Fg47SE46DTjS5Ik5a3CjbBdm2HmfTDtdtixDvqfBGc/A31PML4kSVLeK7wI27kRpt8NM+7K3j74TBg7AQ48JvUySZKkJiucCNu+FqbdATPvhd1bYOD5MG4C7F+ZepkkSdJeK4wI27IcHhoAdTtg0Gdg3HXQ86jUqyRJkvZZYUTY9tUw6EIYcw10Pzz1GkmSpI+tMCKsxzA484nUKyRJkppNYZxe2spnO0qSpOJSGBEmSZJUZIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBIwwSZKkBJJEWAjhjBDCqyGE10IIV6fYIEmSlFLOIyyE0Aq4DzgTGApcEEIYmusdkiRJKaW4EjYGeC3GuCTGuBt4Fjg/wQ5JkqRkUkTYgcBb73p/eePPSZIklYzWqQd8kBDCpcClje/uCiHMTblH+6wHsC71CO0zv36Fza9f4fJrV9gGN+WDUkTYCqDfu97v2/hzfyHGOAWYAhBCqI0xVuZmnpqTX7vC5tevsPn1K1x+7QpbCKG2KR+X4uXIqcBhIYSDQwhtgM8BP02wQ5IkKZmcXwmLMdaFEL4O/BpoBTwSY5yX6x2SJEkpJbknLMb4C+AXe/G3TGmpLWpxfu0Km1+/wubXr3D5tStsTfr6hRhjSw+RJEnSe/jYIkmSpATyOsJ8vFHhCiE8EkJY49EihSmE0C+E8PsQwvwQwrwQwuWpN6lpQgjtQgg1IYRZjV+7f029SXsvhNAqhDAjhPDz1FvUdCGEN0MIc0IIM5vyHZJ5+3Jk4+ONFgGnkh3oOhW4IMY4P+kwNUkI4ZPAVuCJGOOw1Hu0d0IIfYA+McbpIYROwDTgr/zvL/+FEAJQEWPcGkIoB14GLo8xViWepr0QQrgCqAQ6xxjPSb1HTRNCeBOojDE26Yy3fL4S5uONCliM8b+ADal3aN/EGFfGGKc3vr0FWIBPtigIMbO18d3yxr/y80/bel8hhL7A2cD3U29Ry8rnCPPxRlIeCCEMAEYC1WmXqKkaX8qaCawBXoox+rUrLHcCVwENqYdor0XgNyGEaY1P/vlQ+RxhkhILIXQEXgC+EWPcnHqPmibGWB9jHEH2RJIxIQRvCSgQIYRzgDUxxmmpt2ifHBdjHAWcCfxj4605HyifI6xJjzeS1DIa7yd6AfhBjPHF1Hu092KMm4DfA2ek3qImOxY4r/HeomeBk0IIT6WdpKaKMa5o/HEN8GOyW6s+UD5HmI83khJpvLn7YWBBjPH21HvUdCGEniGELo1vtyf75qaFaVepqWKM18QY+8YYB5D9vve7GOMXEs9SE4QQKhq/kYkQQgVwGvChJwTkbYTFGOuA//t4owXAD328UeEIITwDvAIMDiEsDyFcknqT9sqxwIVkfwqf2fjXWalHqUn6AL8PIcwm+8PsSzFGjzmQWl5v4OUQwiygBviPGOOvPuxvyNsjKiRJkopZ3l4JkyRJKmZGmCRJUgJGmCRJUgJGmCRJUgJGmCRJUgJGmCRJUgJGmCRJUgJGmKSSEkIYHUKYHUJo13jC9TyfrSgpBQ9rlVRyQgiTgHZAe2B5jHFy4kmSSpARJqnkND6PdiqwEzgmxlifeJKkEuTLkZJKUXegI9CJ7IqYJOWcV8IklZwQwk+BZ4GDgT4xxq8nniSpBLVOPUCScimE8EVgT4zx6RBCK+DPIYSTYoy/S71NUmnxSpgkSVIC3hMmSZKUgBEmSZKUgBEmSZKUgBEmSZKUgBEmSZKUgBEmSZKUgBEmSZKUgBEmSZKUwP8BPBsL1lERkS8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "x = [2, 4]\n",
    "y = [5, 9]\n",
    "\n",
    "plt.figure(figsize=[10, 5])\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim((0, 5))\n",
    "plt.ylim((0, 10))\n",
    "plt.scatter(x, y, s=16)\n",
    "plt.plot([0, 5], [1, 11], color=\"darkorange\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已知点A(2, 7)和点B(4, 9)在直线L上，求直线L的线性函数：$f(x) = wx + b$ 的参数 $w$ 和 $b$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将点A、B坐标代入直线函数，解方程组\n",
    "$ \\left\\{\n",
    "\\begin{aligned}\n",
    "2w+b=5 \\\\\n",
    "4w+b=9\n",
    "\\end{aligned}\n",
    "\\right.\n",
    "$，得$w=2, b=1$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**问题升级**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本数据存在噪声 $\\epsilon$ ：  \n",
    "$f(x) = wx+b$  \n",
    "$y - f(x) = \\epsilon$，如何求解？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFACAYAAAASxGABAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvX1wVPed5vv8zulGLQzoBdkxINkgtUjuOIszkYgtBNhYtufW1lZtTQz2jWuym4mDcfxCuJ7ZydTeWzWztVV3xjfJ4kAw4cWZ2Z1Zj7HI3NqtramaYCHbCIRNizjY3myklkC0EDZ6bfGiFupzfveP0+fod06fbrWEGkno+VS5Av1yzu905eXJ8/1+n6+QUoIQQgghhMwe2mwfgBBCCCFkoUNBRgghhBAyy1CQEUIIIYTMMhRkhBBCCCGzDAUZIYQQQsgsQ0FGCCGEEDLLUJARQgghhMwyFGSEEEIIIbMMBRkhhBBCyCwTmO0D5EJZWZlcvXr1bB+DEEIIIWRS2tra+qWUd0/lO/NCkK1evRqRSGS2j0EIIYQQMilCiO6pfoclS0IIIYSQWYaCjBBCCCFklqEgI4QQQgiZZSjICCGEEEJmGQoyQgghhJBZJm+CTAhRIYRoFkL8TyHEZ0KIH6Re/0shxCUhxMepf/5lvs5ACCGEEDIfyGfsRRLAn0gpzwohlgJoE0IcS723W0r54zzemxBCCCFk3pA3QSalvAzgcurPV4UQvwWwKl/3I4QQQgiZr9yWHjIhxGoAvw/gw9RLLwshzgkhfiGEKMnwneeFEBEhRKSvr+92HJMQQgghZFbIuyATQiwB8EsAu6SUIwD2A6gC8DVYDtpP/L4npTwopayVUtbeffeUtg8QQgghhMwr8irIhBBBWGLsv0op/xEApJRfSCkNKaUJ4BCAb+TzDIQQQgghc5289ZAJIQSANwH8Vkr5n5TXV6T6ywDgDwF8mq8zEEIIIWT2MUyJPU0dOBntR324DDsbqqFrYraPNafI55RlPYBvA/hECPFx6rV/D+BbQoivAZAALgDYkcczEEIIIWSW2dPUgYMfdGF03MBnvSMQAHY9sXa2jzWnyOeUZQsAP/n7T/m6JyGEEELmHiej/RgdNwAAo+MGTnb2U5B5YFI/IYQQQvJKfbgMhUEdAFAY1FFfVeZ63zAldh9rx9b9p7D7WDsMU87GMWeVfJYsCSGEEEKws6EaAsDJzn7UV5XhlYZq1/ssaVKQEUIIISTP6JrArifWZhRZLGmyZEkIIYSQWWaykuZCgA4ZIYQQskCYq/ETk5U0FwJ0yAghhJAFgt2rFekewsEPurC3qWO2jwRgoqT59vN1kACeOdC64Jr76ZARQgghC4Tb2avldeNe2hLGvuZoVnduITf3U5ARQgghC4T6cBk+6x3B6LiR914tr7g63TWAcz3xrGKrpaPPJRhbon0UZIQQQgi5s8hHr1amvjSvG/dZb9z198MtXTCkVZJs7RxAfbgMmscx07WF01lFQUYIIYQsECaLn8hEtmGATGVGrxv3eyuW4ezFYSRTfWHXxgzsf68TAJA0JT7rHUHJ4mDafRcKFGSEEEIIyUq23i6vE3YkEkNLtB91VcuxfVMlWrssN27cNNHWPeS6blIRXKPjBvQEoAnAlEAoqGFjeOHEXywcL5AQQgghWcm0wshvGMBGzRALaAJ9V8cQ6R7C4RPnoQmg8YUN2PXEWnzYNQhjEsPr+pgBU1qi7MHy4gUVf0GHjBBCCCEAci8/2sMAhilhSonixUGUwCo39sYTACaa8iUsQadpwnG/MmG/ZUrAlHJOZKTdLijICCGEEAIgcyzGzoZqSCnR2NYDAcCQ0ukrO3zivCPU1pUXYejGuPN3TQjsa44iaUroArh3WQifjyR8RVkgJb6SplyQaf0UZIQQQsgCRm3Y1zSBwqCG0XHTJYp0TUAIgeGU2Dp84jx0kT5JaZgmdmyudKY4j0RiTp+YIQEpJXY+Vo0jkRiujCScEuaSRTr+txXL0DM8CgFga03FgipXAhRkhBBCyIJGLVOGAhoerCiGKWVaLIafe+YtZW4M3+2a4nz7zEXXvfqv34QE8P6/24Jvv/khzlwYhCmBRNLAr2PDjjumCSyociVAQUYIIYTccUxlZ6UqtBJJE6aUaHxhQ9rn/PrIXlFyzeoql8OQElv3n3LuWVG6GJ+PjDnXGDckDn7QZZU9TemULpMmYHeQ5XuDwFyFgowQQgi5w5jKCqJc0/vtUNmWaB90TcOJjj6c6hqAaUrUh8tgSun0k9n3lD69Yn7u2kLvHwMoyAghhJA7jqnsrMw1vd8OlZWAI/Zs7FDXbCVNm8KgjrrK5a7pzG9+vRyaEGjt8nfaFkL5koKMEEIIucPwul51lcux+1i7bwlzKun9hinRGIm5BBZgCbBiBFEY1H1LmrarZpgSG8NlMBQ3rTCoI5A6A7AWu4+149ACXDBOQUYIIYTcYXhdL0PKnETOZL1ne5o6cOXqWNr3CoM6ttWUAwAa23oAKXGysx8tqesc2bHBdZ2t+0/5pvvXh8um5O7dSVCQEUIIIXcYXtfLK4D8RI5hSnzr0GlEUpOPv744hDdburAsFMRTNeXY9fhanIz2u9YdLSnQ8ZV7l2Fj2HLD9jR1ONEYl1PN/GcvDuF01wDe2v6wI8q8/WN9V8dwOZ7AZ70jWFdelOa0LQQoyAghhJDbzFSmIGfiun6N+97PmlI6YgywcsOujRnOEnBdpF/nexsrXcJOdbdsTAmcuTCIvU0dzmfVoNn+a2MYNyYmLL1ZZgslj4yCjBBCCLnN7GnqwIH3O5FImr4O0q1c12+6cqdPPMXG146j7+oYkqZ0mvIzrTVKmlYJ8u3n67IOANSHy/DppTgSVo6FgynhcuXUoNlxZcGlX5bZQoGCjBBCCLnNnIz2O6LFz0ECpueiefuvjkRijmiSsGIoTp8fxCc9wxgdnxBNdlN+KKCliSkbTVj33vXEWqc8+cyBVtRVLQcAtHYOoK5qOdaVF+PMhUGo2i4U1NJcud7hUZebtqTActxe3BLOOIBwJ0NBRgghhNxm6sNlOHtxyHGkvA4SkO52SSkhUuuKMgmV+nAZPk4l3gPA5/EENr12HOWlix0R5rfgWxNARUkhttVUONETpgT+9tR5XBuzRNOZC4N49tBpvLX9YdfZPo4NQ0oJQ1r9YnctCrjEmC20bBFnfy+gCQQ04WSPfbd+DSSAR37U7HLuOGVJCCGEkLyws6Eap7sGnNVBtoOk4nW7Gtt6nIZ5VaiorlNd1XLcs7QAvfEEAMsV640nXAu9TYk0UWamxNTDlcvR+MIG55oqqpOnnk1t8jclcHUs6RJaap+Z93sri0JYWRyCrmlobOtxhJgNpywJIYQQkjd0TeCt7Q9b4iZLP5baQC8A30lJr5O2rrwIQynhZqOKMHVf5W8vjzgOWNIEfnHyPE51DkDTRFpZ077OkUgM3/z6Kvz64hCMDD1ndy8tQHlJYdpzeZ/p6doK36BZG05ZEkIIISSvTBbI6pclpoap2kLF66TZU4pHIjHHcfJbGq5rAhv+qskRZIA1VRnpHvIta9pcGUng/zt7KaMYA6zyZ11VGVqi/ZCpZ9E14ZquFAAMKdHaOZAmxgKawD1LC/B0bQWnLAkhhBAye3gFm2FK6EKkOWpe18meUnyloTrNgfP2nD1VU46fHY/Cq61UMVYQEFik67g6lrTOIeGURDMRG7qBT3ymPdXpytFxA4dPnHfljnmF2EJo5rcR0m/z5xyjtrZWRiKR2T4GIYSQBUK+csLygWHKSYVXtu9ueu14VoG1qjiEb3693HHnsrlngFUavXdZyHXNFUUhrCoudJL4I91Dznu19xdjY/juaZ1/riKEaJNS1k7lO3TICCGEEA+Z8rzmIlPZRen33W21FRl7uAAgPjqOXY+vddw5TQj8JjaERNJSZYt0AcOUTgnzrkUBrCopxNCNmxgdNydN4l+ouWNeKMgIIYQQDwtpn6Laq6YJgY/OuzPEHlhZBGAix+wblaV4aM1ytHZZjtaLW8J4oznq9KxdHUvibPcQChfpWFkUAjBR4lzISfyTQUFGCCGEePBbNTTfyVSGVR02w5TYfex3+NtTFzCWNPH7FSX4u+ceSnMMd2yudMVjnIz2Q2AiAsNeu2SYwL9YtQxXUsMFAU1gQxUdMT8oyAghhBAP3gnHfLk4fiIJwJT613Ltd8ulDKtrAn/6B1/Bn/7BV1yvZ3IMMwW92oyOG7g4eMNzkrnfuz4bUJARQgghHm6lL8smF7FlKlEWtkhSc7my9a8ZpsTr77bjb05OpOmrezG99/KKqn3vRfFOJIanasqtHrEsoi+TY+gX9CoBJ26jMKhDExMiLWlKtHYNTPs3vZOhICOEEEKmSC6ulJ8j5RVbJYuDac6TlP4BsF72NHVg/3udaUn5dpq+915qMz0AjBsSvfEE9r/XCV2IrOLTdgxbon3QNc3JF6urWp4W9OqN28iUn0bcUJARQghZcNxqrEUu5T+/Mp9XbBUj6Jo4tAWMmoIvUuf1nu9ktN8lxmxMaQknQPgGxh5u6XKFwSZNOenQgu0YekXe9k1r0hr0c81PI24oyAghhCw4bjXWIpcpTL8ynwRcr22rKYfmESuvv9vuyvlq6x7Cs4dOwzClSzx6F4mraEIAQkysSwpqTryEBPCz4x0uwVdXuXxaz93aNYDGFzbkJObYxJ8dCjJCCCELjluNtchlCjPTYID3NV0TeKWhGnuaOvDMgVb0Do+62t4NCXx4fhCAu0fMXkN0tK0HfdfGMK7sMooNjWLo+k1nh+WD5cV4paEahilhSolQUMf1m9bzW85bujvo5yLeidOncwUKMkIIIQuO+nAZPr0URyJpQhOWo+RXFrRFSUtHH3Rdg5lyqV7aEp50CtMWWhJAS7QfRmozTmvnAOrDE2LMMCW+deg0znjyv/xQe8R2PbEWrz75Zbz65Jex+1g7Drzf6TzPtUQSiaTpfMdM3ftbh04jcmHQ5cBZjfb9ANyCVHURP44NozE1ALB9U6WTQTbd8uN82oRwu6AgI4QQsiBQRUBd1XKsKy9GpNsSJ+d6hh2Ro6KKEhu1xDmZq+YVNYAlgNRr7Gnq8BVjmVYUmRJpjt7Ohmq0dvbjowtDMCVwbSzpRFDYTtaepo40MQYgo9PlnaDsjSdw+EQXdmyuQuMLG7I+92TMp00Itwtttg9ACCGE3A5sERDpHsLhE+fRM3TDESej4yZOdvanfaeloy9tpZBd4pwMw5RojMRcosbu91KvcTI1sehl8SIdmUwjb8+Xrgn0DI06f5ew+sZq7y/GuvIitET70RiJpYmxlUUh7Nhc6et01YfLEPAcINPvNFX8SsYLnbw5ZEKICgD/BcCXkJr0lVL+VAhRCuAIgNUALgB4Wko5lOk6hBBCSC5MVgbzigC/CUcvup7uW+TaO7WnqQNXro65XrNdL/Ua9eEytHUPpYmy62OGr1Czr7H7WLurlDqSGHd9blkoiPrw3a7gVl1YPWmaANavLsVb2x/OWCrc2VCNxkjMtSRcE5iRvjH2oqWTz5JlEsCfSCnPCiGWAmgTQhwD8B0ATVLKvxZC/DmAPwfwwzyegxBCyAJgsjKYVwT4TTh6MT2W0pICHd/b6O8oefGLpTAlENQF1pUX4cUtYQCW8DndNeA07ttk6iczJfDLsz0YvjHucu90j64aSYzjHY9Dt7IohFUlha6BgkzYi8fV3rT1q0tnJLbidm1CmE/kTZBJKS8DuJz681UhxG8BrALwrwE8mvrYfwbwHijICCGE3CKTTU76iQDvhKPXWVNFXEATWBYK5rz4R/2uyrghca4njjeao9j1hJWQ/9b2h/HsodNposyPwqAOAaRd15CWYLQzxq6NGbg+ZrhcsYrSxb6uWCZ3MdNvdqswCiOd29JDJoRYDeD3AXwI4EspsQYAn8MqaRJCCCHTwlqI3Y5Lw6NOz1OmMpgEIKXbfVJ7y/Y1R7HptePYfawdhimxs6EaOzZXYkVRCADQG0/g4Add2NvUMem5djZUY/umNb59YN6+KVuUrUzdxyaoC+eZAprAyqIQtm9ag1UlhWnXLQzqWBYKpj1vYaoXTR1e8PL6u+3Y1xx1foOfvtvunGvXE2udrLGFPgmZT/IuyIQQSwD8EsAuKeWI+p6UUiKDKyuEeF4IERFCRPr6+vJ9TEIIIfMUW1BdTvU6ZWpUV4WXKqr8pgnt921Bsqq40LchPxu6JiCE8En48heMdomwMKg7n/n+I1V4eUsY61eX4OUtYZz44WOQAM52DzkN+iuLQqi9vwQ7NlfiqZpyVyO+7epNNrxwtK3HtW/yaFvPpM9HZpa8xl4IIYKwxNh/lVL+Y+rlL4QQK6SUl4UQKwBc8fuulPIggIMAUFtby9XwhBBCfPEKqlUlhU4ZUi3BZSpp+pUWvSXP6Tahn4z2Q8lrxV2LNCwrXGStQ5IyLfvspS1hnO4awGe9cTywsggvP1aNRQHNVdr7ZVuP65oAcPT7VgyFYUoIwBFUW2sqIDH5LslcfC9mh+WXfE5ZCgBvAvitlPI/KW/9dwD/FsBfp/71v+XrDIQQQuYnU/kffz+x5Nfgn0lU2X1SRyIx9F0dc2V32dip+I1tPRAAxk0TP/nV75yQ10zn8/agCSGcexw+cT5tqfe+5ijO9cQxOm64+sxUsjkUuiacsFj1t5xsl+RTNeXOovKAJrC1piLtM8wOyy/5dMjqAXwbwCdCiI9Tr/17WELsHSHEcwC6ATydxzMQQgiZh0zlf/z9Gs+fOdCa5oa9/Xyd72SfXZZ8paEae5s6fIWLXX60JxsPvN8FID3kNdPZbLGnLvW2z6W6eb3Do5OudNqaJp7K0+7rJ2iziaddj6+dVLTd6ropkh0h5dyvBtbW1spIJDLbxyCEEHKb2Lr/FCLdExGV61eXTCkdfvexdkfQFQZ17NhcOWXx4BU1J6P9rjOp1N5fjLqqMhxNOWhP1ZRbIiflmnmfB4BzLgk40RKAO6ts+6Y1EEK4hBUARzjWVS6HBNKcOvX5QwEND1YUpy0nnyoz8ZsuFIQQbVLK2ql8h6uTCCGEzDly7dmaSlzDVPG6dOvKi5wg2YAmIKV04iR64wnsfy+KlKbC3uNR/Pz9TuzYXIX/09Onpgtr8nFZKAhDSrR2DjhiDLDE2MqiEJ6urYAhJQ4pZ5BSOgKtrmo5WrsG0JZq8P+sN+44daqblUiaOJNamXQrpUZmh+UXCjJCCCFzjlz/x9+vtOlt6J9udpa3RPfppWGsKy9G0rBCUj/uicMwJEwJfB5PpK0lGjck3ngvioBHIGpC4FzPMHrjCbzRHIXmc7ZVJYXY9cRabN1/ynWGxraJQFhvur89QekdVFB3Yt5KqZHZYfmFgowQQsicQ+3ryhTaCvj3NUnAEWmfXorjdNdAxnKd12F7aUsY+5qjOBnth6YJFAY1jI5b7tX1mybO9cSxrrwIbd1DrhR+vyXg9ut2n5idgRYbuuFc05CA4RmZDAU111ol1SlUA2H9lpF7BxVUATg6bnJN0RyGPWSEEELmLH59S6oDpmkCnyhiY8fmSrR4er0ELPHit7/Re/115UXOlKPde/VZb9zVjB/UBcY9IkrA6nP7dWzY9Z4mgJ2PVbtEolru9KKuZtI1AcOUrkEDQ05EWHh5aI3/bkrvNWYqbZ9khj1khBBC7hgMU6JR2cXo54DZosmU0hEbEvAt15kSOHNhEHubOpyym9dhO3txyBFUiaSJi4M38J0Na/Dz962pRgGkiTEA+EZKDBmmxB8dPo1fx4YRCuj4zobVaVOfSVO6VhzZBDSB7210N8qrZULDlHj93XYULw5CTwCjN42cFoWz1Dg/oCAjhBAyJ9nT1IErV8ecvwc0gfqqMrR4GtYvDt7AquJCp4TnLdfZDe3ARAlRDXz99FLcaar3iq2+q2P46PyA83e/MuHLW6rxg8ct10nXBN7xmQb1lh6XhYJprtv3HwlnbZTf09ThuGOhgIba1aUuIUrXa35DQUYIIWROcjLa7+rTumdpQZoDFtCsoNXL8YRrglB1lZ49dNoRZXZ/lt071tLRh9Ili3yb8gHLzfqfl0dc51D50tICSEhsfO24b9yFjXdIQS095hoh4Z2cNKWcUhSIjfrsuq7BvMU4DDIzUJARQgiZk3hdpadrK9IiLS4NjaI3tcNSnSBUm/XrqpbjoTXL0do10UOlTmfaPWZ+FAZ1PLByoq/MS0XpYiekFQD2v9eZlr4PuIcUXn+3HY2RGHQNWFEUwraaipwiJKa7vsmL+uw2TN6ffSjICCGEZGW2dhhmir5Qe6K8Tfm2SPHGYezYXOm4Sd7eNK8YC2pAQVDH0lAQ22oq8PJjYbzRHMWRSAxXRhKuvi3DlC73LGnKrLESe5o6XAIuMW5FaOTye85UDpjqtNkweX/2oSAjhBCSlVvZYWg3omdKsM+Gt6F9KgGw3mb9wy1dkLCWd//Rmx86rprvmSVwbcyAYQKAdGIwttaUQ0A4TtuLqWup2H1umc7rLcNOJuAy/R63gt8ydcZhzD4UZIQQQrJyKzsMvY7Qz45H8cu2HmyrrXBESi4OXCZRmEmkeEXHtTEDe493oLEths+ziDHAHaKqBrHaTtvbz9dhT1MHHvlRM66MTFxraUEAf1y/Bi9uCeP/ONiKMxes6I2PY8OQEnj1SSuw9ePYsPN72AIuV2bCrbRFbEu0D7qmwTAlNoaZvD/bUJARQgjJyq30LnkdIQlrzdDBD7ocUZWLAzcVUWiYEqaUKF4cRPKa6UxOmhLoHc4uxlQ0AVxLJLPGbqh8ZcVSvPqkVUa1xRhguWBH22J49cm12NlQDSkljrb1AAC25tg/ZnMrbqUNYzDmJhRkhBBCsnIrvUteR8hGFVW5iC01nkIA6Bm6gbq/avItg6rxELl6R2pemY0pgdHxJAKaQNKUjhht8enBCmgCl4ZGsftYO05G+zPeR9cEXn3yy3j1yS/neDI3U3UrZ6v/j0wdCjJCCCFZuRVHRXWERhLjSIybLnEDuMWWJgBNWGVMVTjsbKjG6a4BJ77icnwin8w72aiKlsl20awoCqG8pNC1XkgVZ0kTWFlUgFUlhb7BswFNIBTUMHrTQG88gb3HO3BvUQi6gJPELwBsrSmf8m/nx1Tdyplw1MjtgYKMEEJI3lAdIb8VPkC62DrXM+xK07evY5Ui0+/hbYz3a1r3ozCo45naCmdowD6bKs4CKVG4QQlf9TqG6qomuyy6sqgAEiIlxirwg8en35/ljfDYvqnSFeGRjVvp/yO3FwoyQgght4VMTptXbI2OmxnLln5CK6AJ1FUud8qFtmhpbIuh7+oYkqZMK0kGdYGSxUEYUsJIvWEv//5GZSkeWrPc+b635837HBLA2YtDrut/PjKGnY9Vz4j4yRbhMRkzlV1G8g8FGSGELDBupa8oXz1JfsLBe6+XtoSd6UBNCMSGRh0HSkLikEe0tPzwsYyul5TWcMHhE+ehC+Fq1Le/v6q4EJd9Qme92A7fh+cHnddMCSdq41Z/o1txuWYqu4zkHwoyQghZAKjiRtMEPkmJk6n2FeWrJ8lPOKj3+jg2jMZIDE/VlKOuqgytnQN4WonO2Lr/lG/u2M6G6rSSpDfd/0gkhquJ8TTRk6u7pGsCb21/2LWiCbCiNlRnbbrcisvFicr5g5ByspbH2ae2tlZGIpHZPgYhhMxb1ER7b/lu/eqSnEtgW/efcvqlAKspflVx4Yy5Zapw7B0eTQtwtXu67MEAewek+nw2mXZEqp9Vr+f93isN1Wk9b9mezxZ9h1u6XIvDp/L7Zrturucgs48Qok1KWTuV79AhI4SQBYBa9jLlRMzDVB0X1a3xLvaWUkIIkVbOnEqZ8/V3250gWQE4kRM26p/V8p09zfnGe1EkzYn3W6J9aYIs0y5MAFhSoON7Gysd0TMVd8n+vFr+nIm+LbpcCwMKMkIIWQCoQioU0PBgRTFMKafcV5Rtsbc31T5b8KtdkvSKtKNtPY7okgBCQQ1LQ0GnOV9FE0BdpSV2dE1ACJFadzSBrmlpz5BtF+b3NqY7alOFfVtkOlCQEULIHcRUdj5Op+yVScyEAppvqn2m4FdvE73trvVfG3Pdb1koiBOp5nxvKdDSZ9J57sZILC13zDCzt+XkQzzR0SLTgT1khBByB+F1fPx6qGYKb3ZXW/egUy7UBVC7uhSGKaFpAudiQ0gkrfiJ9anXvb1otrtmowug5v5Sy8kLl8GU1qJvv/633cfasa856nLRAprAy1vCeRVGTMInfrCHjBBCFjiTRSTkKiBy+ZzqBG3df8oRYwBQuEh3YiZCAQ2lSwrweTwBUwJt3YMIBXWnPyygWc6YvXMSsHq5HlhZhN+khNzZi0Oovb8EtfeXItJtTTKGgprTn+XdmSkAfP2+Yry4JTyl554qTMInMwUFGSGE3EFMFpGgCohPL8VxumsAhinTREquQsMWOpeGR107H5eFgk5/WSJpYmR03LWO6NqYNRSwZJGOG+MGTFXMBXV8t34Njrb1IJGcWAwe6R7Cy1vC2FC1PK3E6A2NlQA+uTSCN5qjOS8wnw5MwiczBQUZIYTcQfj1RHmjJGwBkUiaTpipd0pS/Zyd1eXXd6YKnYAmsLIohKdrK2BI6Sz4Lgxabte5nrirJJk0pSXGPAn62zetQWvXQFrkhSmB1q4BNL6wIePkpNpnZp+7ZQrPM1WYhE9mCgoyQgiZ5/iV41TB4s3dEkhfum1PSQ5dv4lE0jOqCKDv6ljafknA7RAlTQkJOK6VLoQjDF/cEsYbzVEcicTweTzh3N/bc18Q0CABtCn9ZTZqiTLTM6vDAmosh/e5Mz3PVOFEJZkpKMgIIWSes6epAwfe70QiaeLsxSGc7hrAW9sfdtwfr2gK6sLVrwVYZUIB+Iox+3uZ9kv++uIQ7Mtdjiew6bXj2FZbgZe2hCEBtET7ndT8Vxqqsem142nul01i3MQv23rShNqSAqsMau+ezFSCzBTL4X3uTM9jk2vPGScqyUyRHtBCCCFkXnEy2u8IKVMCZy4MYm9Th/N+fbgMhUEdgCW8vn5fifP3gCawYlkB1pUXpblmgJX1ZX/P3i/GKVijAAAgAElEQVS5+1g7tu4/hZ/86ndImmba93rjCew93oFHf9yMgx90ItI9hIMfdGFvUwd0TWBbbYWTkO8laUr0XU1AfXdlUQiGKZ3dk3tTQslbgjRM6Qikxhc2YFttRcbnnqy8aAs+9eyE5BM6ZIQQMs+pD5fh7MUhx1UypeUQ2eGrLR19WFdeBMOU2BieKB/aZTa130st6xUEBL5WUQJTStRVLochJTa+dtwJaf04Nuzcz4spgd7hCRdMTc3f2VCNxkgso0s2rph0tnAbTb2o7pn8ODbsTFZ+rjhzmbLXvM+drbzIZn1yu6EgI4SQeUK20NfTXQPOYmu710ot6xUGdWzftAYSwLOHTqM+XIa3n69LW8wtYTlSq0oKHRGzrzmKdyKxtLR8b3L+ZNip+bZL9tMcXCe7Ly0U0JBImtAEoAmBl7aEXaJOwnLm1GXefuXEXMuLbNYntxsKMkIImSdk6pvSNYG3tj+ctoD6mQOtLpdn//udkNISOer3veLj6doKR7T4Le32I6AJ3L1kEQau38RNw1+oqan5Oxuq0dgWc7lo9nW8i7631ZTjw/ODjuD8TWwI337zQ0ifz8+Um8VmfXK7YVI/IYTME7buP+VKt7dT6jMxmZhaURTCquJC1FUth4BAa1e/yxWz4y8ylRZtgrrA3UsK8FRNOUwp8fP3Op0mf7sEaif0q8MGN5Mmav7jMVwdSzrXWlkUwsriEHRNc0qstrhUn91ejh7QBEJBDYlx08lAy+d2AkJygUn9hBByB1NXtdzpmwpowlmsnQm/bC4bNRLi49gw7llagG21FU7fmRodoQsgg+mFgCYgJZyG++2b1qB2danjZtmYEjjXM+yKmlgU0PDdjWtcZVXVnVNRXTxbjAGW2/eVe5diY/huullkXkNBRggh8xDDlDjaFoMQmDSSwZvNdc/SAgBwRUJY/VedVpnOE5OxpEDHjZvuAFe/64yOG2jtGoCUE4JJ1XGj42ZaOTHX0qD6OU0IZy1TYVDHxvDdjJ4g8x6WLAkhZJ7gLVkCVrP7gxXFaeuP1AEAb0nSdsG8y7gBqwz6cOVy7H+vE0lTQhfWXkrVYVtSoON7GyvT3DS7XCgBJxcNmCgvzlQ5UV1qbj8PF3qTuQRLloQQMk/JJYjUu68RsIJc7fKg2qjvHQDYsbnS1W/mFz2hCTgxGDamBK4rYiygCZcY80Zq2A6XOvWpCeDeZdZKJe8qp+ks+mYYK7kToSAjhJA5QC7Lr+2y3RElgkLtp1InDL05Wodbupy0fF0TTvSE7WTZTfd2A73tnHlrKPcsLfB1xuxIjWcOtKI+bAXIqsvEV5UU+k5uzuSib0LmMxRkhBAyB8gliNR2hl5pqHZKdt5+Kjsvy+umXRszcPCDLtcC8bqq5Xh+c5WrlKlrwvVdO5jVnmB8urYCANAYibnO29jWg+Eb4xgdN/DppThKlyxyxKK6gzLXZyVkoUFBRgghc4CpBJGqwuz1d9txcfAGiqVEReldzt7Il7aE0yYsvcLJr5SJ1HdPdw3gs944fm/FMjy0ZjlOnx9w9Z9duTrmfN5e3G2LrETSxOXhhBN38WB5satZn6GrhKRDQUYIIdPkVnuhVNQpQntN0db9p5zrAki7156mDtfKo8sjlkj6rDfulAHVCUt7gfhk7tS+5ih+ExtGImki0j0EIYST6g9YDpc6DHDPUiuDzD6LWkY1JWBK6fpdGLpKSDoUZIQQMk1y6fvKFbVRffexdhzyXFcVVvZraukvU7yE6nY9sLII61eX4s2W81ndqUzLyu1n80v2f6WhGroQWcuofs9KCLGgICOEkGlyK71Q2dw1v+tKme5s+U1dAhPTkoDldp3riWN03EBb9xAuDlxPm4o0TInX323H0bYeCFgN+F6X60gk5iT4+01W6prAKw3VkEDq/WLXMvOf/Op3zvWfqinHrsfX3nJUxUw6lITMNhRkhBAyTXLphcokGjK5a4YpoWkirSFeAmn3ejHlfp29OISkIV0riuwyoDfk9fLIGIZHk648sN3H2p3cMQD4YiSBe4tCrj2TV0YSePTHzfg8nkhlimnYsbnKJUD9Mslsx0+9/v73OqELccsO2Uw6lITMNlq+LiyE+IUQ4ooQ4lPltb8UQlwSQnyc+udf5uv+hBCSb3Y2VGP7pjVYURRCyeIgDCldC7SBCdEQ6R7CwQ+68NN327H7WDvebOlKc7zsz5+LDcOU1h7IksVBHInE8M6Zi/gXq5ah9v4S7NhciRe3hPFHb36Ij84PYjwlxpYs0lF7f6kjAu2w2MKg7jqTfT/DlM5Z1J4wQ6Z2ShaFXK/1DieUiA0TRyIx1/P6OXv26+r1k6Z03rsVMt2PkPlI3gQZgL8F8L/7vL5bSvm11D//lMf7E0LIjGALl637T2H3sXZHhOiagBACwzfG0RtPYP97ndj02nHXZ7yi4Rcnz+PgB52u5HvVXVP7tySAy3Fr3+TlkTGcvTiMjeEy7HpiLfY1RxG5MOjqHbt208DZi0OO+Nvb1IGXtoSxrrwIQd1y3dT77WnqwIH3O333XG4M341ttRWOmPOrBPZdHcPepg7n76r480ZwBJQLBDQxI5OVme5HyHwkbyVLKeUHQojV+bo+IYTcLrKVxrwlQWsnZJfzmfpwmbMQHECa+FHXEAGZl2jb11eDXz1mnPMZQOk9A3CuJ45xw1qDtKRAx7KQ5ea1dg444s9maUEAf1y/xjmPuj/yN7EhJJJup0sNnM00PbmzoRpSShxt6wEAbK2p8J2snGpPGKc1yZ1EXndZpgTZ/5BSfjX1978E8B0AIwAiAP5ESjmU4bvPA3geAO67776a7u7uvJ2TEEKy4d0huX51iZPdpabOq9TeX4z68N1o6ehD+xfXcHUs6byXbbejuqdREwJt3YOwNVNAE3h5S9jpy1L3RQKArgGQVnnR7iUzTOk6u3rvdeVFznojv2dTsc+lbgmwyXVHpXd4wNvcr/6WM7X3kpDZYDq7LG+3IPsSgH5YTvx/BLBCSvndya7D5eKEkNkkm1DwEyq22LGnG9W0+4ll4CY0IRAbGs04eWiYEj9NCRjAcpZ+8LjlGt1Mmvj2mx/i00vDKF68CPcWFaKushT/ePaSs5+yMKhhXXmxcw6v43bvsgJcG0s6rp0mrO8IIbA0FMTWDGfa29ThCpwFMgs57++oLjRXBSaQXfgSMp+Y88vFpZRf2H8WQhwC8D9u5/0JIUQl1xLZZKGt3nVG9VVlaPGUMlcWhbCqpNCVdq+Kk73Ho/jo/CDe2v6wcwZdE3j1yS/j1Se/nHamiTgLE6Ycx7aaCkgAn49MTEaOjpvoHriO4sVBlCCI8pLF+OSSlQ8W0AT6r91E0pQIaAKhoIbrYwau37Qct2tjhu80pJ0h5g2czaV/K1Nzf6Z8M/aEkYXEbRVkQogVUsrLqb/+IYBPs32eEELySbbeMD+xlim0ddcTa9PCTiWATy/FncXdFaWLXWLLK04A4MPzg3j20GnX5zKRKavM21dmi67CoI6tNaXYULUcJzv7cWlo1HHSMi0S9womlen0b3n76bzN/ewJIwuZvJUshRD/AOBRAGUAvgDwF6m/fw3Wf+4vANihCLSMsGRJCMmFXB0v+3NvZim7ecuU2zetgRAi63e893j20GmnR8ub2+Ut39loAtj5WLWvMKyrWg5TSvzj2Uu4mhhHYtx0xNaOzZWQgKuvzFuizPZ868qL0NY95DqPt6R4q2QrwRJyJzGnSpZSym/5vPxmvu5HCCG5BoWqn7PRBKAJ4cRVNEZiLgdKXcptk62spmvWtdTcLtVtsicPf3HyvEvg2an4LSlRaUrp7Ij8ODYMw5SOkyVg5YXZq4vs1zI13mdzo17cEsbPjnegMRLD1bEkloWCGachp0u2EiwhCx0m9RNC7hhyXWWkfs7GlMC5nmHsbeqABHDl6pjzXkATrqXcABDUBbZvqswqWOrDZa6ypS34dE044uQHj6/Ftw624qMLVjO7gJWUfzmewMexYdyztMDVi6YiYa05Up9x1xNr0RLtx+X4RC+ZN1oD8N8nSbFEyOyRz2BYQgi5reQaFKp+Tq2W2S6Wt7/rnqUFeKqm3BVuKlPREpPlZD1YUeyUDtu6B53g2JtJE7uPteOZA62QAAKp/zaWmCgzJk2JkcS4674qAkBd5fK01+uqljvfCWgC362vdPrcCCFzEzpkhJB5R6ZeMbsM1xLtg65paIn2O6GlqhhRy3WaEDjXY00e2iLOuzfSLgn+sq3H1QhvO3CZzuMtWyZNOMGxp7sGMsZRqCwtCGBZSDj3VbGeKReRlb94I0LIzEBBRgiZd2TqFfOLZPDrJbM/90pDNV5/tx0XB2+gWEpUlN6FllTz/PZNlWjtmoi5eOZAK8pLF2Poxk2XeMt2HsAd5WBjfS7uvJZJjAU0gW21FRBCYO/xjrTPJU2J1q5+GGa1SxC2dg44Dp/1mYGZ+NkJIXmEJUtCyLxjsqXSuS6d3tPUgcMnzuNyPIHPR8bw0YVBRLqHcPhEFzQBvP18HU6fH7T2RnYPoe3CIHRNYGVRCNs3WQu+sy0Kt9wxieLFQSwp0J0yYmFQxwMrizKWIu0VR/csLYAE8MIjVbhXWfRto+6kVBeY65rgjkdC5hl0yAgh847JAkRzDRhVhZtqPtm9ZBJARFktZEgrMNUwrf6xfc3RtGlNr3NmT0hqADRtou+s5v4SxAZv+JYiDQncuGng2piBwyfO46Pzgxi8dhOAVaBcUVQACWvQwN5JqQpCwzSxY3Ml87wImUdQkBFC5h2TBYjmEjBqmBKaJjL2b10aGkVjJOb7nhrEqoox7zSjKvhMAGZq7eS1sSR+/n4nau8vxRVPPAXgzg8bHTdw9uIQxo2J8FYJ4URwHD5xHuvKi1AY1B0BujF8d9oEJSFkbpPXXZYzBYNhCSEzjXc5t8CES2YLooAmIKVESgs5r9v7KC8O3sCVkYSzzLv2/hI8VLkcrZ0DqKtajqNtPa74CS8rlhVACHfD/pICq5ypLhVX710Y1FGyOOj6Tu39xdgYvtslQDlRScjsMaeCYQkhJBu5purn696NkZgjxgBgRVEIK4tD+F+fX3WCWpOmxIqiEMpLClPxEgKtXdZk5m9iQ0gkJ/4PrSmBjy4M4cyFIavUqSzJzoQQVtO+mphvO2ybXjvuEl2mnAiBNZSwWDpihNwZUJARQmaFXFP1c2Uygae+r2kCV65OiJ2AJvB0rbWcu6172HXd+0oX48iOOuWVtdi6/5RLjKlkqzkENDiul64BW2vKfcuremq60jtZaYfAGqaELgR7xAi5g6AgI4TMCrmm6ufKZAJPfd/bN3bP0gK8uCWMR3/UnCaoDNOEl/pwGc5eHMoYV+FFE8D61aX4u+cewhvNUScn7VTnAITocBaXq7y0JYzGthh6hy3hGApqzrCAX8q+l9l0IAkhU4eCjBAyK2SahJyukPAKvBMdV3CqawCfXRpG0eJFiI+Ou3K/1J6sp2srsK856lqXZKNrmrPuyGZnQzVOdw04i8PV/jP1z7oGfGnpxK5JXRN4paEap5TvftYbd8Sj18UbvGadRxPAg+XFU3LCZtqBJITkFwoyQsiskGkS0k9IvNJQPalI8wq8yyNjjrt0/aa7sd5uyjeldO79zIHWtGlHAPjw/CC+9h9+he/Ur8auxy1B8/q77egeuI7Fi3QsDQXx1NfLoQnhBMnavWZ+DfZ7mjpcURrq0vFMLp4pAVPKKTlcM+1AEkLyCwUZIWRWyFR28xMSkyXvA+kC73BLl+9971qk4aurimGYEvXhCSGoRmAIYe2qtLk6lsT+9zohAJw+P4iPzg86710bM3DmwiDe2v4wXtXUM/mLn5PRflepUxNw3EFXTIbHxZtquGuuWWyEkLnBpIJMCPEKgL+XUk4+MkQImffMdu+Rn5BoycHt8Qq8U10DLuEEWALnq6uKnR2StriTAM7Fhh0RdG9RCJ/HEy7hlDQlGtt68MVIeozFmQuD2NvU4ZQdX3+3HUfbeiAAPFVTjl2PTyz2rg+X4dNLcSSSptNbZotC9dn9XLypkEsWGyFk7pCLQ/YlAGeEEGcB/ALAP8v5EF5GCJkWt9J7NBNiziskdjxShXfaYs77anN7Nv7+uYfw6I+bnbKlSIkfw5TpDpyEE4FhSiB+4ybuLQo53wWs3rD+a2O+jfymhKvs+EZz1Mku29cchYDAq0+u9X0+taT50pYwTncN4LPeOB5YWYS/e+4hLApMb8NdLo3/hJC5w6T/SZdS/t8AqgG8CeA7ADqEEP+PEKIqz2cjhMwCue6B9MO7U3FvU0fGzxqmxO5j7di6/xR2H2uHkVI6tpBofGEDdj2xFv/mFx+6hFHp4kU5uT2LAhpO/Nlj2NVQjfWrS/CDx6rxd889BKHow4AmUFdZhvpwmbP7EQCu3zQxeO0mvrG6BCuLQlhSoEPXhJOW70UViSej/VA/ZkrgqCIovc+nCtZ9zVGc64nj2piBcz1xvNEcnfQ5CSF3Bjn9X6+UI/Z56p8kgBIAR4UQ/28ez0YImUEyCSAvqjiZau/RVMRcruLt45g7F6w3nsCD/+Gf8fSBVtxMmriZNLHt56dQ/X/9E776F/+MH//z79LE3dvP10ECeORHzThzYaL7wvqcxM6GauzYXIm7Fk38V2IiacKUEid++BiWhYKuhv8lBTp2PhbGzscssffC5ipX2XG63IoYJoTMb3LpIfsBgH8DoB/AYQD/Tko5LoTQAHQA+LP8HpEQMhPkWoqcrPcoW1lyskZy9bu9w6M5TQEWBDSMG4brtWtjBj46P4hHf9yMVcWFjsgaN5LY1xzFmQuDTtP+S1vC+KM3P3RNNtpIAK1dA3g1Jdy8fWe6pmFPU4crDiOgCXxvY6VzVsO0JkCf/vkp6LoGwzCxoqgAl+NjqWtYAbC5wEZ8QhYuufSQlQL4ppSyW31RSmkKIf5Vfo5FCJlpco1BmKz3KJuwm0zMqd8NaAIBTSBpyjTxYQu3lo4+LCkIOKuMvPQOJ9DnyQ6TgJPx9emluCtc1UtAE677mh7FZpgSJ6P9LnfsnqUFrudSn8kmFNDw0JrSjA35mUQtG/EJWbhMKsiklH+R5b3fzuxxCCH5Yqbcl2zCzhZzdm7YMwdaXYJD/W7SlFhZFMKqksI08eEncgSAgJ65j0vF1k+JpOkrxpYU6FgWCmJrTYXrvt7faGO4DBJwvfZ0bYWr70t9Jhu73Nn4wgbf82UStWzEJ2ThwhwyQuYg+YiemCn3JRdhpwqOTy/FcbprAEnDxOWRhJOtFQpqeLq2wld8+IkcCfiKMcOQWFlUgL5rNxEK6PjKiqX49FIco+PpK48AYFVxCM1/usV3ejHTb5Ttd1N/D5vJBC9DWwkhXijICJmD5GPtzUy5L7kIO1VwJJKmU0JUCWoaDCnT1hIB/iInEyaAkUQSLz4axktbwth7vAM9Q6NIXhvzFXC9wwk8+uNmrCwqdHrM9jVHXeLX+xtl+93s36Ml2gdNCMSGRiGAjM/mfT72ihFCAEDMh0ix2tpaGYlEZvsYhNw2tu4/hUj3xDTg+tUlGctfKrMd6mrf/51IDH1Xx5A0Zdoib5XCoI4dmyvT9jjWVS2HgMCpTmsJ98XBG7gc9+8DU6+1rrzICX0NaAJSSmSrcHq/o55nOuw+1u4I6WzXMkyJvU0dvllkhJD5jxCiTUpZO5XvTC9xkBCSV6YbPTGVHLCpkGtkxp6mDhx4vxOX4wlHjN27LITCoP9/1ajRDurZD584D00AR79fjyM76rCquDDtu+vvL0ZQF65rfdYbd/WofWlZCOtXl+ChNaXw0zve79xq1ESusRXZssgIIQsTliwJmYNMt99rOr1JXlfNr4SXawn1ZLTfSbwHLGfsi5EEaleXwjBNx+2y3bPCoI66yuXYfawdb7Z0uc5+uKULMvVb+K0b+saaUpy9OJx2BnVy0+5RM0yJZw+dTiudFgZ1PLDS7ZDdSvmQpUhCyHShICNkDjLdfq/pCAKv2DrdNZC263EyoWeLukvDo2nXNyRgSomj3693PvvTd9vRmNr12No14NuEf23MwMEPuiDgL1CfOdCaVo604zGWFgTwx/VrHCGrawJvbX8Ye5s60BK1yqCGKbExXIYXt4TxRnN0RqImGFtBCJkuFGSEzEGm2ws2FUFg38PrTPmV8CYTeqqos5d123izvnRNQAiB4RvjGB038PmIe4m32nPmdcpUEVgfLsPZi0O+/WnXbyahCbh+s2wid6aiJhhbQQiZLhRkhMxBvK6VlBJCiEkF2lQEQaZA02WFQdy4aTjRFPVVlot0umsAn14axrLCAI5EYngnEsMffn0VNCHwNyfPO9eRAO5apOGmIREK6PjOhtVpwlB13Ew5IcK8TfaA2ylTn2tnQzVOdw34TnCqy74zMdsDEIQQokJBRsgcxFsibGzrcRylmYrB8GZ9BXWBgC7wRTzhiKR1q4pgSIlHftTs9H1dvzmRjP9Gcyf0VM+WTUATGEtKJE2JgGbFPniFTl3VcnwcG0bSlNAFUHN/CSTgiL83mqM43NLllCD9yqTeMmRvPIHPU2fXNeDS0Ch2H2vPKLTyES1CCCHThVOWhMxBvFOWAph0ei/XSUi/e1gREZYbZfdlmRL46MIQ9jVHnalJLxJwvR7UBe5ZWuC8Zpccs51HCIFvrFmODVVlaIn2Y19zFC9uCeP3VhY5k5G2U+fFdgSPfr8eJ/7sMex8rBorikIQEOiNJ7JOmk51kfdUf19CCJkKdMgImYN4e8EMKXH4xPmc0/FzcXzUe1waGkVvhpyvqegOw5QoL1mMoZSbB1gi78D7nTjdNeAs/G7tHHBEW9KU+OXZHgxdv4lE0sTZi0NobIth8NqY49Q9WF7s2w/nV3ZsifY7mWXZJk2nOgBBR40Qkk8oyAiZg3h7wQxTQhciY7O+YUo0RmIux6cl2gdDShxNTTM+VVOOXY9bmVdeIWNKif3vdfq6YDaagJMndtOQvin4pgRig9exY3Olq+SYSJr48PwgAGsv5LryIhQGdUcMidRn7Guo+yfN1JSmt+xomBLfOnQakVQP2We9cQhkF1re8NntmyrR2pXbRCTXHRFC8gkFGSFziEyN5pM16+9p6sCVqxO9XQFNQNc0l8ja/14ndGFdx+v2bN+0BvcsLcjokgHAvUUhDF2/idFxE9la30XqHhLAvuZomsiz9lsOo3jxIpTAWvAtIbGvOerrxmVzBCNKQ//ouImTnf14+/m6jJOm3ufesbkypw0IADPGCCH5hYKMkDnEdMtiJ6P9LuFzz9ICGKZ0vZY0pePqeN2e1q4BbKutcO5doAtIWE6YzcjouJMVlq2KWV5SCMOU2NlQjcZIzFfkXb9p4vrNBEIBDR+eH8B40sDiRbrjqNksKdDxvY2VGfdleuMy6qvKsorXW3G5mDFGCMknbOonZA4x1UZzG+8QwNO1FagPlyHgKfP99vIINvxVE4RSfrTdnp0N1dixuRLrV5fg+4+GseORKtc1H1hZ5Pzd65AtKdCd1yLdQ3j20GkAwLbaCuc7ftiLx8/G4mliLBTU8L2NlRlXC9WHyxAKWM9gp/dPJpKmu5IK4LojQkh+oUNGyBxiumWxTO6NTPWQjSTGcX3MwLXUP/Y6I1NK13LrbH1raqK9JgTO9QxjdNxEYVDHslDQEVSmBM5cGMTepg7nXGo/mUqmxePZnLFsz2yLpEylX7pchJC5ipBy7o9u19bWykgkMtvHIGTKTDV81DAl9jZ1+IqMW2Hr/lOIdA+5XltRFELLDx/zbfKf6jkNmd4DVnt/MerDd6Olow+XRxK4PJxwSp2rikNYUVQIXRNo6x6Esv4SAU3g5S3hW2qY332s3Sm/FgZ17NhcyQZ8QshtQwjRJqWsndJ3KMgIyR/5FAZ+Igqw+tBaOvqg6xrMVMyE3xSlKnx2H2vHgfc7Xcu739r+8KRi0D5DS0cfLscTTr9YQBP4+n0l+OTSROI+gLRrG6bEpteOu/rMVhaFcCIlFKeLV4CuX12Sc/M+IYTcKtMRZCxZEpJH8hmV4DcAIIG0dUj2FOWLj1Zh//udTlyFt8lfjZ2wS46vNFRndM68sRO6gGuPZc/QDdc57GurERa6JlzDBHb/22Slx8ngRCQhZL5BQUZIHplpYaAKlN7h0TSxJyXSRJA9Rdn4wgZIwHHKAprAQ2uWY/exdlwaHnV9x94FqQo879SnN3ZCjSVLmhIScLLGbPx+g2x9XX6iM5tIzOWahBAyF6EgI2SGuZXw0clQBUpAEwik9kjaQkcCjgC0ySYEPzo/gE8uWZ9X3S17VVFLFofPGzsBwDlPQBOAlFhXXoSkYSKgazBMiY1h92/gdcC8PXN+DmM2kWgzlSXrhBAyF6AgI2SGmU74aK6lOVWgJE2JlUUhrCopdIk9AaAl2gddSxdB3pVFke4hR1RJIO16qsArDOqoq7QctZPRfvTG3a7ayqICbK25D41tMfRdHcPlkTEMjyZdfXN+GwLslVCfXoq71ivtbKj2dRiziURCCJmv5E2QCSF+AeBfAbgipfxq6rVSAEcArAZwAcDTUsqhTNcgZDaZbv/SdPrGcg2E9QqUp2srXJ8zUqVCKSfcLpnh+4A7csLven47NQ+lzun9JVYWF+LVJ9fiVGfmXZKvv9vulEw/jg3jnqUFzlnsTDJrDdIIpLSepXhx0En09xOJ7A8jhNwJ5NMh+1sAPwPwX5TX/hxAk5Tyr4UQf576+w/zeAZCfMlFbE01Nd++5qXh0bRS4mTkKuJsgWQ7YC1Rq4Rnn189s4169ky5YJlyv7ylv637TznXlpjIESsM6tgYvhtA9r65o209LoduJDHu9JmpmWSj4wYa23ownFpSXhjUoQkwS4wQcseSN0EmpfxACLHa8/K/BvBo6s//GcB7oCAjs0AuYmuqTteepg4nOgIAlhYE8Mf1a7IKBiqXWKcAACAASURBVD8RFwpo0ITA1v2n0sSiLZC8fVRSSggh8GaLW4x5z+73/cKgju/Wr4EE8MyB1qxuoCq2QgEND1YUu8JlgewN9d4rLi0I4Jn19zlhs7+JDSGRlNAEcC2R9P392R9GCLkTud09ZF+SUl5O/flzAF/K9EEhxPMAngeA++677zYcjSwkchFbU52QVKMjAOD6zaTj6mRCFYa6sJwqAGjrHkLSlGjrHsIb70Xx+/eV4O+fewiLUquCvOdX3SQv3rMbpoQppVMK/ObXy9HaNYC2VD/ZZ71xl0CdbEjB+3yqYPI6kd/8ejl+/v7ElOe22vtcn3320GmnbDk6npyy00gIIfOVWWvql1JKIUTGVFop5UEABwErGPa2HYzMO6bT65WL2Mrk9GS6X324DGcvTjTJ29ERmZwcw5RojMQcEWVI4MZNw9XXJQGMGxIfnR/Eoz9uxoplIei6hv/1+YjrWqqbBAB3LdLw1VXFvpONe5o6nEb6wqCOMxcGHTEGAKPjJg63dDmlUG/f14uPhnMOWfU6kds3rcHLW8K+7pkdFGufI2lagwLeoQVCCLkTud2C7AshxAop5WUhxAoAV27z/ckdyFR7vYDccqoylcYy3W9nQzVOdw04Do8dHQH4i7g9TR24cnXMdW2/vY42vcMJ9A4nfN9bEtKRNKUjsrZvyrwRwOuufdYbT7vvtTEDBz/ogpQSf3Pygqvv62hbDK8+mVu50HsvOw8t09kmG1oghJA7ldstyP47gH8L4K9T//rfbvP9yR3IdKYab6UPKdP9dE3gre0PY29TR1rTvRrvYIu4k9F+1yqjJQU6DFNidNxEQBMQAk6qfjZCQQ3baiqgKYvAs7lJXtHzwMoi/CY27Cq32s/W2NaDa2NJ1+t918aw+1j7jDmRKmzYJ4QsVPIZe/EPsBr4y4QQPQD+ApYQe0cI8RyAbgBP5+v+ZOFwK2n4k5U7/d7Pdr9MTfcli4MuEXckEgMAV4/Ud+vXuETVjkeq8J2/+chx3DIR1DRIWAn2XoFpmBKvv9uOo209gJQoL10M03QHtiYNEw9WWOVNXRM41zOM0XEThUE9rQkfsETiwQ+6ZsyJVGHDPiFkoZLPKctvZXirIV/3JAuT6boq3l2M3mZ2wL88mcv9vC6aNupWVJ/HE04+mD2N+YPHLTForwb69psfoj5chm+sKcWhD7rSHCybq2NJHD7RBV0I3/LqG81RZ63R5RGrRFoY1LCuvBjneuKuxeevNFRjb1NHWu6Yn3uWbyeSEEIWEkLKud8vX1tbKyORyGwfg9xh7D7Wjr3HO1zu04qiEFYVFzpu2DMHWhHpnsguXr+6JKeG9t3H2l2xEroArt1Mn4AErCiIb6wpdRLq1fKm1Q9mOWdHIhfxxchYRrdsSYGO5zZWuly+rftPuc7v/byaRbaiKISWHz7mWuxtu2sCQHnJYnxyacI9UxP4CSGETCCEaJNS1k7lO1ydRBYU3uXcXnHTd3UMl+MJxw2bbjnU66IdicQyCjIJ4MPzgwDgW97cezwKISxXq+b+EpimRPsX13DV09tlN+KrLl99uMxXkGkCeGBlkROvYT/73qYO1/JwVRhurSnFhqrl7O8ihJA8oM32AQi5ndglyEj3EK5cHYOeapLShFU6tMWJXZLb2VCNHZsrsX51iVPSywW7VGdPFG6tKbcWbk/C6LgBCSs7zMYaCgCu3zTR1j2ETdV347sb1yAUSP+P7+i4gcMtXdh9rB2GKbGzoRrfWF2S9rn1q0vxd889hHuWFjivJU2Jk539zt/9JiTVZ8pljRQhhJDcoENGFhTe5dxLCnR85d5l2Bi2+qVsRygU0CAAbHztOASAp2rKfUNQc8FIiby7lxY4pb9zPVYivZeAJrCtphyaEGnrjYCJbLO3n69zHDhNTDTiA+lO2T88X4ef2o39ALbWVDj9attqK1ylVdUBvJVhCUIIIVODgowsKOrDZfg4Nuw4YYlxExvDZU5SvJ6actSEQFv3IOxe9v3vdfo2zeeCWvoLBTSUlxSi5K4CXE2MpwmuUFDDy49VY1HAmpzc09ThWg6uCaC+qiwtDX9vU4dLwHnjOF598st49ckvp50t24ACIygIIeT2wZIluSMwTIndx9qxdf8pp1znx86G6oxlOrXMaJgS6mCht5w3lXurrlwiaeKjC0O4HE9g9KaBlUUh12cT4ybeaI46Z31pSxXuWqRBE8CSggBeejTsK4wkgKWhoFMWDWgCl4ZGfc+jnndPUwdeaaj2LUN6y64sURJCSP6gQ0buCHJN65+sTGfjddICmshYspvs3mrpT8WQgJQSK4tC6I1bCfy28LMF0J/+wVfwp3/wlZyfPaAJLCnQkRg30RtPYF9zFI2RGLbVVjjTl9PZbEAIISS/0CEjdwR+6fmZ8GvUV12jn/zqd0iaJsqWLMKSAh0ri0J4MYMzlcu91fvZy8NVyksXwzafdA0Zna1cnl1dcWT/a288gYMfdGFvU8eUfytCCCG3Bzpk5I5gKg3ofmGlam7Yx7FhAHAS9NV9inY2V2MkhmtjSSwNBVFeUojCoObkc3nvrd7vJ7/6nbOoO6AJVJQuxrnYsBO/ISUcAZWrc+W3CskOfLVRe8rYrE8IIXMPCjJyR3CrDeh+LhMwIWTs9Px3IjFXyv61MQOX4wksKdCxYlkBKkrvwomOPpzqGoCZCnpVg1p3Pb7WGRyorypDS7TflYJv3zrXJHy/Z39xSxhvNEdxJBJD39UxR1jawovN+oQQMvdgUj9ZsKghsZom8EkqOsJujLeFzI7Nla7dlJlQv2czWaK96sx5r/XylvCkgizbLk57+lIVXmzMJ4SQ/MOkfnJHM9ki8Fw/Y6M2t4cCGh6sKIYpJeoqlwMQaO2acJwe/VFzVjEGuIWYTSanyz5nS0cf1pUX4bPeuCsCQwjAkNJZ+J2J199td0qgH8eGISXw6pPWvbhHkhBC5g8UZGTesKepAwfe70QiaeLsxSGc7hrAW9sfdgmWqUwQeuMoTCmdyIvX321Hz9Ao3onEcLKzH1euJlzfvWuRhptJE+P++74d1FJhJkeuMKhjXbm772vcsEJqJ8s++2Vbj6uB/2hbDD94vDpnUUoIIWRuQEFG5g0nlX4rUwJnLgy6di/an1EnCFuifZCp173iJFNz+56mDrzRHIWRMrzsSAqblUUhnPjhY9j02vG091RWLCvAfcsnespigzecni5NuPvFDNPEjs2VGcNdM+HXcMBYC0IImX8w9oLMG+rDZVCNHnuNkPcz9h7IUEBDbzyBPU0diHQPYe/xDjx76LQTJ5FpT+XJaL8jxrwENIGnayuga8JXDLkQAud64mi7OIyPzg/icjzhuFmmhPMshUEdG8N3Y9cTa/Hcxkrn/LlMQKo7MgOawNaacsZaEELIPIQOGZk37GyoxumuAZy5MAhTWmuGvIJFnSDUhMCZC4OOcPK6arom8EpDNSSAlmg/ZOr79eEyRLqHfM9wz9ICR7htrSl3+rf8uJZIZuw7U3vW1EnHqU5Aeqc2X2mohhAdjLUghJB5Bqcsybwi2+Sgt6H/ZLTfV1itX12Cxhc2AHBPOdoTka80VONbB1vx0QX3d70Tk/ZZWqJ90ITAxcEbGLh+E+OGne5v5YqpbpuuAV9aGsLTtRV5m3rkdCUhhMwu05mypCAjc4apTEj64RVX68qLELkw6BZEAqhdXQojlRHmFW1LCnQ8t7ESL6WyvFqifdA1DYYpsTE8ubjZuv9U2vWujxmQsEqU61eXpg0iEEIIubNg7AWZtximxLcOnUYkVY78rDc+pWZ0w5RojMRcvVOGaeJLy0KuxvvCRTrOpaYbP+sdwbryIhQGded718YM1/7HIzs2ZHTgdqbKieprdVXLXeXCZaGg06RvSsCU2WMsCCGELEwoyMisY4uxj84POq+Njps5J9UDlii6cnXM+XtAE9gYvtsV6BrQBMaSplNSzDTdqO5/VEWh3/Siev3PekewfdMa7Nhc6ZQLDWnFV7CfixBCSDYoyMiss6epA5ELg67XNIFJxYvqWPUOj7qa69XmewHg7TMX8cXImGsyUp1u9Evi98ZO+E0vSgnXa61dA2h8YYOrz8zbdE8IIYR4oSAj0ybXnq/JPncy2g/voOJdiwIZk+rt672j7GoMaAIBTTjrjrbWlLvuKYQ7pkITQMnioHMPe7ox0/5HwIrU+PRSHImkCQHg0vAopITrvtkWi99qjxwhhJA7FwoykkauK4py7fmaLKhUFTqAJZaujiUzJtWr17NJmhIrlhUAQkAAaO0awKeX4k6vWMBzflNaga/73+vEL9t6sK22AjsbqvFKQ3XahKKNN3ajd9jqTQtoAiuLJiYnM8HAVkIIIZlgMCxJwxYOke4hHPygC3ubOpz3DFNi97F2bHztOD46P6ikzZsZA0gnCyrd2VCNFx6pwvrVJVhZFHIl2PtdU72ejS4AIQS+GEmgN55AW/cQRlN7jUbHDSwJ6WmiDHD3i+1t6nAcLbvsqApRXRMwTJnm5iVNiVUlhWmfn+rvQAghZOFCQUbSyCYcbLF22bMyKFvPV13VcleafF2lf1mv8YUN2FZbMWlSvZrGb2M7XrZY8ibhb6upwMtbwli/ugQPrSlFYdD9b317zdLuY+3Yuv8Udh9rdxL9J7t3rs366ndvpcHfFsXZzkkIIWR+wZIlSSPTjkfA3536/9u79+Co7usO4N9zdxdJvCSBYweQCBYSbm3XToKIERgaLOOmbSbpjHGcuGmTxgEy2CbU7bTx9L9OJ06mTbBxHQp+NJk0BAcnmWY6aWos44dAOEiOi41d68VDQthGQhJgS6Dde/rH7r269+7dhx67V6y+n5kM0u7e3d/eeODM+Z3fOVZ/rQfqa1K2hrCoKpo6zqLu4S4IgDuXV8S7zSeiJ79O9dZ7NradRShkIBYzcVNFKY71DNonI/1Cko/OLYJCAFU0dfZBNf7drB5j3nqxkGH4bik6v1Pd0vnYuKYKhzqS+5M5+d2HsXbhT4Vbn0REhYeNYSlJuk7vzuarYUNw9ZwiV9d55/PWeCBn4OQVNgT3r6tOG1A439NiNX492j1or0VVXU1g/Trll0QMbF671A60nN+z0dMk1uro79fNP1MANJ5rsuVtPuucPEBERMFjY1iaFM6TgV5+WR7viUkrcBqOmnYBfCpRU12tJfwyS35ZOWcPsYMdvairmg9A8PRBZz+x5M9z9jfzfk8FfDODflu4mYKr8VyTrXQZTCIiujIxIKOseAOlvZvqfAvYDZ/TjOmEDXEFFH7bcc4AxOLsIeYMdESAxw+0pxz4na7WLdWW4ngCoFwGTZO19UlERFMHA7ICkss+V85A6c3Tgzjc2WfPg3R+TizmTksZEg/KrF5dTgLgk4vLsWVdtf2YX2Zp76Y6CGAP8e7qH4IAvn3KttbXYF9zl2tckv15jlo3p0zB5ngCoFwGTekymEREdGViQFZAJrvY29sJ328r8ljPeagqRAQH23sRChkoiRgYGjHtGjJTFXVV87Gvpdt1OlMBvHF6ED840G6v05lZChuC0/1D2NHQZvcI+9ITh/Hu4DAU8UzYq519uKVqPpo6+uzg8K7ayqQ6NlPVd4sVAB55vhU7X+xA1FS83jUAVcWDd1xnPz+eAChkCB6or4ECaGzvhQJsBEtERCkxICsgk1235AzwnJ3wrayX9Tn7Wrox8OFIxgBIRHzHEzW2n4Um1m+dYtzXEj8BafUIU1UcPn7ONe/SVOC3J/rRfLLfDg4FmevcvJ5t6bazd1FT8WxLtysgs4w1A8nTkERElC0GZAUk27qlbAMLZ4AXNRWRULwjfUX5TLxxegBDIyZKIiEI4Mqemaq+p/5SjSdytpt4vWsAV88pgiQ+ExgN+t47n7wNCSCpkayVzco2+Mk2Z7WjoQ27XurAcNTEa6f6cbizD3s2rrQbxqY7jDDZhf1ERFRYGJAVkGzrlrLN3HiL6UdiivcvXIKq4qaKMrsHV0wVT75y3BUIenuHmY56swfqa/Do863Y19INqOJYz6Ar8OsZHHZl5ASwtynTGW/x/J3LK+wty7Ah2LC80vd1B9t77fFOpgJHTpzDYw1t2LZ+WcbDCDwNSURE6TAgKyDZ1jplk7mJjwhSlM2MIHrRxEhsdEvvzPlLGBiK2r21YqYiJOIKBP3mTTqDPxGxtzn9RE3FwtJinB8eSephJgBmFYXw4eUYTAVCBnDNnMyzJFPZdvuypPX7WV19FV471e+aBmDdu3SHEXgakoiIMmFAVsBSbU1mk7nZ0dBmZ72c2SrL0EgMzzR32fVZ3kAwVe8wvwDGT0kkhA3LK7DzpQ7X44YAW2+rwZZEt/3G9tGO+c4M2ljqvbINZL3DxYsjhn3v/O4pT0MSEVG2GJAVsFStKqzC+abO1Jkbb/3YwtJiKNxbh2cvXMKjz7faJyxTBX0WZ/BXt3Q+fneq39VFvygs+HhlOWKmiZBhYF9Lt52Zs6xYMs8OcLatXwYFfLdfc1FQHzIEezauTJpiALA3GBERTQwDsgKWqmv+sZ7z2Ly2Ku24HW/Gx9oOXPPdF+weX1FT8fTB44iZiqERE693DWBfcxfuXF5hb3eWaRiV82bB1OSZj97erR+vLMeejSt9tzsBYHZRCD++95aU39HKwD2Q6EWWblt2rCcmM/UqYzaMiIgmggFZAXMGVQL3acTG9vQn/lK1jrirttLVCd9Z32UV5O98scP+vThsYPF8AOoeAN7U0ZdUpH/q3Ie4e1cTTjt6nllKIiF8/dYqzAgbKb+jlYHb0dCG9y9csl/jnQYAjL0lBVtYEBFRLjEguwKMtwO/M6g6PTCEnoHRthFHuwdQ9+3nUTFvJlTjgc1966rx+IF21+d4gw6/TvgCd7DlrDUbjpp4NdE77FjPoB3IrK6+Cq93DdivFcS3QM8MDie1oZg9I4QbFpX6Nlj1Cxzv3tXkWsPVc4qSthDH2pKCLSyIiCiXGJDlQaaAKtPzY8nO+L3XtvXLcOcPDroCssux+GnJM+fjmaRjPedxuLMPR7sH036OlSVzdsKfN3uG671TGRox7SawjW1n8YnKUrx15jyGRuKtJKwgyps5m1MSTrkuv61Cv+1WbwA71pYUbGFBRES5xIAsDzIFVJnmRGbbpmJHQxt+duQU3rtwCaYiMQYIePCOZbi15iN468yFlCcb42sbTPqcLeuq8eWnXsWx0wMonTkDC0pLsMpzKKCxvdcVkIUNIGomf4YhcDWBDScaqnoDsLAhUFXENH7NxeHYmLJT2RTYj7UIn0X7RESUS6Kaqd1m8Gpra7W5uTnoZYzbhp2H0Hyy3/59xZJyV0G993lrNFFJJITNa6tcJwmtx7wByfb9rb6F8JGQYMunq3Ffok3EM81dvk1WSyIh3FRRameiQgKUzAjhUtRMOulYEglh45pr7dOVhiF4o3u0c7/zfZzfacWSeYiZ6vquXrOLQvja6mtxuPMcmk+eSwwmBwCxO/tvXltl9zrLxSB1IiKiiRCRFlWtHcs1gWTIROQEgAsAYgCiY130lSbTdpfzee+cyHQNRlMN/3YaiSl2vdRhZ902LK+AKvCL17qhqq4TkFs8QZu3IavFO7+yKCSYP6cIoaER3LCwFD/8q09h10sdrh5h1gnLHQ1truHhzgxZ2BB8/dZ4sLlh5yH7PkRNYGFpERaVl/g2nmWRPRERXemC3LJcp6q9AX5+3mTa7nI+b4jgqCPblK7BaKrh315+LS8OPVRvP28Fdvc8cRirq6/CorISnBlMXRPmnV95KaZ4d3AYpgItJ/tR/70XcVdtJZ7ZvCopa+X8rnVV82EmgkMA2LB8tNO+Xx1YqsazLLInIqIrHWvI8iBTjyrn8zFTfRuP+vE2b51dFMJ118zBmfPDSUX2fgO4Ld5s000VpUnBnbWFqQBuXFiKFUvm4anG40lZPav1xe6XO11Zq3R9vP72j65L+m5b62ugqtjX0g0BEFONj2hKXMMieyIiKiRBBWQK4DkRUQC7VHV3QOuYcsbSYNTbOmJ4xMSamo8kFdlHQpKYTeke92NpbDvryjad7LuIb/zhUlfmSjE6QPxo9yBuuXYeNq+tSsrqWYZGYnilLX6i0ltnls0WY8gQ17zLJ185jpCIfQ2L7ImIqJAEUtQvIotU9bSIXA1gP4AHVPVlz2s2AdgEAIsXL15+8uTJvK9zqouZ6uqcDwALSosBxHt6RU1NOrG4Ysk87Nm40rWV+IVdTfhtoleYZWFpMe6qrbSL5dMdTLCyek82drrqzmY7BoA7s2je6/2+146GNjzleb8FpcVYVFbCIn4iIprSrpiiflU9nfjzfRH5JYBPAXjZ85rdAHYD8VOWeV9kDmTT4DXbJrAxU/HI8604PzxiPxYyRgMxQTwzVhwO4cKlKIB4QGSqJr2f6VN31jM4jF0vtduHAQxDUBIxXLVto58bz1y90nYWLacG7MedwZQzKMu0xeg3OilsiN04lkX8RERUaPIekInILACGql5I/HwHgH/M9zqCkM3JwGxPD+5oaMPOFztcdV6q8VorIL4nPBJTqMbsejBnIGQFdM+2dOOCI6hzGo6qfRigOGzg5soymKoptwidvc68GTFBPDuX7nqLszYOiGfa5hZH7Ewgi/iJiKjQBJEhuwbAL0XE+vw9qvqbANaRd6kGYTszYulOD1rZs8a2s3jnvYtJJyp9El2ImgpD4tmyP1hUii3rqgH4B3R+rKeHoyZM1bTbjNZA8XJEUFE+E0e7+zEc1ZRbpal4C/a/fmtyLzZnhm0sWUX2LiMioqko7wGZqnYCuDnfnzsVpBqE7T3hWBIJuV5jd+Fv7rK3JNOJhASqo6OITAXMmOK1U/34wYF2bFu/DAfbe9O+j0i8Zqv/g8u+25SAO8DxNofdsHweVi2dnzSc3O9ab3CUqmA/VRH/WLKKmV7HoI2IiILAthd5lGoQtjMjFjNN+/SiXxPUTIojBjatqYIhklRkHzXVzrh5T2haFpQWoaJ8JlYvHW0Um+oko3Nd3oa2TZ192PeNVSm3FdMFR6lOmqY6fZptT7JsXseGs0REFAQj6AVMF/Z2Y3svVjkyRqurr0JJJAQgXux+a/VHsG39MjuYCRmSVFPlJIj3CAPiRfM3V5Thm7fHA5d7b61C2JHdCRtiZ7m21tdgy6eXIhJyZ38Wz5tlf/aMsIFt65dh76Y6KIC7dzVh+/5WxBKRl3NdVtG+9T0y9QXzC47Gy3sPU3123dL59v0IG4K6quTXTea6iIiIssUMWZ6kyrxk00/LudXpFQ6JPWvSe4rSaq76bEtyJ/yQIXjwjutwqKPP1c4i5rONmWrtznVlU/Sf6jtNtLHr+HuSJX9XNpwlIqIgMCDLE2/mZe+RU3imuQsC4M7lFa7O9d46pvvWVfuOVgobgqKwAVUz6RQlMBp0PXhHcid8izcAubU6u6xRqmAy23qryWzsmm0z3aaOPnuLNmoqmjr7crouIiKibAXSGHasamtrtbm5OehlTMj2/a2uuZPOodrAaGuHO5dXAACeeLkTw9F45/s5RWF8dfUSbLs9HnA81tCGZxwF/tZYI+v6bbcvyzow8hvV5L3Wu/ar5xS5msZeKZzfoyQSwua1VawPIyKiSTeexrAMyPIkZioefb4V+1q60Xvxkr3N6GdOUdhu5uo0a4YBEcGc4ggAuAaAO5uuTnagYQVtziDwSgxosgk+iYiIJmo8ARmL+vPEOZsxXTAGwDcYA4APLpu4eCmGM4PDrmBMJHl4eCYxU7F9fys27DzkKtT3e9zaElxUVmJv+V2JBe/W93AemCAiIpoKWEOWA6l6WXlPS1rxgF87MG+n+1QMAT6aoV+Yn1SF+unaPrDgnYiIKDeYIcuBHQ1t2PVSB5pP9uOxF9pwzxOHETPV1Z4hbAgMEZgab1vhTdZcM7cIoSwSOKbGB4FvXrsUK5aUY/PaqqRCdG/W63LUxL7mLt/2DunaPmytr8HmtVUpP4eIiIjGhxmyHDjY3msX5JsKvHr8HNZ89wXcubwCG9dUoamzF6f7h+zZjDGN14d9cNm032NhaTHurl2Mgx29qKuaD1OBn7d0ofeDy64tz7Ahdu+ybJuwHu7sw/sXLrnew8p2pcuCZXuakYiIiMaGGbIcWF19VVLGq2dwGI8faMerx/uwd1Md7qqtRJEjBSYiKInE/+8oiYSwpuZquymriOBwZx++sGIxbq4oc73v1XOKMmaqvFmvYz2Drg79zvdgFoyIiCj/mCHLga31NTjc2YcjJ8656sBMBY6cOIfHGtqwtb4G+1q60DMQz5JdvBTDorJiLCwrcfW/yjTr8gu1lb7F6ZejJr781Kt4q2cQc0siKIkYdo3ZDQtLcbR70Pc9xpoF4+xHIiKiiWNAlgMhQ7Bn40q7VYTzRKSpwJONnVAA54dGXNcNDo3g4LfqXY95s1t+sy6B5MDoUEcvjpyId+D3BnuZZlSOBWc/EhERTRy3LHPEyjQ1/v1tuOXaea4tzIuXYtj9cifmlkRc19ywsDTpfbKZdQmMBkbNJ/ux++VOvN414HqfwaEReyblPU8chgLYu6luwu0fOPuRiIho4pghy5GYqXjk+db4HElV1H6sHG+dOY+Ll0aDl+vnzkZl+Uwc6xnE9QvmYvnHylH3cIM9TumB22pgqqJsZgTliLhmUXp5A6PZRSGMxEZbbNywsDQn2Sy2wiAiIpo4BmQ5sqOhDTtf7LCL589evIzlHyvH/3YNYDhqwhAgHAphz8aVCBmC7ftb8fiBdvv1O1/swG+Pn3PVehkCVzbLuU1pGOKqE/vqqiU4cqIfx3oGccPCUvz43ltwzxOHfWdSTgRnPxIREU0cA7IcOdje6zrJGDUVMdPEzZVldrH/0e4BPNbQhm3rl/m+/ljPYNoAypnxKg4buLmyDKZqyrFAuchmsRUGERHRxLGGLEdWV1+FsCMgsvqFUskVVwAAB+tJREFUxUx1jDky7Zorv9ffsLDUVT/mDaCc25TDUROmatqxQGxpQURENDUxQ5YjW+troKrxGjLArv9SwDdL5ff6+29LfxpyrBmvkCH2Ghrbe6GJz2WbCiIiomCJahYDEwNWW1urzc3NQS9jXLztKO7zaTkx3oAoZioea2gb03tt399qb3OWRELYvLaK241ERESTSERaVLV2LNcwQ5ZjqU42pguCsm22Op76Lb82Fdb1bPJKREQUDAZkOZYuAErFGcS9eXoQhzv77OHkEw2S0m1zsskrERFRMBiQ5dh4TjZ6i/WtU5mTESSla1MxnuCRiIiIJo6nLHMofqIy3th1YWkxNq6pwpZ11fjec++g7uEGrHq4Ad977h3ETHcdn7M7vyFwnMqcWCd8a0uysb0Xq3xqzrxTAdjklYiIKD+YIcuhHQ1tePKV467Gro8faHc1jN35YgdCIq5MlDOLZYjgaPeA3fB1IkFSpi1JNnklIiIKBgOyHPLbAlRFUgNY79ags1jf7yTlZK4n1ecSERFR/nDLMoecW4DFYQOGCE4PDMFZkh82xM56xUzF9v2t2LDzELbvb0XMVDtIsgaD372ryX5uIuvhliQREdHUwQxZDqXaegwJUDIjhLnF7oHh6bYUJ+MEJLckiYiIpiZmyHLIym7t+8YqxEzF0IgJAIgp8PsL5uLQQ/V48I7RMUd+W4qWdM+NZz2pxisRERFR/jEgy5NstgvTvYbbjURERIWLW5Z5ks12YbrXcLuRiIiocHGWJREREdEkGs8sS25ZEhEREQWMARkRERFRwBiQEREREQWMARkRERFRwBiQEREREQWMARkRERFRwBiQEREREQWMARkRERFRwBiQEREREQWMARkRERFRwBiQEREREQUskIBMRD4jIu+ISLuIfCuINRARERFNFXkPyEQkBOBxAH8M4HoAXxKR6/O9DiIiIqKpIogM2acAtKtqp6peBrAXwOcDWAcRERHRlBBEQLYIQJfj9+7EY0RERETTUjjoBaQiIpsAbEr8eklE3gxyPdPQVQB6g17ENMN7nn+85/nHe55/vOf5d91YLwgiIDsNoNLxe0XiMRdV3Q1gNwCISLOq1uZneQTwngeB9zz/eM/zj/c8/3jP809Emsd6TRBblkcA1IjItSIyA8AXAfwqgHUQERERTQl5z5CpalRE7gfwPwBCAJ5W1WP5XgcRERHRVBFIDZmq/hrAr8dwye5crYVS4j3PP97z/OM9zz/e8/zjPc+/Md9zUdVcLISIiIiIssTRSUREREQBY0BGREREFLApHZBx5mX+iUiliBwQkbdE5JiIfDPoNU0HIhISkd+JyH8FvZbpQkTKRORZEfk/EXlbROqCXlMhE5G/Tvyd8qaI/FREioNeUyESkadF5H1n704RmSci+0WkLfFneZBrLDQp7vk/J/5uOSoivxSRskzvM2UDMs68DEwUwN+o6vUAVgK4j/c9L74J4O2gFzHNPArgN6r6ewBuBu9/zojIIgBbAdSq6o2In7D/YrCrKlg/BPAZz2PfAtCgqjUAGhK/0+T5IZLv+X4AN6rqTQBaATyU6U2mbEAGzrwMhKqeUdXXEj9fQPwfKY62yiERqQDwpwCeDHot04WIlAJYC+ApAFDVy6o6EOyqCl4YQImIhAHMBNAT8HoKkqq+DOCc5+HPA/hR4ucfAfizvC6qwPndc1V9TlWjiV8PI94EP62pHJBx5mXARGQJgE8AeDXYlRS8RwD8HQAz6IVMI9cCOAvg3xNbxU+KyKygF1WoVPU0gH8BcArAGQCDqvpcsKuaVq5R1TOJn98FcE2Qi5mGvgbgvzO9aCoHZBQgEZkN4OcAtqnq+aDXU6hE5LMA3lfVlqDXMs2EAXwSwE5V/QSAD8BtnJxJ1Cx9HvFAeCGAWSLy5WBXNT1pvNcV+13liYj8A+KlQD/J9NqpHJBlNfOSJp+IRBAPxn6iqr8Iej0FbjWAz4nICcS35W8Tkf8IdknTQjeAblW1sr/PIh6gUW7cDuC4qp5V1REAvwCwKuA1TSfvicgCAEj8+X7A65kWROSrAD4L4M81i6avUzkg48zLAIiIIF5X87aqfj/o9RQ6VX1IVStUdQni/42/oKrMHOSYqr4LoEtErks8VA/grQCXVOhOAVgpIjMTf8fUg4co8ulXAL6S+PkrAP4zwLVMCyLyGcRLUT6nqh9mc82UDcgSxXDWzMu3AfyMMy/zYjWAv0A8U/N64n9/EvSiiHLgAQA/EZGjAD4O4NsBr6dgJTKRzwJ4DcAbiP/bw3E+OSAiPwXQBOA6EekWkXsBfAfAehFpQzxb+Z0g11hoUtzzfwUwB8D+xL+j/5bxfTg6iYiIiChYUzZDRkRERDRdMCAjIiIiChgDMiIiIqKAMSAjIiIiChgDMiIiIqKAMSAjIiIiChgDMiIiIqKAMSAjomlFRFaIyFERKRaRWSJyTERuDHpdRDS9sTEsEU07IvJPAIoBlCA+0/LhgJdERNMcAzIimnYS83GPABgGsEpVYwEviYimOW5ZEtF0NB/AbMRnzRUHvBYiImbIiGj6EZFfAdgL4FoAC1T1/oCXRETTXDjoBRAR5ZOI/CWAEVXdIyIhAIdE5DZVfSHotRHR9MUMGREREVHAWENGREREFDAGZEREREQBY0BGREREFDAGZEREREQBY0BGREREFDAGZEREREQBY0BGREREFLD/ByIoqzAaGt5FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "x = np.arange(1, 11, 0.01)\n",
    "noise = np.random.randn(1000)\n",
    "y = 2 * x + 1 + noise\n",
    "plt.figure(figsize=[10, 5])\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim((0, 12))\n",
    "plt.ylim((0, 26))\n",
    "plt.scatter(x, y, s=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.2 多元线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$f(x_i)=w_{1}x_{i1} + w_{2}x_{i2} + ... + w_{d}x_{id} + b$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量形式：$f(x_i) = W^T x_i + b$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e.g: $f_{颜值}(x) = 0.3·x_{身高} + 0.7·x_{五官} + 0.5·x_{胸围} + 0.6·x_{衣品} +\\ ...\\ + 0.6·x_{气质} + 5$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1 最小二乘法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "找出最佳参数，使得数据与实际数据之间误差的平方和为最小。  \n",
    "即 **残差平方和的总和最小化**  \n",
    "**几何意义**： 使预测值和真实值的欧式距离之和最小"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.1.1 问题定义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "给定数据集$D=\\left\\{\\left(\\boldsymbol{x}_{1}, y_{1}\\right),\\left(\\boldsymbol{x}_{2}, y_{2}\\right), \\ldots,\\left(\\boldsymbol{x}_{m}, y_{m}\\right)\\right\\}$，其中$\\boldsymbol{x}_{i}=\\left(x_{i 1}\\right. \\left.x_{i 2} ; \\ldots ; x_{i d}\\right), y_{i} \\in \\mathbb{R}$。  \n",
    "线性回归试图学得： $f\\left(x_{i}\\right)=w x_{i}+b$，使得 $f\\left(x_{i}\\right) \\simeq y_{i}$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.1.2 基本求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "采用SSE（Sum of Squares Error）计算预测值 $f(x_i)$ 和 $y_i$ 之间的误差 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**一、 目标函数（损失函数Loss function、代价函数cost function）**:  \n",
    "> $\\begin{aligned} J(w, b) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} \\end{aligned}$   \n",
    "> 求使得目标函数 $J(w, b)$ 取最小值的 $w$ 、 $b$   \n",
    "> $\\begin{aligned} (w^{*}, b^{*}) &=\\underset{(w, b)}{\\arg \\min } \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} \\\\ &=\\underset{(w, b)}{\\arg \\min } \\sum_{i=1}^{m}\\left(y_{i} - w x_{i}-b\\right)^{2} \\end{aligned}$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**二、 对 $w$ 求偏导** ：  \n",
    "$\\begin{aligned} \\frac{\\partial J({w}, {b})}{\\partial {w}} &=\\frac{\\partial}{\\partial {w}}\\left[\\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right)^{2}\\right] \\\\ &=\\sum_{{i}=1}^{{m}} \\frac{\\partial}{\\partial {w}}\\left[\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right)^{2}\\right] \\\\ &=\\sum_{{i}=1}^{{m}}\\left[2 \\cdot\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right) \\cdot\\left(-{x}_{{i}}\\right)\\right] \\\\ &=\\sum_{{i}=1}^{{m}}\\left[2 \\cdot\\left({w} {x}_{{i}}^{2}-{y}_{{i}} {x}_{{i}}+{b} {x}_{{i}}\\right)\\right] \\\\ &=2 \\cdot\\left({w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}+{b} \\sum_{{i}=1}^{{m}} {x}_{{i}}\\right) \\\\ &=2\\left({w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{b}\\right) {x}_{{i}}\\right)  \\end{aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**三、 对 $b$ 求偏导** ：  \n",
    "$\\begin{aligned} \\frac{\\partial {J}{({w}, {b})}}{\\partial {b}} &=\\frac{\\partial}{\\partial {b}}\\left[\\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right)^{2}\\right] \\\\ &=\\sum_{{i}=1}^{{m}} \\frac{\\partial}{\\partial {b}}\\left[\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right)^{2}\\right] \\\\ &=\\sum_{{i}=1}^{{m}}\\left[2 \\cdot\\left({y}_{{i}}-{w} {x}_{{i}}-{b}\\right) \\cdot(-1)\\right] \\\\ &=\\sum_{{i}=1}^{{m}}\\left[2 \\cdot\\left({b}-{y}_{{i}}+{w} {x}_{{i}}\\right)\\right] \\\\ &=2 \\cdot\\left[\\sum_{{i}=1}^{{m}} {b}-\\sum_{{i}=1}^{{m}} {y}_{{i}}+\\sum_{{i}=1}^{{m}} {w} {x}_{{i}}\\right] \\\\ &=2\\left({mb}-\\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}\\right)\\right) \\end{aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**四、 令$\\begin{aligned} \\frac{\\partial J(w, b)}{\\partial w} = 0 \\end{aligned}$** ：  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "source": [
    "$\\begin{aligned} 0={w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{b}\\right) {x}_{{i}} \\end{aligned}$  \n",
    "$\\begin{aligned} {w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\\sum_{{i}=1}^{{m}} {bx}_{{i}} \\end{aligned}$   \n",
    "\n",
    "\n",
    "> 令$\\begin{aligned} \\frac{\\partial J(w, b)}{\\partial b}=0 \\end{aligned}$ 得：  \n",
    "> $\\begin{aligned} {b}=\\frac{1}{{m}} \\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}\\right) \\end{aligned}$  \n",
    "> 因为 $\\begin{aligned} \\frac{1}{{m}} \\sum_{{i}=1}^{{m}} {y}_{{i}}=\\overline{{y}}, \\frac{1}{{m}} \\sum_{{i}=1}^{{m}} {x}_{{i}}=\\overline{{x}} \\end{aligned}$  \n",
    "> 所以 $\\begin{aligned} {b}=\\overline{{y}}-{w} \\overline{{x}} \\end{aligned}$， 代入上式得：  \n",
    "\n",
    "$\\begin{aligned} {w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\\sum_{{i}=1}^{{m}}(\\overline{{y}}-{w} \\overline{{x}}) {x}_{{i}} \\end{aligned}$  \n",
    "$\\begin{aligned} {w} \\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\\overline{{y}} \\sum_{{i}=1}^{{m}} {x}_{{i}}+{w} \\overline{{x}} \\sum_{{i}=1}^{{m}} {x}_{{i}} \\end{aligned}$  \n",
    "$\\begin{aligned} {w}\\left(\\sum_{{i}=1} {x}_{{i}}^{2}-\\overline{{x}} \\sum_{{i}=1}^{{m}} {x}_{{i}}\\right)=\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\\overline{{y}} \\sum_{{i}=1}^{{m}} {x}_{{i}} \\end{aligned}$  \n",
    "$\\begin{aligned} {w}=\\frac{\\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\\overline{{y}} \\sum_{{i}=1}^{{m}} {x}_{{i}}}{\\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\overline{{x}} \\sum_{{i}=1}^{{m}} {x}_{{i}}} \\end{aligned}$    \n",
    "\n",
    "> 由于 $\\begin{aligned}\n",
    "\\bar{y} \\sum_{i=1}^{m} x_{i}=\\frac{1}{m} \\sum_{i=1}^{m} y_{i} \\sum_{i=1}^{m} x_{i}=\\bar{x} \\sum_{i=1}^{m} y_{i}\n",
    "\\end{aligned}$，  \n",
    "> $\\begin{aligned}\n",
    "\\overline{{x}} \\sum_{{i}=1}^{{m}} {x}_{{i}}=\\frac{1}{{m}} \\sum_{{i}=1}^{{m}} {x}_{{i}} \\sum_{{i}=1}^{{m}} {x}_{{i}}=\\frac{1}{{m}}\\left(\\sum_{{i}=1}^{{m}} {x}_{{i}}\\right)^{2}\n",
    "\\end{aligned}$  \n",
    "代入上式得  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "求得 $w$ 和 $b$ 最优解的**闭式解**（closed-form）：  \n",
    "\n",
    "$\\begin{aligned}\n",
    "{w^*}=\\frac{\\sum_{{i}=1}^{{m}} {y}_{{i}}\\left({x}_{{i}}-\\overline{{x}}\\right)}{\\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\frac{1}{{m}}\\left(\\sum_{{i}=1}^{{m}} {x}_{{i}}\\right)^{2}}\n",
    "\\end{aligned}$  \n",
    "\n",
    "$\\begin{aligned} {b^*}=\\frac{1}{{m}} \\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}\\right) \\end{aligned}$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.1.3 向量求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将 $b$ 纳入 $w$，记作$\\hat{\\boldsymbol{w}}$ :   \n",
    "\n",
    "$\\begin{aligned}\n",
    "\\hat{\\boldsymbol{w}}=(\\boldsymbol{w } ; b)\n",
    "\\end{aligned}$  \n",
    "\n",
    "即 $f(x_i) = x_{i1}w_1 + x_{i2}w_2 +\\ ...\\ + x_{id}w_d + 1 · w_{d+1}$   \n",
    "\n",
    "\n",
    "将 $x$ 也写成向量形式，并扩展一列填充为 $1$ ，记作 $X$ : \n",
    "  \n",
    "$\\begin{equation}\n",
    "\\mathbf{X}=\\left(\\begin{array}{ccccc}\n",
    "x_{11} & x_{12} & \\ldots & x_{1 d} & 1 \\\\\n",
    "x_{21} & x_{22} & \\ldots & x_{2 d} & 1 \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots & \\vdots \\\\\n",
    "x_{m 1} & x_{m 2} & \\ldots & x_{m d} & 1\n",
    "\\end{array}\\right)=\\left(\\begin{array}{cc}\n",
    "x_{1}^{\\mathrm{T}} & 1 \\\\\n",
    "x_{2}^{\\mathrm{T}} & 1 \\\\\n",
    "\\vdots & \\vdots \\\\\n",
    "x_{m}^{\\mathrm{T}} & 1\n",
    "\\end{array}\\right)\n",
    "\\end{equation}$  \n",
    "\n",
    "$\\begin{aligned}\n",
    "\\hat{\\boldsymbol{w}}^{*}=\\underset{\\hat{w}}{\\arg \\min }(\\boldsymbol{y}-\\mathbf{X} \\hat{\\boldsymbol{w}})^{\\mathrm{T}}(\\boldsymbol{y}-\\mathbf{X} \\hat{\\boldsymbol{w}})\n",
    "\\end{aligned}$  \n",
    "\n",
    "$\\begin{aligned} J(\\hat{\\boldsymbol{w}}) = (\\boldsymbol{y}-\\mathbf{X} \\hat{\\boldsymbol{w}})^{\\mathrm{T}}(\\boldsymbol{y}-\\mathbf{X} \\hat{\\boldsymbol{w}}) \\end{aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上式展开，得：  \n",
    "$\\begin{aligned}\n",
    "J({\\hat{w}})=\\boldsymbol{y}^{\\mathrm{T}} \\boldsymbol{y}-\\boldsymbol{y}^{\\mathrm{T}} \\mathbf{X} \\hat{\\boldsymbol{w}}-\\hat{\\boldsymbol{w}}^{\\mathrm{T}} \\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}+\\hat{\\boldsymbol{w}}^{\\mathrm{T}} \\mathbf{X}^{\\mathrm{T}} \\mathbf{X} \\hat{\\boldsymbol{w}}\n",
    "\\end{aligned}$  \n",
    "\n",
    "对 $\\hat{w}$ 求导，得：  \n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{J}({\\hat{\\boldsymbol{w}})}}{\\partial \\hat{\\boldsymbol{w}}}=\\frac{\\partial \\boldsymbol{y}^{\\mathrm{T}} \\boldsymbol{y}}{\\partial \\hat{\\boldsymbol{w}}}-\\frac{\\partial \\boldsymbol{y}^{\\mathrm{T}} \\mathbf{X} \\hat{\\boldsymbol{w}}}{\\partial \\hat{\\boldsymbol{w}}}-\\frac{\\partial \\hat{\\boldsymbol{w}}^{\\mathrm{T}} \\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}}{\\partial \\hat{\\boldsymbol{w}}}+\\frac{\\partial \\hat{\\boldsymbol{w}}^{\\mathrm{T}} \\mathbf{X}^{\\mathrm{T}} \\mathbf{X} \\hat{\\boldsymbol{w}}}{\\partial \\hat{\\boldsymbol{w}}}\n",
    "\\end{aligned}$  \n",
    "\n",
    "由矩阵微分公式：  \n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial \\boldsymbol{a}^{\\mathrm{T}} \\boldsymbol{x}}{\\partial \\boldsymbol{x}}=\\frac{\\partial \\boldsymbol{x}^{\\mathrm{T}} \\boldsymbol{a}}{\\partial \\boldsymbol{x}}=\\boldsymbol{a}, \\frac{\\partial \\boldsymbol{x}^{\\mathrm{T}} \\mathbf{A} \\boldsymbol{x}}{\\partial \\boldsymbol{x}}=\\left(\\mathbf{A}+\\mathbf{A}^{\\mathrm{T}}\\right) \\boldsymbol{x}\n",
    "\\end{aligned}$   \n",
    "可得：   \n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{J}({\\hat{\\boldsymbol{w}})}}{\\partial \\hat{\\boldsymbol{w}}}=0-\\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}-\\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}+\\left(\\mathbf{X}^{\\mathrm{T}} \\mathbf{X}+\\mathbf{X}^{\\mathrm{T}} \\mathbf{X}\\right) \\hat{\\boldsymbol{w}}\n",
    "\\end{aligned}$  \n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial \\mathrm{J}({\\hat{\\boldsymbol{w}})}}{\\partial \\hat{\\boldsymbol{w}}}=2 \\mathbf{X}^{\\mathrm{T}}(\\mathbf{X} \\hat{\\boldsymbol{w}}-\\boldsymbol{y})\n",
    "\\end{aligned}$  \n",
    "\n",
    "令$\\begin{aligned} \\frac{\\partial \\mathrm{J}({\\hat{\\boldsymbol{w}})}}{\\partial \\hat{\\boldsymbol{w}}} = 0 \\end{aligned}$   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解得 $\\begin{aligned}\n",
    "\\hat{\\boldsymbol{w}}^{*}=\\left(\\mathbf{X}^{\\mathrm{T}} \\mathbf{X}\\right)^{-1} \\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}\n",
    "\\end{aligned}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.2 梯度下降法Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.2.1 梯度和参数更新  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**梯度**：关于目标函数 $J(w, b)$ 的偏导数组成的向量，即$\\begin{aligned}(\\frac{\\partial J({w}, {b})}{\\partial {w}}, \\frac{\\partial J({w}, {b})}{\\partial {b}} )\\end{aligned}$  \n",
    "$\\begin{aligned}\n",
    "J(w, b) = \\sum_{{i}=1}^{{m}}(wx_i + b - y_i)^2\n",
    "\\end{aligned}$  \n",
    "\n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial J({w}, {b})}{\\partial {w}} = 2\\sum_{{i}=1}^{m}(wx_i + b - y_i)x_i\n",
    "\\end{aligned}$  \n",
    "\n",
    "$\\begin{aligned}\n",
    "\\frac{\\partial J({w}, {b})}{\\partial {b}} = 2\\sum_{{i}=1}^{m}(wx_i + b - y_i)\n",
    "\\end{aligned}$    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用梯度去**更新参数** $w$ 和 $b$   \n",
    "  \n",
    "$\\begin{aligned}\n",
    "w \\leftarrow w-\\eta \\frac{\\partial J(w, b)}{\\partial w}\n",
    "\\end{aligned}$\n",
    "\n",
    "$\\begin{aligned}\n",
    "b \\leftarrow b-\\eta \\frac{\\partial J(w, b)}{\\partial b}\n",
    "\\end{aligned}$  \n",
    "  \n",
    "其中 $\\eta$ 是学习率（learning rate），需要手动设置，通常取 $0.01, 0.001, 0.0001, ...$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./imgs/2.gradient_descent1.png\" width=\"600px\">   \n",
    "  \n",
    "<img src=\"./imgs/2.gradient_descent2.png\" width=\"300px\"> <img src=\"./imgs/2.gradient_descent3.png\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.2.2 问题  \n",
    "\n",
    "* 为什么是减去梯度（乘以学习率），不是加？  \n",
    "> $ J(w) = 3(w - 3)^2 +1 = 3w^2 - 18w + 28$  \n",
    ">  \n",
    "> $ \\begin{aligned}\n",
    "\\frac{\\partial J({w})}{\\partial {w}} = 6w - 18\n",
    "\\end{aligned} $   \n",
    "> \n",
    "> 已知在 $w=3$ 时 $J(w)$ 取得最小值 $1$  \n",
    ">  \n",
    "> 点 $(0, 28)$ 处的导数为 $-18$  \n",
    ">  \n",
    "> $w$ 要从 $0$ 增大到 $3$ ，就得加一个正数（减一个负数），即 $0 - (-18)*\\eta$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAGtCAYAAACr05cdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4XWd9L/rvb08at+Z5sCVbsgbPjmI7TuLEcUICISRQhvQECJSSHqBATzkFWnrLbe9zb4G2h+lwKGFqeuASIAkJSYAMzujEOJ5ly5ItT5rncWvew3v+2Hs7iiNr3Hu/a639/TxPHkuyZH/B0nevd1jvEqUUiIhoZWy6AxARWQHLlIgoAlimREQRwDIlIooAlikRUQSwTImIIiBqZSoiPxGRXhE5NetjWSLynIg0h37NDH1cROQ7InJOROpFZFu0chERRUM0r0z/A8AdV3zsywD2KaUqAewLvQ8A7wRQGfrvAQDfj2IuIqKIi1qZKqVeATB4xYfvBvBQ6O2HANwz6+P/qYL+CCBDRAqjlY2IKNIcMf778pVSXaG3uwHkh94uBtA26/PaQx/rwhVE5AEEr16RkpJyTXV19eXfm/EFcKbHg5LMJGQmu6IQn4h0aBmYwLTPj3X57pj8fUeOHOlXSuUu5WtiXaaXKaWUiCz5Xlal1IMAHgSAuro6dfjw4cu/5w8o1PzDH/CxXWX4u3fVRC4sEWl187+8iPVF6fjefbFZThGRlqV+TaxX83vCw/fQr72hj3cAKJ31eSWhjy2J3SZYm5uK5h7PioMSkTFMef1oHZxARV6q7ijzinWZ/hbA/aG37wfwxKyPfzS0qr8TwMis6YAlqcxLRXPv2MqTEpEhXOgbR0ABlflxWqYi8gsABwBUiUi7iHwCwNcA3CYizQBuDb0PAL8DcAHAOQA/BPDp5f69lXmpaB+axMSMb0X5icgYmnuDI83KvNjMly5X1OZMlVJ/epXf2jvH5yoAn4nE3xt+9TrXO4ZNJRmR+COJSKPmnjHYbYLynBTdUeZluTugKkKvXs09HOoTWUFzrwdl2clwOYxdV8ZOtwxl2clw2oXzpkQW0dw7ZvghPmDBMnXYbViTk4pzvVzRJzK7aZ8fLQMThl98AixYpgBQkc8VfSIruNg/Dn9AGX5bFGDRMq3MS0Xr4AQmZ/y6oxDRCoTXPmJ159NKWLRM3VAKON/Hq1MiM2vuHYNNYPiVfMCqZTprexQRmVdzjwers1OQ6LTrjrIgS5ZpWXYKHDa5vNmXiMypuXfMFPOlgEXL1OWwoSwnhXtNiUxsxhfApf5xVLJM9arMS+Uwn8jEWgbG4QsoUyw+ARYv00sD45jyckWfyIzC2xs5zNesIt+NgAruUyMi8znb44EIsDaXZapVeJ6Fm/eJzKm5dwylmclIchl/JR+wcJmuyU2BTYBzPCiayJTO9YyZZvEJsHCZJjjsKMtO4ZUpkQn5/AFc6B9DpUkWnwALlykQnLg+wytTItO5NDABr98c9+SHWbpMqwrcaBmY4Io+kcmcDV0EVRfwytQQ1uW74Q8oXOjjij6RmZzpDq7k88rUIKpCr2pnOdQnMpWzPR6UmeSe/DBLl2lZdgqcduG8KZHJnOnxYJ0JDoSezdJl6nIET90/280yJTKLKa8fl/rHUWWilXzA4mUKAOsK3LwyJTKR831jCKjgz66ZWL5Mq/JT0T40ibFpn+4oRLQI4TUOXpkaTPjEmWZenRKZwpnuMTjtgjITnK4/m+XLlCv6ROZytseDtbmpcNrNVU/mSrsMpZnJSHTacKabt5USmcGZbo9pzjCdzfJlarMJ1uW7eWVKZAKeKS86hicvjyjNxPJlCgQnsrmiT2R84YOJzLb4BMRLmRa40eeZxuD4jO4oRDSP8J5wXpkaVHj+hUN9ImM70+NBssuO4owk3VGWLC7KNPwqd4Z3QhEZ2pluDyrz3bDZRHeUJYuLMs1zJyA9ycl5UyKDO9vjQZXJ7skPi4syFRFU5bt5jz6RgfWPTaN/bMaU26KAOClTAFhXEDx1XymlOwoRzeHybaQmXHwC4qhMq/Ld8Ez50D06pTsKEc3h8ko+r0yNLTx04CIUkTGd6RlDRrITue4E3VGWJe7KlNujiIzpbE/wNlIR863kA3FUppkpLuS5E3iPPpEBKaVwtttj2iE+EEdlCgQntnllSmQ8XSNT8Ez7THcg9GxxVabr8t1o7vXAH+CKPpGRnDHpgdCzxVWZVhW4MeUNoG1wQncUIprF7Cv5QLyVaXhFn0N9IkM50+NBQVoi0pOduqMsW1yVaWXoNjXeCUVkLGd7PKaeLwXirEyTXQ6sykrmlSmRgfgDCs09Y6a9Jz8srsoUAE/dJzKY1sEJTPsCpr0nPyzuyrSqIBUX+sYx7fPrjkJEAM50jwIAy9RsqgvS4AsonO8d1x2FiAA0dnlgE5ap6dQUBv/BmkKvhkSkV1P3KMpyUpDksuuOsiJxV6Zl2SlwOWxo4oo+kSE0dXtQU5CmO8aKxV2ZOuw2rMtPRWMXr0yJdBuf9qFlYALVJt8WBcRhmQLBeVNemRLpF96mWF3IK1NTqg49+rl/bFp3FKK41tQVKlNemZpTTehVMPwPSUR6NHWPIjXBgZJM8z3a+UpxWabhV0Gu6BPp1dTlQXWBeQ+Eni0uyzQ7NQF57gQ08sqUSBulFBq7R1FdaP4hPhCnZQoEJ7x5ZUqkT8fwJDxTPlRbYFsUEMdlWlPgRnPPGHz+gO4oRHEpvGZRwytTc6sudGPGH8DFft5WSqRDeGRYxStTcwsPLRq535RIi8ZuD1ZlJSM1waE7SkTEbZmuzU2Fwya8E4pIk6auUUvsLw2L2zJ1OWyoyEtFE8uUKOamvH5c7B+3xJ1PYXFbpkBwvylvKyWKveaeMQRUcCHYKuK6TGsK09A1MoXhiRndUYjiSmNo8YlXphYR/ofk1SlRbDV1eZDktGNVVrLuKBET12UaHmJw3pQotpq6R7GuwA27zfy3kYbFdZnmuhOQleLilSlRDCml0Ng1aqn5UiDOy1REUF3g5l5Tohjq80xjaMJrqW1RgKYyFZH/JiINInJKRH4hIokiUi4iB0XknIj8UkRcschSXZCGM92j8AdULP46orh3ust6i0+AhjIVkWIAnwNQp5TaAMAO4F4AXwfwTaVUBYAhAJ+IRZ7qQjemvAG0DPC2UqJYCE+r8co0MhwAkkTEASAZQBeAWwA8Evr9hwDcE4sg4Qd5cd6UKDaaukZRmJ6IjOSYDD5jJuZlqpTqAPCvAFoRLNERAEcADCulfKFPawdQPNfXi8gDInJYRA739fWtOE9lfipswhV9olhp6vZY7qoU0DPMzwRwN4ByAEUAUgDcsdivV0o9qJSqU0rV5ebmrjhPotOO8pwULkIRxcCML4BzvWOWmy8F9AzzbwVwUSnVp5TyAngMwPUAMkLDfgAoAdARq0A1PCiaKCbO943BF1C8Mo2QVgA7RSRZgg9+2QvgNIAXAbw/9Dn3A3giVoFqCtPQNjgJz5Q3Vn8lUVw63Rm8aKnhlenKKaUOIrjQdBTAyVCGBwF8CcBfi8g5ANkAfhyrTG8+YI9DfaJoOt01igSHDWtyUnRHiTgtp7Iqpb4K4KtXfPgCgO0a4mB9UTqA4KvmtWVZOiIQxYXTnaOoLkyDw269+4Ws979oGfLTgreVhocgRBR5Simc7hpFrQWH+ADLFEDwttLawjQ0dI3ojkJkWZ0jUxiZ9KK2iGVqabVFaTjbPQYvn1ZKFBXhkR+vTC1ufVEaZvwBnO8b0x2FyJIaOkcgYr3bSMNYpiHhV0vOmxJFx+nOUZTnpCDFIk8jvRLLNKQ8JwUJDhsaWKZEUWHlxSeAZXqZw25DdWEar0yJomBk0ov2oUnLLj4BLNO3qC1Mw+muUSjFs02JIqmxy9qLTwDL9C1qi9IwMulF58iU7ihElnJ5JZ9XpvEh/KrZ0MH9pkSR1NA5ipzUBOS5E3VHiRqW6Sw1hW6IvPlYBSKKjNNdo1hv4atSgGX6FskuB8pzUrgIRRRBwTNMPZYe4gMs07cJL0IRUWQ093rg9StLLz4BLNO3qS1KQ/vQJEYmeLYpUSTEw+ITwDJ9m8vH8fHqlCgiTneNIslpR1m29c4wnY1leoXLt5WyTIkioqFzFDWFbthtojtKVLFMr5DrTkCuO4GLUEQRoJRCY+eo5Yf4AMt0TuuL0tDQyb2mRCvVPjQJz7QPtYXpuqNEHct0DrWFaTjXO4Zpn193FCJTa4iTxSeAZTqn2qI0+AIKzT0825RoJU53jcImQFW+Nc8wnY1lOgcuQhFFxunOUazJTUWSy647StSxTOdQlp2CZJedi1BEK3S6c8Tyt5GGsUznYLMJani2KdGKDI3PoHNkyvJ3PoWxTK8ifFtpIMCzTYmWIzxNFg+LTwDL9KrWF6VhbNqH1sEJ3VGITOlk6CjLDUXW3xYFsEyvakNx8BvgFPebEi3LqY4RFGckITPFpTtKTLBMr2JdvhtOu1x+dSWipWnoHMXG4vi4KgVYplflcthQVeBGQwcXoYiWanTKi4v949hQHB/zpQDLdF4bi9NxsmOED9gjWqLwTpgNvDIlIPiNEH5ELREt3qnQ9Nj6OFl8Alim8wqvQp7ivCnRkpzqGEFBWiJy3Qm6o8QMy3QeVQVuOGxchCJaqpMdI3E1xAdYpvNKdNpRme/GKd4JRbRo49M+XIizxSeAZbqgjcVpOMVFKKJFa+wahVKIq21RAMt0QRuL0zE4PoOukSndUYhM4fKdTyxTmm196BuC86ZEi3OqYxS57gTkpyXqjhJTLNMF1BamwW4TrugTLdKpjhFsiJPDTWZjmS4g0WlHRW4qy5RoESZn/Gju9cTdfCnAMl2UDcXpONkxykUoogU0do8ioN6cHosnLNNF2Fichv6xafR6pnVHITK0hjhdfAJYposS/sY42c6hPtF8TnWMIivFhaL0+Fp8Alimi1JblAYRrugTLeRkR/CZTyKiO0rMsUwXIdnlwNrcVDTwoGiiq5r2+XG2Jz4XnwCW6aKFj+Mjormd6fbAF1BxOV8KsEwXbUNxOnpGp9Hr4Z1QRHM5FTpInVemNK/wJmSevE80t5MdI0hPcqIkM0l3FC1Ypou0vjidi1BE82joHMGG4vhcfAJYpouWmuBAeU4K74QimsOML4CmLk/cPNZ5LizTJdhQlM4yJZpDU/coZvwBbCrJ0B1FG5bpEmwsTkfnyBT6x3gnFNFsJ0I3tGwq4ZUpLUL4G6W+fVhzEiJjqW8bRlaKK24XnwCW6ZJsKE6HTYATbRzqE81W3z6CTSXpcbv4BLBMlyQlwYGKvFRemRLNMjHjQ3OvJ67nSwGW6ZJtKslAfTufCUUUdqojeOze5jieLwVYpku2uSQdA+Mz6Bie1B2FyBDCIzVemdKSbC4NfsPU8zg+IgDBn4Wi9ETkuhN0R9GKZbpE1QVpcNltONHGeVMiIHhlujHOh/gAy3TJXA4bagrdOMFFKCKMTHhxaWAi7of4AMt0WTaVZAQn3QNchKL4Vt8RvKjYzDJlmS7HppJ0jE37cKF/THcUIq3Cawcc5rNMlyW8CMXN+xTvTrQNozwnBelJTt1RtGOZLsPa3FQku+zcvE9xL3znE7FMl8VuE2woTsdxbo+iONY7OoXu0SkuPoWwTJdpS2kGGjtHMeML6I5CpEX4pKh4v/MpjGW6TJtK0jHjD+BMt0d3FCIt6tuHYZPgo9CJZbps4a0g3G9K8epE+wjW5buR7HLojmIILNNlKslMQmayk4tQFJeUUqhvH+bi0yxaylREMkTkERFpEpFGEblORLJE5DkRaQ79mqkj22KJyOUTpIjiTfvQJIYnvFx8mkXXlem3AfxBKVUNYDOARgBfBrBPKVUJYF/ofUPbXJKOsz0eTMz4dEchiqnw9BbvfHpTzMtURNIB7AbwYwBQSs0opYYB3A3godCnPQTgnlhnW6pNJRkIqOB5jkTxpL59BC67DVUFbt1RDEPHlWk5gD4APxWRYyLyIxFJAZCvlOoKfU43gPy5vlhEHhCRwyJyuK+vL0aR57aplM+Eovh0vHUYNUVpcDm47BKm4/8JB4BtAL6vlNoKYBxXDOlV8Bj7OU8RUUo9qJSqU0rV5ebmRj3sfPLciShMT7y8344oHvj8AdR3DGNrKYf4s+ko03YA7Uqpg6H3H0GwXHtEpBAAQr/2asi2ZFtKM3C8bUh3DKKYaer2YMobwNZVLNPZYl6mSqluAG0iUhX60F4ApwH8FsD9oY/dD+CJWGdbjq2rMtA2OIn+sWndUYhi4njoYPStpYbecBNzunbbfhbAz0XEBeACgI8jWOy/EpFPAGgB8EFN2ZZk66rgN9Tx1mHcWjvnNC+RpRxrHUZ2igulWUm6oxiKljJVSh0HUDfHb+2NdZaV2lCUDrtNcKxtiGVKceF42xC2lGZARHRHMRQuxa1QksuOmkI3jrVyRZ+sb2TCi/N945wvnQPLNAK2lmbiRNsw/HyMCVlceLN+eHqL3sQyjYCtqzIwPuPHuV4+xoSs7VjrMETAe/LnwDKNgPCr9LFWbpEiazveNoTKvFS4E/mYkiuxTCOgLDsZGclOzpuSpSmlcLxtGFu4WX9OLNMIEBFsKc3AMW7eJwtrGZjA0ISX86VXwTKNkK2lmWjuHcPolFd3FKKoCF8s8Mp0bizTCNm6KgNKAfV8/DNZ1PHWYSS77FiXz5Oi5sIyjZDNoVdr3qdPVnWsLXiyvt3GzfpzYZlGSHqSExV5qVyEIkua8vpxunOU86XzYJlGUHARahjBEwSJrKOhcwS+gOJ86TxYphG0dVUGBsdn0Do4oTsKUUSFR1w8w/TqWKYRFD6SjEN9sppjbcMozkhCXlqi7iiGxTKNoHX5qUh22S+f90hkFcdbh7GFh5vMi2UaQQ67DZtK0nlbKVlKr2cKHcOTHOIvgGUaYVtXZaKhcxRTXr/uKEQRcbQleHHAlfz5sUwjbEtpBnwBhYZObt4nazjSMgSXw4YNxWm6oxgayzTCwofmHm3hvClZw5GWIWwqTkeCw647iqGxTCMsz52I0qwkHGnhvCmZ35TXj1Mdo7hmNYf4C2GZRkHd6iwcbhni5n0yvVMdI5jxB1imi8AyjYJrVmeif2wabYOTuqMQrUh4hLWNZboglmkUhF/FD7cMak5CtDKHW4ZQlp2MnNQE3VEMj2UaBevy3XAnOHCY86ZkYkopHG0ZwjWrs3RHMQWWaRTYbYKtqzNx5BLLlMyrZWACA+MznC9dJJZplNStzsTZXg9GJnnyPplTeGTFMl0clmmU1K3OhFJ8YimZ15GWIbgTHajMS9UdxRTmLVMRuU5Evici9SLSJyKtIvI7EfmMiPDB2fPYXJoBu02435RM62jLELatyoSNJ+svylXLVER+D+DPATwD4A4AhQBqAfw9gEQAT4jIe2IR0oxSEhyoKXTjMOdNyYRGJr042+vhEH8JHPP83keUUv1XfGwMwNHQf/8mIjlRS2YBdauz8MtDbfD5A3DYOaNC5nGsdQhKBaeraHGu+hMeLlIR+YSIVM73OTS3a1ZnYtLrR2OXR3cUoiU52jIEm7z5oEha2GIul1YB+IGIXBCRX4vIZ0VkS7SDWUFdGTfvkzkdaR1CTWEaUhLmG7zSbAuWqVLqq0qpWwCsB/AqgL8BcCTawaygMD0JRemJ3LxPpuLzB3CsdZhD/CVa8GVHRP4ewPUAUgEcA/DfESxVWoRryrJw+BKvTMk8mro9mJjx8378JVrMMP99ALIBPA/gMQBPKKW6oprKQupWZ6JrJPjYByIzONrKzfrLsZhh/jYAtwJ4A8BtAE6KyP5oB7OKy4ee8OqUTOLQpSHkpyWgOCNJdxRTWbBMRWQDgPsA3A/gQwA6ALwQ5VyWUV3gRorLzs37ZApKKRy6OIhry7Igws36S7GYpbqvAXgFwHcAHFJK8WbzJXDYbdiyKoNlSqbQPjSJ7tEp7CjnSVFLtWCZKqXeHYsgVla3OgvffaEZnikv3IlO3XGIrurgxeB01LUs0yWb73bSJ0XkLhF520+/iKwRkX8SkT+Lbjxr2FGehYACr07J8A5dHER6khPr8ty6o5jOfHOmnwRwI4AmETkUOuDkRRG5COAHAI4qpX4Sk5Qmt3VVJhw2wRsXuQhFxvbGpUFcW8bDTZbjqsN8pVQ3gC8C+KKIlAEoADAJ4KxSivt8liDJZcemknSWKRlar2cKF/vHce+1pbqjmNJVy1REPABmP15Twu+LyDSA8wC+opTaF9WEFnFteRZ+sv8iprx+JDr5/HEynvAJZ9s5X7os8x104lZKpc367/L7CF6l/gWAb8csqcntKM+C169wrHVYdxSiOb1xcRBJTjs2FPOo4uVY1rlwSim/UuoEgO9GOI9lXbM6CyLgUJ8M642Lg9i2OgNOHhe5LCv6f00p9YNIBbG69CQnagrS8MalAd1RiN5mZNKLxu5RXFvGIf5y8SUohraXZ+FIyxBmfAHdUYje4mhL8DBozpcuH8s0hnaUZ2HKG8CpzhHdUYje4uDFQTjtgq2lPNxkuVimMRS+q4TzpmQ0hy4NYkNxOpJc3GmyXCzTGMpJTcDa3BSWKRnKlNeP+vZhDvFXiGUaY9vLs3Ho0iD8AbXwJxPFwLHWYXj9Ctu5+LQiLNMY21GeBc+UD03do7qjEAEIDvFFggfy0PKxTGNsO+dNyWDeuDiIqnw30pN5otlKsExjrCgjCSWZSSxTMgSvP4CjrUOcL40AlqkG28uz8MbFQSjFeVPSq759GBMzfly3Jlt3FNNjmWqwvSwLA+MzON83rjsKxbk/XgiOkHawTFeMZapBeEh18CJvLSW9DpwfQHWBG1kpLt1RTI9lqkF5Tgry3AmXrwqIdJj2+XG4ZRA7eVUaESxTDUQE163NxoHzA5w3JW3q20cw5Q2wTCOEZarJdWuy0T82jXO9Y7qjUJw6cH4AIsDONVzJjwSWqSa71uYAAA5c4Lwp6XHg/ABqCtKQkcz50khgmWpSmpWE4owkHDjPMqXYm/L6cbR1iEP8CGKZaiIi2LkmGwcuDCDA+/Qpxo63DWPaF8B1a1mmkcIy1WjX2mwMT3jR1O3RHYXiTHi+lHc+RQ7LVKPwVcHr5/s1J6F488cLA1hflIb0JN6PHyksU42KMpKwOjsZf+QiFMXQlNePY63DvIU0wlimmu1am42DFwbh8/O5UBQbR1uGMOPnfGmksUw127kmG55pHxo6eb4pxcaBCwOwCVDHw6AjimWqWfjqgPtNKVb+eGEAG4vTkZbI+dJI0lamImIXkWMi8lTo/XIROSgi50TklyISFzuJ89yJqMhLxevcb0oxMDnjx/G2YezkED/idF6Zfh5A46z3vw7gm0qpCgBDAD6hJZUG163JxuFLg/By3pSi7HDLILx+xc36UaClTEWkBMCdAH4Uel8A3ALgkdCnPATgHh3ZdNi1NhsTM8EnRBJF0/7mfjjtgh3cXxpxuq5MvwXgiwDCl2LZAIaVUr7Q++0Aiuf6QhF5QEQOi8jhvr6+6CeNgfDBvK+f41Cfomv/uX5sW5WJZJdDdxTLiXmZisi7AfQqpY4s5+uVUg8qpeqUUnW5ubkRTqdHVooL1QVuzptSVA2Oz6ChcxQ3VOTojmJJOq5MrwfwHhG5BOBhBIf33waQISLhl8sSAB0asmlzQ0UOjrQMYXLGrzsKWVT4TrvrK1mm0RDzMlVK/a1SqkQpVQbgXgAvKKXuA/AigPeHPu1+AE/EOptON1TmYMYfwBuXePo+Rcf+5n64Ex3YVJyuO4olGWmf6ZcA/LWInENwDvXHmvPE1I7ybLjsNrx61hrzwGQsSim82tyP69Zkw2E30o+9dWidhVZKvQTgpdDbFwBs15lHpySXHXVlmdh/joeeUOS1Dk6gY3gSf3HTGt1RLIsvUQZyQ2UOmro96PVM6Y5CFhN+kebiU/SwTA3kxorg7oTXeHVKEba/uR9F6Ykoz0nRHcWyWKYGsr4oDZnJTrzazDKlyPEHFF4/P4DrK3IQvD+GooFlaiA2m2BXRQ72N/fzEdAUMQ2dIxiZ9OIGbomKKpapwdxYkYNezzSa+QhoipDwfGn4ibgUHSxTgwlfPbzCLVIUIfub+1Fd4EauO0F3FEtjmRpMSWYy1uSkcIsURcTkjB+HLw1xFT8GWKYGdENlDg5eGMS0j7eW0socujSIGX+At5DGAMvUgG6oyMGk14+jLTySj1bmlbN9cDls2FnO80ujjWVqQDvXZsNuE+w/x3lTWpmXz/ZhR3kWklx23VEsj2VqQGmJTmwpzcB+7jelFegYnkRz7xhuWmeNoyqNjmVqUDdW5qC+YwRD4zO6o5BJvXwmOLK5uYplGgssU4O6aV0ulAJeaeZQn5bn5bO9KM5IwtrcVN1R4gLL1KA2lWQgM9l5+eqCaCm8/gBeOzeA3etyeQtpjLBMDcpuE+xel4uXz/YhEOCtpbQ0R1uGMDbt43xpDLFMDWxPVR4GxmdwsmNEdxQymZfP9sFhE+yq4JaoWGGZGlhwiAa8xKE+LdHLZ/uwbXUm0hKduqPEDZapgWWluLCpJAMvne3VHYVMpNczhYbOUQ7xY4xlanB7qnJxvG0Yg9wiRYv0ytng/mRuiYotlqnB3VyVB6WAV7lFihbp5bN9yHUnoLYwTXeUuMIyNbhNxenISnFx3pQWxR9QeLW5D7sruSUq1limBmezCXZX5nCLFC1Kffswhie8uIlD/JhjmZrAnuo8DI7PoJ5bpGgBLzb1wibBJzZQbLFMTeDGyvAWKa7q0/z2NfXimtWZyExx6Y4Sd1imJpCV4sLmkgzOm9K8ukYm0dA5iluq83VHiUssU5PYU5WHE+3DGBib1h2FDOqFpuDI5daaPM1J4hPL1CRuqQ5ukeLVKV3NC429KM1KQkUeT4nSgWVqEhuK05CfloDnG3t0RyEDmpzxY/+5fuytzueWKE1YpiYhIthbk4+Xz/ZhyssH7dFbvX6+H9O+APZyiK8Ny9REbqvJx8SMH3+8MKA7ChnM8429SHHZsb08S3eUuMUyNZHr1mYjyWnHvkZukaI3KaXwQlMPdq/LRYKDD87ThWVqIolOO26szMHzjT1QindUkkUSAAAVw0lEQVRDUVBD5yh6RqdxSzWH+DqxTE3m1tp8dI0Ej1gjAoB9jb0QCd4pR/qwTE3mluo8iICr+nTZC0092FKagZzUBN1R4hrL1GRyUhOwbVUm500JQPAg6BPtI9jLq1LtWKYmtLcmDyc7RtA1Mqk7Cmn2/Ongi+reGt5CqhvL1IRuC/3g8OqUnmnoxqqsZFQXuHVHiXssUxOqyEvF6uxkzpvGudEpL14/34/b1/OuJyNgmZqQiGBvdT5ePz+A8Wmf7jikyYtNvfD6FW5fX6A7CoFlalq31uZhxhfAy2d58Em8erah5/KCJOnHMjWp7WVZyEx24g+nunVHIQ2mvH68dKYXt9Xmw2bjEN8IWKYm5bDb8I7aArzQ1ItpHw8+iTevnevH+Iwft6/nKr5RsExN7I6NBRib9mF/c7/uKBRjzzR0w53gwK61fNaTUbBMTez6tTlwJzg41I8zPn8Azzf2Yk91HlwO/ggbBf8lTMzlsGFvTR6ea+yB1x/QHYdi5HDLEAbHZ7iKbzAsU5O7Y0Mhhie8OHhhUHcUipFnGrrhcthwc1Wu7ig0C8vU5G5al4skpx2/P9WlOwrFgFIKzzb0YHdlDlISHLrj0CwsU5NLctlxc1UunmnogT/AM06t7mTHCDqGJ/EODvENh2VqAXdsKED/2DSOtg7pjkJR9nR9F5x2we21LFOjYZlawC3VeXDZbfj9Sa7qW5lSCk/Vd+GGihykJzt1x6ErsEwtwJ3oxI2VOXimoZuPM7GwE+3BIf6dm4p0R6E5sEwt4o4NBegYnsTxtmHdUShKnq7vhNMuuK2Wdz0ZEcvUIt6xvgAuuw1PnuCqvhUFAgpP13dhd2Uu0pM4xDcilqlFpCc5cVNVLp6q7+SqvgUdaxtG58gU7txUqDsKXQXL1ELes7kIvZ5pHLrEDfxW83R9F1x2G27lEN+wWKYWsrcmD0lOO5480ak7CkVQIKDwu5NduKkqF2mJHOIbFcvUQpJdDtxam4/fn+rmvfoWcrR1CN2jU3g3h/iGxjK1mLs2FWJwfAavnx/QHYUi5Kn6rtChNhziGxnL1GJuqsqFO9HBob5F+PwBPFXfhT1VuUjlvfiGxjK1mASHHXesL8Azp7p5Ar8FvH5+AP1j03jv1mLdUWgBLFMLumtzETzTPrx0hg/bM7vHj3UgLdGBm6vydEehBbBMLWjX2mxkpbg41De5iRkf/tDQjTs3FSLRadcdhxbAMrUgh92Gd20swPONPRib9umOQ8v03OkeTMz4cfcWDvHNgGVqUe/dWoIpbwC/P8nbS83q8WMdKEpPxPayLN1RaBFYpha1bVUGynNS8NjRDt1RaBn6x6bxSnM/7t5aDJtNdMehRWCZWpSI4H1bi3HgwgDahyZ0x6EleupE8IyFezjENw2WqYXdE9pO8/gxXp2azePHO1FTmIaqArfuKLRIMS9TESkVkRdF5LSINIjI50MfzxKR50SkOfRrZqyzWU1pVjJ2lGfhsaMdPDTaRC72j+N42zDeu5WHQJuJjitTH4AvKKVqAewE8BkRqQXwZQD7lFKVAPaF3qcV+pNtJbgQ+uEkc/jN0XaIAO/ZzCG+mcS8TJVSXUqpo6G3PQAaARQDuBvAQ6FPewjAPbHOZkXv3FiABIeNC1Em4Q8oPHKkHbsrc1GQnqg7Di2B1jlTESkDsBXAQQD5SqnwPp5uAHOe6iAiD4jIYRE53NfHO3wW4k504vb1BfjtiU7eXmoCr53rR+fIFD5YV6o7Ci2RtjIVkVQAjwL4K6XU6OzfU8EJvjkn+ZRSDyql6pRSdbm5uTFIan7v21aMkUkvXmzq1R2FFvDrI+3ISHbi1lrePmo2WspURJwIFunPlVKPhT7cIyKFod8vBMCf/Ai5oSIHee4E/Ppwu+4oNI/hiRk809CNe7YUI8HB20fNRsdqvgD4MYBGpdT/mPVbvwVwf+jt+wE8EetsVuWw2/D+a0rw4pledI1M6o5DV/HbE52Y8QXwgboS3VFoGXRcmV4P4CMAbhGR46H/3gXgawBuE5FmALeG3qcI+dC1pQgo8OrUwH51uA21hWlYX5SuOwotQ8xPm1VK7Qdwtfvj9sYySzxZnZ2C6yuy8ctDbfjLPRW8RdFgTneO4lTHKP7vu2p1R6Fl4h1QceTea1ehY3gS+8/1645CV/j1kTa47DaeEGViLNM48o71+chMduLhQ626o9AsU14/fnOsA7etz0dmikt3HFomlmkcSXDY8SfbSvDc6R70j03rjkMhT9d3YXjCi/u2r9IdhVaAZRpn7t1eCq9f4dEjXIgyip8dbMGa3BRctzZbdxRaAZZpnKnIc+Paskz88lAbDz8xgIbOERxrHcZ9O1YjuGuQzIplGofuvXYVLvSP48CFAd1R4t7PD7YiwWHDn2zjwpPZsUzj0J2bCpGZ7MT/PtCiO0pc80x58fixDty1uQgZyVx4MjuWaRxKdNrxoWtX4dnTPegc5h1Rujx+rAMTM358eOdq3VEoAlimceq+HauglMLPD/LqVIfg//etWF+Uhs0lvOPJClimcao0Kxl7a/LxizfaMOXl0XyxdrhlCE3dHnx4JxeerIJlGsc+tqsMg+MzeLqej4OOtZ++dhHpSU7cvYWPJrEKlmkc27U2GxV5qfjPA5d0R4kr7UMT+MOpbvzp9lVIdsX8eAyKEpZpHBMR3H/dapxoH8Gx1iHdceLGQ69fgojgo9dx4clKWKZx7r3bSpCa4MB/vH5Jd5S4MDbtw8OH2vDODQUoykjSHYciiGUa51ITHPjQtaV4qr4LHdwmFXWPHmmHZ8qHP7uhXHcUijCWKV3+wf7J/ouak1hbIKDw09cuYktpBratytQdhyKMZUoozkjCXZsK8fAbrRiZ9OqOY1kvNPXi0sAEPsGrUktimRIA4JO712B8xo///yDPOo2WH7xyHkXpibhjQ4HuKBQFLFMCAKwvSscNFTn46WsXMe3jJv5IO3RpEIcuDeGTu9fAaeePnRXxX5Uue2D3GvR6pvHE8U7dUSzn+y+dR1aKC/deywOgrYplSpfdWJmDmsI0/PCVCwgEeNZppDR2jeKFpl58bFcZklx23XEoSlimdJmI4C92r0Fz7xiea+zRHccyvv/SeaS47Lj/ujLdUSiKWKb0Fu/eVIiy7GR8Z18zT+KPgNaBCTxV34n7dq5GerJTdxyKIpYpvYXDbsNn9lSgoXMU+xp7dccxvX9/5TwcNhu3Q8UBlim9zT1bi1GalYTvvMCr05VoH5rArw+34QN1JchPS9Qdh6KMZUpv47Tb8Jd7KlDfPoKXzvTpjmNa33vxHASCz+yp0B2FYoBlSnN679YSFGck4ducO12W1oEJ/PpwO/50eykPNIkTLFOak8thw6f3rMXxtmG8dJZXp0v13ReaYbMJPs2r0rjBMqWr+sA1pSjNSsI3/nCG+06X4GL/OB471oEP71jNudI4wjKlq3I5bPjCbVVo7BrFk/W8K2qxvrOvGU674FM3r9UdhWKIZUrzes/mItQUpuHfnj2LGV9AdxzDO9UxgsePd+Bju8qR607QHYdiiGVK87LZBF+8owqtgxP4xRs8UWo+Sin88+8bkZHkxKf38Ko03rBMaUE3r8vFjvIsfPeFZoxP+3THMayXzvbhtXMD+NzeSqQl8m6neMMypQWJCL70zmr0j83gB69c0B3HkPwBha/9rgmrs5Nx3w4+KC8esUxpUbatysSdmwrxg5fPo31oQnccw3n0SDvO9Hjwxdur4XLwxyoe8V+dFu3v3lUDEeCff9ekO4qhjE558Y1nzmDrqgy8ayNP0Y9XLFNatOKMJHzqpgo8fbILB84P6I5jGN96rhkD49P4x/esh4jojkOasExpSf7ipjUozkjCPz7ZAJ+fW6Waukfx0IFL+C/bV2FTSYbuOKQRy5SWJNFpx9/fWYOmbg9+HucP31NK4R8eb0BaogN/c3uV7jikGcuUluyODQW4sTIH//LMGXSNTOqOo80TxzvxxqVBfPGOamQku3THIc1YprRkIoL/956N8AUC+L8ePxWXp0oNjs/g/3nqNDaXZuBDdaW645ABsExpWVZlJ+MLt1Xh+cZe/O5kt+44MfdPTzZgdMqLr//JRthsXHQilimtwMevL8PG4nR89bcNGJnw6o4TM8+f7sHjxzvxmT0VqC5I0x2HDIJlSsvmsNvwz+/biKGJGfzjkw2648TEyKQXX3n8JKoL3Pj0zTyrlN7EMqUV2VCcjs/sqcBjxzrwVBwc0/ePTzagzzONb7x/E+90orfgdwOt2GdvqcCW0gx85TenLL26/8TxDjx2tAOf21vJPaX0NixTWjGn3YZvfmgLvP4AvvCrE5Y8lb91YAJf+c0p1K3OxF/yUSQ0B5YpRUR5Tgr+4d21eP38AL7/8nndcSLK6w/g8788BhHgW/dugcPOHxt6O35XUMR86NpS3LW5CP/27Bnsb+7XHSdivv77JhxrHcb/996NKMlM1h2HDIplShEjIvja+zaiIi8Vn3v4GDqGzT9/+sTxDvxo/0V8bFcZ7tpcpDsOGRjLlCIqJcGBf//wNZjxBfDpnx3BlNevO9KyNXSO4EuP1mN7WRa+cmeN7jhkcCxTirg1uan4tw9uxon2EdMuSPWOTuGB/zyCjCQXvnffNjg5T0oL4HcIRcXt6wvwd++qxtMnu/D1P5jrMOmxaR8+/h+HMDQxgx9+tI5PGaVFcegOQNb1yRvXoG1wEj945QKKM5Pw0evKdEdakNcfwKd+dgRN3R786P46bCxJ1x2JTIJlSlEjIvjqXbXoGpnCPzzRgESHHR+81rgnLPn8AfzVw8fxanM/vvH+TdhTlac7EpkIh/kUVQ67Df/zv2zF7nW5+NJj9Xj0SLvuSHPy+QP4/MPH8fTJLnzlXTX4II/VoyVimVLUJTrtePAj1+D6tTn4m0dO4OE3jHVC/7TP/5Yi/eTuNbojkQmxTCkmEp12/PCjddi9LhdffuwkvvX8WUMcKj0y6cX9P3kDT5/swt/fySKl5WOZUswkuYKF+oFrSvCt55vxpUfrte5DbRucwAf+/XUcaRnCNz+0GX9+I4uUlo8LUBRTTrsN33j/JhRmJOE7+5rR2OXB/7pvG0qzYnub5nOne/CFXx2HAvDQx7djV0VOTP9+sh5emVLMiQj++rZ1+OFH63BpYBzv/u5+PHG8IybD/skZP/7pydP45H8eRmlWMp7+7I0sUooIlilpc1ttPp767A0oz0nB5x8+jj9/6DA6o3g//2vn+nH7t17BT167iI/sXI1HP7ULq7J5cAlFhhhhEWC56urq1OHDh3XHoBXyBxR++tpF/OuzZ6AU8LFdZfjUzWsj9vjkxq5R/OszZ7CvqRflOSn45/dtxM412RH5s8maROSIUqpuSV/DMiWjaB+awDefa8Zjx9qR7LTj/deU4CPXrUZFnnvJf5Y/oPBKcx8eev0SXjrTh7REBz51cwU+fn0ZEp32KKQnK2GZkiU0dY/iwZcv4Kn6Lsz4A6gucOMdtfmoK8vC5tIMpCc53/Y1Sim0D03ieNswXj/fj2cbejAwPoNcdwLu27EKH99VjvTkt38d0VxYpmQp/WPTeOJ4J55p6MahS4MIf6tmJDtRkJaIBKcdUAojk150j05hyhsAAKS47NhTnYd3bijEbbX5fPAdLRnLlCxrdMqL+rYR1HcMo3N4Ej2j05jxBcvTnehAQVoiVuekYEtJBqoK3CxQWpHllKmh9pmKyB0Avg3ADuBHSqmvaY5EBpGW6MQNlTm4oZLbmMiYDPPyLSJ2AN8D8E4AtQD+VERq9aYiIlocw5QpgO0AzimlLiilZgA8DOBuzZmIiBbFSMP8YgBts95vB7Djyk8SkQcAPBB6d0xEzkQhSw4AMzxe0yw5AWaNFmaNjqqlfoGRynRRlFIPAngwmn+HiBxe6uSzDmbJCTBrtDBrdIjIkle2jTTM7wAw+0TektDHiIgMz0hleghApYiUi4gLwL0Afqs5ExHRohhmmK+U8onIXwJ4BsGtUT9RSjVoihPVaYQIMktOgFmjhVmjY8lZTb1pn4jIKIw0zCciMi2WKRFRBLBM5yAi/yIiTSJSLyK/EZEM3ZmuJCJ3iMgZETknIl/WnedqRKRURF4UkdMi0iAin9edaSEiYheRYyLylO4s8xGRDBF5JPS92igi1+nOdDUi8t9C//6nROQXIpKoO1OYiPxERHpF5NSsj2WJyHMi0hz6NXOhP4dlOrfnAGxQSm0CcBbA32rO8xYmu/XWB+ALSqlaADsBfMbAWcM+D6BRd4hF+DaAPyilqgFshkEzi0gxgM8BqFNKbUBwgflevane4j8A3HHFx74MYJ9SqhLAvtD782KZzkEp9axSyhd6948I7nk1EtPcequU6lJKHQ297UHwB75Yb6qrE5ESAHcC+JHuLPMRkXQAuwH8GACUUjNKqWG9qeblAJAkIg4AyQA6Nee5TCn1CoDBKz58N4CHQm8/BOCehf4clunC/gzA73WHuMJct94atqDCRKQMwFYAB/Ummde3AHwRQEB3kAWUA+gD8NPQlMSPRCRFd6i5KKU6APwrgFYAXQBGlFLP6k21oHylVFfo7W4A+Qt9QdyWqYg8H5q/ufK/u2d9zlcQHKb+XF9SaxCRVACPAvgrpdSo7jxzEZF3A+hVSh3RnWURHAC2Afi+UmorgHEsYiiqQ2i+8W4EXwCKAKSIyIf1plo8Fdw/uuAeUsNs2o81pdSt8/2+iHwMwLsB7FXG24xrqltvRcSJYJH+XCn1mO4887gewHtE5F0AEgGkicjPlFJG/MFvB9CulApf5T8Cg5YpgFsBXFRK9QGAiDwGYBeAn2lNNb8eESlUSnWJSCGA3oW+IG6vTOcTOqT6iwDeo5Sa0J1nDqa59VZEBMF5vUal1P/QnWc+Sqm/VUqVKKXKEPz/9AWDFimUUt0A2kQkfLrRXgCnNUaaTyuAnSKSHPp+2AuDLpbN8lsA94fevh/AEwt9QdxemS7gfwJIAPBc8N8ef1RK/Ve9kd5ksFtvF3I9gI8AOCkix0Mf+zul1O80ZrKKzwL4eegF9QKAj2vOMyel1EEReQTAUQSnzY7BQLeWisgvANwMIEdE2gF8FcDXAPxKRD4BoAXABxf8c4w3giUiMh8O84mIIoBlSkQUASxTIqIIYJkSEUUAy5SIKAJYpkREEcAyJSKKAJYpWZaI/I2IfC709jdF5IXQ27eICM9boIhimZKVvQrgxtDbdQBSQ+cE3AjgFW2pyJJYpmRlRwBcIyJpAKYBHECwVG9EsGiJIob35pNlKaW8InIRwMcAvA6gHsAeABUw/kEbZDK8MiWrexXAf0dwWP8qgP8K4JgBj1Ukk2OZktW9CqAQwAGlVA+AKXCIT1HAU6OIiCKAV6ZERBHAMiUiigCWKRFRBLBMiYgigGVKRBQBLFMioghgmRIRRcD/AbzSc1R+n6ZpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=[5, 7])\n",
    "w = np.arange(-3, 10, 0.01)\n",
    "j = np.square(w - 3) * 3 + 1\n",
    "plt.xlabel(\"w\")\n",
    "plt.ylabel(\"J(w)\")\n",
    "plt.xlim(-3, 10)\n",
    "plt.ylim(0, 100)\n",
    "plt.plot(w, j)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 什么时候通过加上梯度值（乘以学习率）更新参数？  \n",
    "> 目标函数要求极大值时，通过加上梯度值（乘以学习率）更新参数  \n",
    "> 这种方法叫 **梯度上升法**。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 为什么要乘以学习率？   \n",
    "梯度向量 $(1, 2)$ 和 $(10, 20)$ 表示同一个方向  \n",
    "\n",
    "<img src=\"./imgs/2.learning_rate1.png\" width=\"500px\">  \n",
    "<img src=\"./imgs/2.learning_rate2.png\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.2.3 各种梯度下降的方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 批梯度下降BGD（Batch Gradient Descent） \n",
    "  \n",
    "    * 将整个训练集计算梯度，更新参数，多次迭代。  \n",
    "    * 计算慢，计算量随训练集的增大而增大，计算冗余。  \n",
    "    * 新数据加入，需要重新计算，更新模型。  \n",
    "    * 对于凸函数可以收敛到全局极小值，对于非凸函数可以收敛到局部极小值。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 随机梯度下降SGD（Stochastic Gradient Descent）  \n",
    "  \n",
    "    * 从训练集中**随机**抽出（有放回的）一条数据，计算梯度，更新参数，多次迭代。  \n",
    "    * 计算快，更新参数快。  \n",
    "    * 随机采样代表全训练集，新数据加入无需重新计算整个数据集。  \n",
    "    * 噪声大，并不是每次迭代都向着整体最优化方向。虽然训练速度快，但是准确度下降，并不是全局最优。  \n",
    "    * 更新比较频繁，会造成 $loss$ 有严重的震荡。    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 小批量梯度下降MBGD（Mini-Batch Gradient Descent）  \n",
    "小批量随机梯度下降 Mini-batch Stochastic Gradient Descent   \n",
    "  \n",
    "    * 从训练集中**随机**抽出（有放回的）长度为 $batch\\_size$ 的Mini-Batch样本代表全样本， 计算梯度，更新参数，多次迭代  \n",
    "    * 可以降低参数更新时的方差，收敛更稳定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./imgs/2.gradient_descent4.jpg\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.2.2.4 超参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 只控制模型的训练过程\n",
    "* 需要手动设置\n",
    "* **超参数调优**：多次调整超参数的组合，获得使模型训练效果最好的超参数  \n",
    "  \n",
    "    * learning_rate\n",
    "    * batch_size\n",
    "    * epochs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**名词解释**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* batch_size: 每次从训练集随机抽取的mini-batch的样本数   \n",
    "  \n",
    "* step（Iteration）: 用一个mini-batch求梯度 更新一次参数，叫做one step(1步)  \n",
    "  \n",
    "* epoch: 将训练集的全部数据完整训练一次叫做一个epoch  \n",
    "  \n",
    "* steps_per_epoch: 一个epoch需要经过的step数。$\\begin{aligned} steps\\_per\\_epoch = \\frac{sample\\_length} {batch\\_size} \\end{aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.3 问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * 什么时候用闭式解， 什么时候用梯度下降法 ？  \n",
    "  \n",
    "     *  一般特征数比较大，$X^T X$ 不可逆的时候  \n",
    "     * 数据集较大，较复杂的时候，闭式解矩阵乘法和矩阵求逆<a href=\"https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra\">运算量大</a>，梯度下降法更快  \n",
    "         \n",
    "         \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Lasso(索套回归)和Ridge(岭回归)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1 Overfitting 过拟合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.3.1.1 问题描述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "过度拟合训练集，模型只在训练集表现很好，在测试集表现很糟。  \n",
    "出现原因：  \n",
    "* 数据过少\n",
    "* 模型过复杂  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFACAYAAAASxGABAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE/BJREFUeJzt3X+MpVd5H/DvszuuyAJqTNm6LjAyQoiK0Ma0EzQxdZWwm8qhKJBUqoJaalQkV1VIIUKqSOutHbkKCCeklVrROuGHpRIsxA9hVSTF3iChxCzxeouMjdM6SmFjZ1kvQi2obht25ukf99raml3vTNj3nrtzPx9p9L73/THvo1f27FfnnHtOdXcAABhn3+gCAABWnUAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMNja6AJ24oUvfGFfc801o8sAALioBx544JvdfXA391wWgeyaa67J8ePHR5cBAHBRVfX13d6jyxIAYDCBDABgMIEMAGAwgQwAYDCBDABgsMkCWVW9pKo+X1VfraqHq+od8+O3VtXjVfXl+c/rp6oBAOByMOW0F2eTvKu7T1TV85M8UFX3zM/9Wnf/yoTPBgC4bEwWyLr7VJJT8/3vVNUjSV401fMAAC5XCxlDVlXXJHl1ki/ND729qh6sqg9V1ZUXuOemqjpeVcfPnDmziDIBAIaYPJBV1fOSfDLJO7v720k+kORlSa7NrAXtV893X3ff0d0b3b1x8OCuVh8AALisTBrIquqKzMLYR7v7U0nS3ae7e6u7t5P8epLXTFkDALBA21vJfbcmd10/225v7ezciptsDFlVVZIPJnmku99/zvGr5+PLkuSnkzw0VQ0AwIIduy25//bk7JPJ6RNJKrnuloufW3FTtpC9NslbkrzuGVNcvK+qvlJVDyb58SS/MGENAMAinTw6C1zJbHvy3p2dW3FTfsvyd5PUeU59dqpnAgCDrR+atX6dfTJZO5CsH97ZuRU35TxkAMCq2TySpGatX+uHk82bd3ZuxVV3j67hojY2Nvr48eOjywAAnrK9NRsTdvLorOVr80iyb//oqpZCVT3Q3Ru7uUcLGQCwe1/8peRL70n6bPInx5Lu5LW/NLqqy5bFxQGA3XvoI7Mwlsy2D394aDmXO4EMANi97/na3vm+x8dOCWQAwO790FuTfVfM9vddMfvMn5kxZADA7v3oLUnt943JS0QgAwB2b9/+2Sz7Ztq/JHRZAgAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABwLLb3kruuzW56/rZdntrdEVcYtayBIBld+y25P7bk7NPJqdPJClrSO4xWsgAYNmdPDoLY8lse/LesfVwyQlkALDs1g8lawdm+2sHkvXDY+vhktNlCQDLbvNIkpq1jK0fTjZvHl0Rl5hABgDLbt/+2Zgx48b2LF2WAACDCWQAAIMJZAAAgwlkAACDCWQAAIMJZAAAgwlkAACDCWQAAIMJZAAAgwlkAACDCWQAAIMJZAAAgwlkAKyW7a3kvluTu66fbbe3RlcEWRtdAAAs1LHbkvtvT84+mZw+kaSS624ZXRUrTgsZAKvl5NFZGEtm25P3jq0HIpABsGrWDyVrB2b7aweS9cNj64HosgRg1WweSVKzlrH1w8nmzaMrAoEMgBWzb/9szJhxYywRXZYAAINNFsiq6iVV9fmq+mpVPVxV75gff0FV3VNVj863V05VAwDA5WDKFrKzSd7V3a9Mspnk56rqlUneneRod788ydH5ZwCAlTVZIOvuU919Yr7/nSSPJHlRkjcmuXN+2Z1J3jRVDQAAl4OFjCGrqmuSvDrJl5Jc1d2n5qe+keSqRdQAALCsJg9kVfW8JJ9M8s7u/va557q7k/QF7rupqo5X1fEzZ85MXSYAwDCTBrKquiKzMPbR7v7U/PDpqrp6fv7qJE+c797uvqO7N7p74+DBg1OWCQAw1JTfsqwkH0zySHe//5xTdye5cb5/Y5LPTFUDAMDlYMqJYV+b5C1JvlJVX54f++dJ3pvk41X1tiRfT/L3JqwBAGDpTRbIuvt3k9QFTh+a6rkAAJcbM/UDAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQASyj7a3kvluTu66fbbe3zn/N7/3L5D+sJ3esz/bPdx2w9NZGFwDAeRy7Lbn/9uTsk8npE0kque6W773mS+9J+uzs8++/N6n933sdsPS0kAEso5NHZ2EsmW1P3nv+a54KY0my/d3zXwcsPYEMYBmtH0rWDsz21w4k64fPf02d09Gx74rzXwcsPV2WAMto80iSmrV4rR9ONm8+/zXdycMfnl37Q289/3XA0qvuHl3DRW1sbPTx48dHlwEAcFFV9UB3b+zmHl2WAACDCWQAAIMJZABT2sl8YsDKM6gfYEo7mU8MWHlayACmtJP5xICVJ5ABTGkn84kBK0+XJcCUdjKfGLDyBDKAKe2bry1p3BjwLHRZAgAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMNlkgq6oPVdUTVfXQOcdurarHq+rL85/XT/V8AIDLxZQtZB9JcsN5jv9ad187//nshM8HALgsTBbIuvsLSb411e8HANgrRowhe3tVPTjv0rzyQhdV1U1Vdbyqjp85c2aR9QEALNSiA9kHkrwsybVJTiX51Qtd2N13dPdGd28cPHhwUfUBACzcQgNZd5/u7q3u3k7y60les8jnAwAso4UGsqq6+pyPP53koQtdCwCwKtam+sVV9bEkP5bkhVX1WJJbkvxYVV2bpJN8Lck/nur5wGVqeys5dlty8miyfijZPJLs2z+6KoBJTRbIuvvN5zn8wameB+wRx25L7r89OftkcvpEkkquu2V0VQCTMlM/sFxOHp2FsWS2PXnv2HoAFkAgA5bL+qFk7cBsf+1Asn54bD0ACzBZlyXAn8nmkSQ1axlbP5xs3jy6IoDJCWTActm3fzZmzLgxYIXosgQAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGOyigayqfr6qrlxEMQAAq2gnLWRXJbm/qj5eVTdUVU1dFADAKrloIOvum5O8PMkHk7w1yaNV9ctV9bKJawMAWAk7GkPW3Z3kG/Ofs0muTPKJqnrfhLUBAKyEnYwhe0dVPZDkfUl+L8lf7e5/kuRvJPm7E9cHTGV7K7nv1uSu62fb7a3RFQGsrLUdXPOCJD/T3V8/92B3b1fVG6YpC5jcsduS+29Pzj6ZnD6RpJLrbhldFcBK2skYslueGcbOOffIpS8JWIiTR2dhLJltT947th6AFWYeMlhV64eStQOz/bUDyfrhsfUArLCddFkCe9HmkSQ1axlbP5xs3jy6IoCVJZDBqtq3fzZmzLgxgOF0WQIADCaQAQAMJpABAAwmkAEADCaQAQAMJpDBsrPEEcCeZ9oLWHaWOALY87SQwbKzxBHAnieQwbKzxBHAnqfLEpadJY4A9jyBDJadJY4A9jxdlgAAgwlkAACDCWQAAIMJZAAAgwlkAACDCWTwbCxbBMACmPYCno1liwBYAC1k8GwsWwTAAkwWyKrqQ1X1RFU9dM6xF1TVPVX16Hx75VTPh+/b9lZS+/P0/yaWLQJgIlO2kH0kyQ3POPbuJEe7++VJjs4/w3I6dlty6veTbCfZl/ylH7FsEQCTmCyQdfcXknzrGYffmOTO+f6dSd401fPh+3byaLL1v+cftpPemi1jBACX2KLHkF3V3afm+99IctWFLqyqm6rqeFUdP3PmzGKqg3OtH5p1Uya6KwGY1LBvWXZ3V1U/y/k7ktyRJBsbGxe8DiazeSRJzQbyrx/WXQnAZBYdyE5X1dXdfaqqrk7yxIKfDzu3b/9sigvTXAAwsUV3Wd6d5Mb5/o1JPrPg5wMALJ0pp734WJIvJnlFVT1WVW9L8t4kP1FVjyY5PP8M0zLbPgBLbrIuy+5+8wVOHZrqmXBeZtsHYMmZqZ+9z2z7ACw5gYy9z/QVACw5i4uz95m+AoAlJ5Cx95m+AoAlp8sSAGAwgQwAYDCBjEvLnF8AsGvGkHFpmfMLAHZNCxmXljm/AGDXBDIuLXN+AcCu6bLk0jLnFwDsmkDGpbXTOb+2t2bjzU4enbWqbR6Z3QsAK0ggYwyD/wHgacaQMYbB/wDwNIGMMQz+B4Cn6bJkDIP/AeBpAhljWPAbAJ6myxIAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBbNltbyX33Zrcdf1su701uiIA4BJbG10AF3HstuT+25OzTyanTySp5LpbRlcFAFxCWsim9v22cJ08OgtjyWx78t5LXSEAMJgWsql9vy1c64dm9519Mlk7kKwfnqxUAGAMgWxq52vh2k0g2zySpGb3rR9ONm+epEwAYByBbGrfbwvXvv2zAGfcGADsWQLZ1LRwAQAXMSSQVdXXknwnyVaSs929MaKOhdDCBQBcxMgWsh/v7m8OfD4AwFIw7QUAwGCjAlkn+VxVPVBVNw2qAQBgKYzqsvyb3f14Vf3FJPdU1R909xfOvWAe1G5KkvX19RE1AgAsxJAWsu5+fL59Ismnk7zmPNfc0d0b3b1x8ODBRZcIALAwCw9kVfXcqnr+U/tJ/naShxZdBwDAshjRZXlVkk9X1VPP/83u/u0BdQAALIWFB7Lu/qMkP7zo5wIALCvTXgAADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADDYkkFXVDVX1X6vqD6vq3SNqAABYFgsPZFW1P8m/S/KTSV6Z5M1V9cpF1wEAsCxGtJC9JskfdvcfdfefJrkryRsH1AEAsBRGBLIXJfnjcz4/Nj8GALCS1kYXcCFVdVOSm+Yf/29VPTSynhX0wiTfHF3EivHOF887XzzvfPG888V7xW5vGBHIHk/yknM+v3h+7P/T3XckuSNJqup4d28spjwS73wE73zxvPPF884XzztfvKo6vtt7RnRZ3p/k5VX10qr6c0l+NsndA+oAAFgKC28h6+6zVfX2JP85yf4kH+ruhxddBwDAshgyhqy7P5vks7u45Y6pauGCvPPF884XzztfPO988bzzxdv1O6/unqIQAAB2yNJJAACDCWQAAIMtdSCz5uXiVdVLqurzVfXVqnq4qt4xuqZVUFX7q+q/VNV/Gl3LqqiqH6yqT1TVH1TVI1X1o6Nr2suq6hfmf1MeqqqPVdVzRte0F1XVh6rqiXPn7qyqF1TVPVX16Hx75cga95oLvPPb539bHqyqT1fVD17s9yxtILPm5TBnk7yru1+ZZDPJz3nvC/GOJI+MLmLF/Jskv93dfyXJD8f7n0xVvSjJP02y0d2vyuwb9j87tqo96yNJbnjGsXcnOdrdL09ydP6ZS+cj+d53fk+SV3X3X0vy35L84sV+ydIGsljzcojuPtXdJ+b738nsHylLW02oql6c5O8k+Y3RtayKqvrzSf5Wkg8mSXf/aXf/j7FV7XlrSX6gqtaSHEjyJ4Pr2ZO6+wtJvvWMw29Mcud8/84kb1poUXvc+d55d3+uu8/OPx7LbBL8Z7XMgcyal4NV1TVJXp3kS2Mr2fP+dZJ/lmR7dCEr5KVJziT58Lyr+Deq6rmji9qruvvxJL+S5GSSU0n+Z3d/bmxVK+Wq7j413/9GkqtGFrOC/lGS37rYRcscyBioqp6X5JNJ3tnd3x5dz15VVW9I8kR3PzC6lhWzluSvJ/lAd786yf+KbpzJzMcsvTGzIPyXkzy3qv7B2KpWU8/mujLf1YJU1b/IbCjQRy927TIHsh2tecmlV1VXZBbGPtrdnxpdzx732iQ/VVVfy6xb/nVV9R/HlrQSHkvyWHc/1fr7icwCGtM4nOS/d/eZ7v5ukk8luW5wTavkdFVdnSTz7ROD61kJVfXWJG9I8vd7B5O+LnMgs+blAFVVmY2reaS73z+6nr2uu3+xu1/c3ddk9t/473S3loOJdfc3kvxxVb1ifuhQkq8OLGmvO5lks6oOzP/GHIovUSzS3UlunO/fmOQzA2tZCVV1Q2ZDUX6qu5/cyT1LG8jmg+GeWvPykSQft+blQrw2yVsya6n58vzn9aOLggn8fJKPVtWDSa5N8suD69mz5i2Rn0hyIslXMvu3x3I+E6iqjyX5YpJXVNVjVfW2JO9N8hNV9WhmrZXvHVnjXnOBd/5vkzw/yT3zf0f//UV/j6WTAADGWtoWMgCAVSGQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQASulqn6kqh6squdU1XOr6uGqetXouoDVZmJYYOVU1b9K8pwkP5DZmpbvGVwSsOIEMmDlzNfHvT/J/0lyXXdvDS4JWHG6LIFV9BeSPC+zteaeM7gWAC1kwOqpqruT3JXkpUmu7u63Dy4JWHFrowsAWKSq+odJvtvdv1lV+5PcV1Wv6+7fGV0bsLq0kAEADGYMGQDAYAIZAMBgAhkAwGACGQDAYAIZAMBgAhkAwGACGQDAYP8PpSUU+GxG/90AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(1, 11, 0.01)\n",
    "noise = np.random.randn(1000)\n",
    "top_x = x[noise>2]\n",
    "top_y = 2 * top_x + 1 + noise[noise>2]\n",
    "\n",
    "# y = 2 * x + 1 + noise\n",
    "\n",
    "plt.figure(figsize=[10, 5])\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim((0, 12))\n",
    "plt.ylim((0, 26))\n",
    "plt.scatter(top_x, top_y, s=12, c=\"darkorange\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFACAYAAAASxGABAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsvX1wVPeZ5/v5ndONWrxJQrJjQJJBEmTuOBdnI2Ej3mJH2HVra6t2ZyzjydRkZzI2xuMMtm+25mbq1vhmpvzHTHY2ixfiELBxsjszWTs4U7Vbt7bqDpbxi0CykRgPMZkJegEkIcASeoMgCfU5v/vH6XN0zunTUgvUINDzqXJJ6j5v3ZWyv3me7/N9lNYaQRAEQRAE4fZh3O4HEARBEARBmO+IIBMEQRAEQbjNiCATBEEQBEG4zYggEwRBEARBuM2IIBMEQRAEQbjNiCATBEEQBEG4zYggEwRBEARBuM2IIBMEQRAEQbjNiCATBEEQBEG4zcRu9wNkQ0lJiV61atXtfgxBEARBEIRpaW1t7dda3zOTc+4IQbZq1SpaWlpu92MIgiAIgiBMi1Lq3EzPkZalIAiCIAjCbUYEmSAIgiAIwm1GBJkgCIIgCMJtRgSZIAiCIAjCbUYEmSAIgiAIwm0mZ4JMKVWmlDqilPqlUuqUUurF1Ot/rpQ6r5T6NPXPv87VMwiCIAiCINwJ5DL2Ign8B631CaXUEqBVKXU49d5urfV/yuG9BUEQBEEQ7hhyJsi01heAC6nfryil/hlYmav7CYIgCIIg3KncEg+ZUmoV8K+Aj1Mv/bFS6qRS6k2lVFGGc55VSrUopVr6+vpuxWMKgiAIgiDcFnIuyJRSi4GfAy9prUeAfUAl8GWcCtr3o87TWh/QWtdorWvuuWdG2wcEQRAEQRDuKHIqyJRScRwx9nda678H0Fpf0lpbWmsbeB14KJfPIAiCIAjCLcS24Nifw1tbnJ+2ld17KSxbs/vwaer3HWP34dNYtr5FD357yZmHTCmlgIPAP2ut/7Pv9eUpfxnAbwGf5eoZBEEQBEG4xTS/Asf/GpLX4NIJQMHG707/Xoo9DW0c+LCT0QmLU70jKOClx9be6k9xy8llhWwT8A3ga6GIi/+olPqFUuok8Cjwf+bwGQRBEARBuJV0NTiCC5yfXe96b1nn3mP38L+jfvB77B7+d1jnGtJOP9rez+iEUzkbnbA42tF/Sx77dpPLKctGQEW89b9ydU9BEARBEG4z5XVO9St5DWILoXyb99ae0d/jwLViRklwKlmJGr3MS6HTN1WVcKp3hNEJi/y4yabKklv7/LeJXOaQCYIgCIIw39jwMqCcylj5NtjwZ95bR8cfYJQhAEZJcHT8gTRB9kLdGhRwtKOfTZUl7Kpbc8se/XYigkwQBEEQhJljW44nrKvBqYpteBkM0/ln43fTvGEAm6ru4VTvlcnqV1V6ioJpKF56bO288I35EUEmCIIgCMLMafoL+PgvQSehtxm0hk1/MeUpL9StQWvNodYeFGBpjWVrTENh2Zo9DW0cbe9nU1UJL9StwTSinE93JyLIBEEQBEGYOZ/9xBFj4Pw89eNpBZlpKJRSDF2bYHTC4o2PzmAqpyI2X6crXW5JUr8gCIIgCHcZacWr7KpZmaYo5+t0pYsIMkEQBEEQZs4DfwBG3PndiDt/Z8GmqhLy4yZAYIpyU1UJiZgjSwwFhlLzJhQWpGUpCIIgCMKNUPtdUGbkNOVU+KcoayuKsbSmft8xaiuLWVdaSMu5AWwNJ3uG2NvQNm/alkrrua8+a2pqdEtLy+1+DEEQBEEQZshUZv3dh097vrH8uEnRwji9w2PeuetXFXHouY2369FvGKVUq9a6ZibnSIVMEARBEIScMZVZP+wbKyROftycd6GwIIJMEARBEIQIZiuGIsqs/9Jja7FsjWEoDAW2hkTc4MnqUgyl5l0oLIggEwRBEAQhgtmKoYhahWTZmq+/3szxMwNoHBP/g6WFvLhtrRcMO98QQSYIgiAIQhqZKltT4a+q1VYWA3CsvZ91pQVYtmZzlVP12tPQRstZR4yBUyHrGriWy48z5xFBJgiCIAjzjGzakTey5NtfVfu029lZmbQ1+XGTHVtWo4Gn9jfROzRKONGi78r4vJqqDCOCTBAEQRDmGdm0I6PWHF1P2ux9r413Uq89UV3KS6k2o2VrDrV0e1W1pE9xjU5YHGrt8RL6Y4bCAGzf/ZK2zqoKd7cigkwQBEEQ5hnZtCOj1hx9cmaA1nODntja934HClBK8bOWbi76IitU6hpuhUyl7gWO+FqcZ3J13PKOjxlqXk1VhhFBJgiCIAjzjGzbkWHhdqp3OFD5Sto6UPnyc9/SPJ6oLuPnJ5xq2sqifAavXWd0wiYRM1ChVUv3LsmbV1OVYUSQCYIgCMI8wrI1ttYULoxTRJz66rKMQigs3B5YURCokAFcHUumibH8uMlT68vR4Im1gV9f58GyQmytMZSi9dyAd7ypoGzZQrb/6BimaWDb+qaiNu5ERJAJgiAIwjxiT0Mbb3x0xhNZhiIgesKTkju2VNDU6eSC7fxqJb//5se0nBv0TPmjE0liqdZkzFDcuySP+upSLK358dEznlgbS9qc6Brkua9WcqzjMkmfgSx/gcnJniFGJyZfvJmojTsRWS4uCIIgCHMd24Jjfw5vbXF+2tZ0Z2Qkyj/mxzX8t5wb5I2PzmAoOPTcRl56bC0/+qCDX5wfCUxIJm24Z/EClhckuHdJHk9Ul6KBNz46E/CIAUxYmn3vd3BheDTw+njSDoixTM92NyMVMkEQBEGY6zS/Asf/GpLX4NIJQMHG787oEm7l6/zQqFfRivKPhQVbY3sfGmhs6+NXl65GtifLixdxsmfYM/8XLYynHeeStDXD164HXpuw0vdqz7fVSVIhEwRBEIS5TleDI8bA+dn17owv4Va+LqQmIVcUJNi5tSLNP1ZbWUws1cKMGQpDKQ582Elr1xBXx5Peccp3DcvWARGncQRVFKaCgoULcLukKmQRW7TAYHlBgqKFcSytscKBZXcpIsgEQRAEYa5TXgexhc7vsYVQvm3Gl/BXvpK2Znj0Osc6L/PU/iZ2Hz6dUfj0DI5GVruWFyT46Dtf46XH1rKpqsQTYPlxkye+spJ1pQXEzeAs5ZK8GNX3L2Pg6ji2dlYmLS9IkB83vHO/tLKQoWsT9A6P8cZHZ9jb0Dbjz3onIi1LQRAEQZjrbHgZUE5lrHwbbPgzYGYLwP0TkwC/vm7zyRln0tE10O+qW8PPW3u8KcqkrRkZm0ABfrmWiBtsrynz7vVC3RoUeEvBLa052TPMhBU0+mvgx0fPMJZ0rmZrp8q2vbqMxvY+TMPgVO/wjFc23Q2IIBMEQRCEuY5hOp6xkG9sT0Mb+z/o8CYYmzsv89MdG9KmJl999zSHWroxDYibKs2z5QofDXx+Zdx7XQGj162AGDMUrFtZgKU19fuOeULwpcfWesKpft+xQDVuZVE+SileT20HcMmPm2yuuoeXHluLBm97gP/9+eIjE0EmCIIgCHMYV1C564p+6ysrMZSiqeMyvUOjjKXyI2wNx88OsLehzVvgfbS9H8NQnDg34MVMKPBM/S4xQ3F+cJRDLd3B1yPEm62he3CUX6SiM6LiKaKCZxt9LVOAxXkmz2ye9LAdneb9ux0RZIIgCIIwh9nT0Ma+9zs8ofTDIx3eSqJYqD1pa7xKl1ttMhSBmAqN03L84heWEDMNugau0XdlnN7hMWKGCkxgrist8KYnXcJrkKLaiuEW5q66NWgIiLRnNldMKeLC79/tiCATBEEQhDnM0fb+QNVKQ8DjtTjP5Np1C1s7Qqu2opifHe/yBFOUV39swmbLGqdVWL/vmDd5mbQ1KwoSrCzKZ1NlCc8/WsUPj7TzUdvnXBwZZ3h0ggdWFLB+1TIONjoVskTMwFAq0L40DRVoYUK0SPMz3ft3O0rruT9OWlNTo1taWm73YwiCIAjCLWf34dO8dqTdE2Hhpd07tqzG1pqfHDvLeNKmZHEeF4fHAr6v5UsXcOnK9YA4W16QoPE7X/PiMNzK1M6t6ZWp3YdPB47ZsWU1hlI0tvfROzzGxeExbA35cYMdWypQSmU1aHC3opRq1VrXzOQcqZAJgiAIQo7IdgoyvK4IoKnjMpuqSvjWo1VorXmntQeA3/5KqeMh65ysJH399WYvFd+tdgVQThXL9hVh+q6Ms7ehLavKVDgstqnzMoee24gG9r7X5lujZAeWjWe7/mgm06J3KyLIBEEQBCFH+KtPU4kT/3Gfdg8BTvvQPefbj3+Rbz/+xdBZk9f5Ze9wxmeIGU4WWDLUu0za2vN+hduLYTZVlfDZ+WHGkjYKOD80Sv2+Y/QOjQaqboYizV/2dks3jdMIrWy/p7sZCYYVBEEQhBwx3d7IqOOStvbEU7b7HH9zRUHk63FT8fwjVTxRXZqWnO9OP1q2Zvfh09TvOxYIiPW/bmvNutJCDOV42HqHxmg5N8ilkWA1rub+osC9Yoai78o4LecGOfBhZ8aQ12y/p7sZqZAJgiAIQo6Iin9w23ONbX2YpoFtawxDkR83GJ2wvcnJTLsmo/jbpx/mGwc/5kTXIJatsbUjhkoWLaD5zGUsy2ZdaQFJyyZmGli2ZnNViRePEVWdClftlEofEPAnYsQMxYaKEl7ctgZTKY529HN+cJTeVAt1qpDXqO9pviGCTBAEQRByRJQ/yy90XBIxgwfLCrG1praiGFAc63CS6xvbnRiLcLsv7Lv66Y4NAOxtaOPtlm76roxzYcT5B8ho2I+qTr302Nq0ql0U/kiNpK1582gnSjnP6nrbLo44hn/TgPODo+w+fDrts8z3CUsQQSYIgiAIs0qUQd0vgsIBqABjSRtbaw49t9F7TR1mSl9VpsrWS4+tpbG9P83c7/q5dvnEkJWqzrmrkZQCQyksW6etWnJZtMDgSysLsWyNaShO9gwxOuGkzl4dtzjwYad3vZPdQ55g0xp6h8e89/2fJSomY74hHjJBEARBmEVcoZTJN7WpqoRELPif30TcSGvTTeermup9/7JvP+5kJThi7OuvN3P8zIAXkaFTaf+/+3oz33q0ip1bK1hekPDaqE7kRSVv76zlnT/ayE93bGDHlgri5mS1y32Wo+393hYBwDeJOT89YtMhFTJBEARBmEUytQBdXqhbQ3PnZT5JCSFDwYOlhWlturCvqraimN2HT3uVt9rKYm/y0fBVtkxD8ULdGrTWvH28i8+vXPcEV9LW3tSjYShazg4QbkbaGj45M8A3Dn6MZWtnKbiGn59wVjdZWnv3MQ2FUgod8pJtqiwJJPPfiC9uviGCTBAEQZhf2BY0vwJdDVBeBxtedpZ3zxLTGdRNwxFOroaxNdhap8VBhH1Vltbecu7Pzg+zrrSAmKkg6VzjZM8QexvaeOmxtZ5QuvzriYDgUsDnI2PRWWU+NE6lzNaOqFpXWuBli73x0RlMpTyRGd4kkIg7vrfaymJ2bKmgqbPf88X5s9NcJIPMQQSZIAiCML9ofgWO/zUkr8GlE4CCjd+dtctPZVB3xcf5odHAzsioilHYV1W/75hXeRtL2rScGwxMPY5O2Bzt6PcGBw42dqaZ8aOWhWfC32I81TucsernF6AxQzE24Tzbqd4Rdm6tCPji/NlpLpJB5iAeMkEQBGF+0dXgiDFwfna9O6uXd4XUoec2etUqF1d8uBWqFQUJdm6tyGqq0O8LCy8MB7ydkpu/9x6vHWn3kvtdYobiK+VFhItPi/NMVhQkprz30vy4d++wgHyhbg07t1awflUR9y7Jm3GGmmSQOUiFTBAEQZhflNc5lbHkNYgthPJtt+zW4SiJFYUJNLD9R8cwTQPLsr1ssnD7zvWFHWrt4crYBGMTNklbEzMU9y7Jo2zZwsDEo0vcVNyzOI/66jL++GtVfOPgx147Mi+meGBFAV0D17yKXRTLl+axvbossurnr+SFd15m4xWTDDIHEWSCIAjC/GLDy4ByKmPl22DDn930JbP1QYXFh2kYaZlkLuH2nesLc71cMUOxfGke5cWLsGxN18C1NDEWlT320x0b2NvQxtGOfgw1GVsRMxTxiJZmftxky5p7s4qluJE8Mckgc1BaZ9dLvp3U1NTolpaW2/0YgiAIwl3AzZjIM53rrwy5Fasna8oiw1z/y7unOdTa42V1TWWwX7+qKODBqt93jJZzg97fi/NMLFszOmF713OJm4o/+mqVk5yf4fOFr1ddXkDMNPns/BCFCxdwX0E+W1KJ/tN9R2LOn0Qp1aq1rpnJOVIhEwRBEOYVN2Miz3RuuBXZOzzGD95zjPVLE3GeqC7lpW1rI6tcmVqF4fadG+Lqx+8TC1/BsjWGYkpR5K/YJWIGMdMReM9sqZyxoBJz/s2RM0GmlCoD/hvwBZz/nRzQWv8XpdQy4G1gFXAW2K61Hsx0HUEQBEGYTabLCYNgtae2shiApo7L9A6NRp4blWpvaUcwXR232Pd+hxcVkeYjK0iwojCBaRjersmkZWMaireOd/FGYydLEnFKi/L5Rc9Qxs8VFna2xsscy1Sx8rcL/e3LGxFU2XyvQmZyWSFLAv9Ba31CKbUEaFVKHQb+AGjQWv+VUupPgT8FvpPD5xAEQRAEj2xM5OHF2oBnoDeVI7b8Yayu4f7No2fSphvdc6PEW37cZHtNWZpw2X34NK8dafcE1tVxa8rW5uI8kz/ctJrmzgFazg14uyP7roxzYXgso8DyG/KdWA3Hg3YjgkrM+TdHzgSZ1voCcCH1+xWl1D8DK4F/CzySOuy/Au8jgkwQBEG4RWRjIs+0WDtpaxbnmVy7bqWFsSqlSGbI+HLT66e6v78q1zs0GtnGdOMu/H6xRNzgmc2Ocd+ytWfYPz84Sm9KxGUjsG5WUIk5/+a4JaZ+pdQq4EPgS0CX1row9boCBt2/MyGmfkEQBGEmzMRgHnXsnoY29n/Q4e1idIVQftykaGHcEzrgVKee3lzB0fb+gEF++dI8nP/M4UVOvHakPeMzff8ffsUPj7STKbfVVFCzahm21pHJ9+HPF46gCE9bRn0PrpjLdE0hO+akqV8ptRj4OfCS1nrE/R8ngNZaK6Ui/6enlHoWeBagvLw8148pCIIg3EVMZzD3izCl4B+7hkjamk+7h9AaXtzm7Jt087oMBfctTbC9pgxLa9746IxXQbs6bnHgw07WlRaQHzc9AfTU+vLAPf0CyX0mN1X/aHs//3LxSpoYW7TAQCnF0kSc+uqyiInJzAJrphWr8GYA4daSU0GmlIrjiLG/01r/ferlS0qp5VrrC0qp5cDnUedqrQ8AB8CpkOXyOQVBEIS7B8vWHGrpntJg7hdsfpK25p3Wbr79uNP+c7uGSRtWFuV7bUFTKd5o7PT8YqMTFpZts3NrRUYBFGV61xCoxIX5zRUFvPVsbUA8RkVpRFUDRWDdWeRyylIBB4F/1lr/Z99b/xP4feCvUj//R66eQRAEQbi7yKYVuaehjc+vjHt/u/6tsEcrKowVoO/qOLsPn6a2sjjSU+UKHQ2BluDmqnsCAsiyNd//h19xqKWbK2MTXPeVvxJxg9qKYg61dGcUYwroGbjGlv/4HheHx1KLvofTqn0SN3F3kMsK2SbgG8AvlFKfpl77v3GE2M+UUk8D54DtOXwGQRAE4S4im1bkoZbugCH+3iV5XmvQH97qxkSYKX+Ye8aEpTnwYSc7tqyesuI13RLxr7/ezCdnBiI/x4OlhWgICEeXuKnIixmMTdhcGAm+Pzph09je51Xq3CXiM42bkBDXuUcupywbcQR+FHW5uq8gCIJw9zJd1lVUdWx7TRmmoSLzv1YW5XsG+TePBluQbx49w9ObK3jr2do0sWLZmlffPc07qcR9yzcgN50YA7C1pqnjctokZSJu8NzWShpDAwJ+TMPwPmu47ZrtdKRU1eYektQvCIIg3DFMF81wtL0/sjoWdW44/0spAgLn6rjF/g86aOrop3twFAVe4v6ehjb2vd/h3csf/LqnoY2Ws5nFmNtC1eA9j7tuaXtNGc8/WsWxzsveZGcYK/WiX2CCM+35zOaKrOImJMR17iGCTBAEQbhjiGoT+ttvhqHIjxuMTtie6HKrW9OdW1tZzI4tFYFK2VjS5pOzk5WqH7zXzs9be4D0fLLG9j40cLCxM01IxRSo1HP8q7JCnn+0ylmjFHoedy/mye4hL29skS/3LBE32FzliNCwwHSzyLJBQlznHrJcXBAEQbij8cdJJGIGD5YVYmudVZZWVFaX36yfiZjhJPRr39/V9xdxsmc4cJ6hYP2qZTy0epkXlTFdJlh44XfN/YVsrronTbjdTG6YZI7lljmZQyYIgiAIUbjVqca2PkzTwLb1DRnM/e23saSNrTWHnts443Pd1t1bz9aicPZAXhweS1vaDU5FbNECg+uWJhEz+YONq2jqvJyxhfjU/qbAfdxqWnhX5qaqkrTpzvD0psvNxFpIJMbcQwSZIAiCcEsIT/bZoYBVgBNdgzR3XuanOzZkLcqma79NNVEYPre2ongy8wsCYszdXenutBxP6tTvGtNQbKoq4bPzw4wlbQwFD6wo8CpP4fuYhhHYlam1xtLO56+5v4gdWyoCKfzC3Y8IMkEQBOGWEJ7sK1oYT2sL2hqOnx3w9kNmw3SJ9FNNFLrnNrb3YRoGh1p76Lsy7okuNxojP26yY8tqDKVobO/jVxevcmU8CQQra/50f/+ey/AzNmbYlWlraDk3yMbKksgqn8RV3L2IIBMEQRBmnSjhEG4PFhL3Vg35sTW80diJJj2VPhMa0KkssXAkhU7dz73v2y3dNPqeKxzy6uIuEv+N+5ayuWrSZ6WBT7uHvePcqUnX1+Xqq9EJ25teDLcIwxOWYVGWaepR4iruXkSQCYIgCLNOlHAIt+2e+MpKjp8d5LPzQyil+PV1C3fOzN0PmY3gCN+rufMyrecGPZGjwBM9MUPRd2WcC8NjAUETjpBwGZuw2VxVEkjfzxQ8S8RnzNQ+bWzrY11pAZat2VhZTHPnAC3nBrxJykxTjxJXcfcigkwQBEGYdaYyy7ttO0vr1FSiTSJm8NCqZZzqHQ6Es2YjOML3OtU7HBBMGkc0rSzK5/zgKL3DY4Hr76pbg2GoyNyvpK052tHP849W8XsHP+YfuwaZ8K1A8gfPQnbp/S2plmZ+3GDn1kovdT889RiFxFXcvRi3+wEEQRCEu49NVSXkx01gMj3ebdsdem4jLz22lqaOyalEdzry6c0VaefN5F6xlDDyNzld0XTouY08WVOWdv09DW1e7pehYEVhgvy4ETjm9w5+zCdnBgJiDILVMXCmF3fVrWFjyie2p6HNC3J1A2PDLU33PP93k6lN+0LdGnZurWD9qiJ2bs0uBFa4M5AKmSAIgjArRIWsTjUpGFXt2TWNQT8Ktyr1dks3n4+MeRW2BaaieNECypYtorG9Hw1869GqtOs/tb/JW/Bta1hRkGB7dVngmDcaO9Pu6wbPAnz/H37ledZWFuXz2Xmn8hdui/orcIZixhUuiau4exFBJgiCIGTNVFN+YS/Xzq0VGfPAHPO7pnBhnCLi1FeXeaZ5V3Bkm1PmntPY3s+FVDsSnHZjefEiL6zVL4781z8/NBqYpvTnfrnHhFmSF+PpzavZVbeGV989zQ/ea/ciMnp9z+Bvu4ZjMdavWiYVLsFDBJkgCIKQNVNN+c3EcL6noS2QXG8o0tp0UcuzT/WOoLVGKZUmCjdVlXCia9CrQtkaTvUOZ3wm//VjhnIqYzVlAZHkP0YBMVPxlfIifvLNh/jRBx08tb+Jf7l4JS081vWjudlmuw+fprGtjwfLCrFsHZjaFAQQD5kgCIIwA6JEl0uUb+xGrhN1jMvohMWh1h4OfNhJy7lBDnzYyd5UBetbj1ZxX0HCOzYRN3hgRUHGZzoaygJbWZSf5t/yH6OBL5cV8vbOWn70QYf3DFdTeWR+1q9a5vm83EiN1q4hTvYMe1ObIsYEP1IhEwRBELJmqim/6QJap1oCHhUPETX5aCi4OpaMzBUzDMXA1XHvuAdLC/nJNx/iD378Cad6h3lgRQHPP1qV8bO4lSx/5S3T580UkwHw0KqiwKaB+n3HpsxBE2EmgAgyQRAEYQZkEl1hb1lUO87f/otaAh4+1p18hMkWoK1hdCIZmSvmF2/OsZoffdDhecj+qXuIbxz8GCvlRQsb/C2teT3UjvV/3tqKYiytqd93LCAop1to7hd1mXLQBEFpHe58zz1qamp0S0vL7X4MQRAEIcVUeynz4yY7t1akCY36fcdoOTfo/b04z+Sbm1YDk4u1X0gJsy3fey9gjg+zoiCRlisGQe/W05tX8ZNj5yJbiqaC6vuLsDXefZ/a3xR4vuUFCRq/8zVPXO0+fHpKQZmp0uXPGAs/7/pVRVkvQhfuHJRSrVrrmpmcIxUyQRAEYcZMtZcyk6HfP2UIThr/vvfbAafa9Wn3EIdauildtpDPr2QWY/lxk/rqUjTw46NnvNfzYoovlxV5IuloR3+kGAOwNHxy1hFf/k0Cn3YPeaGyfVfGAzs1/W1KNzctGzHlnxz1izoJdhX8iCATBEEQssatjB1s7AwIMP9eykyeMFtrYqYCn0ZytJlO/a7pHR7j4shYWmK+ixs3YWnNvvc7Aon8Xy4LereissOi8G8SONTS7VWw3JR+V5BF+clmuux7Op+dMH8RQSYIgiBkJKo1+fqHnV6VC5xpxierSzGUyig0/DEXYaKWa0etMQL44n1LeOmxtdTvOxY4xzlPB8TQb64o4JMzA9N+Rv8mgSdryjJWsKLE1EyXfUuwq5AJEWSCIAhzEduC5legqwHK62DDy2CY6cc0/QV89hNnV9ADfwC1300/7iaIak36xRg404wvblvriQ0Xy9a8+u5p3mntof/qeNraIUNBzf1FbKgo4VBrN31Xxkna2vNndQ1c4+LwmJfxFTMUm6scgRRuLzrXU1j2pCj726cf5hsHP+ZU7zD/231LAGjtmhwUMA24d3Ee5cVTJ/m7RIkpWfYtzBYiyARBEOYiza/A8b+G5DW4dAJQsPG76cd8/JegUz3AT/4KlJl+3A0wVWsyXL3qGrjGU/ubAqb8V9/IuV1hAAAgAElEQVQ9zY+PnvHWGPkxDfjCkskQVtNQvLhtTdpy7af2NwWS9/17I1+oW4PWmjd992g9N8Dmv2qgrHiRl+rvb2G6n8t/H8s3jOCvcLnVL//nimpFyrJvYbYQQSYIgjAX6WpwxBg4P7veTRdaXQ2TYgzAnog+7gaISsnPj5s88ZWV/P0/nqd3yBFKpkEgxkFrTfOZgSlbhTX3L0sTSlHVp7DY2V5T5p1jGopvP/5FjnVc9iYjkzZcGBnnwoiTRRbVQgyvZtryvfcCgvONxk40BKZG3WGDJ2vK0oSZeMKE2UIEmSAIwlykvM6pjCWvQWwhlG+LPqa3eVKUGfHo40JkY0QPB58uzjN5ZnMFltYMXL3u3E7BwniMK6lJRjdF/9JI5glJ5/52Vkb4KLET3m/p30MZJpv1TZ9fGQ+8dnXc4sCHnYGpUXfY4MCHnVMKPEG4GUSQCYIgzEU2vAwop+JVvg02/Fn0MVrDqR87xz7wB9HHhcjGiB6uTj2zucIz07seMluDRgcS9xXRZnyXRNzANIysjPBRYscfG+ESMxRxU6V51LJZ35RJyPmnRv2vi0dMyBWyy1IQBGEuYqS8YL/zkfMzyqhvmLDpL+DZLnj2nPN7Fob+bPZIvlC3hh1bVrO8IEHRwjiW1l7CfcxXzRq9brGutJCa+wtZV1qAxgldDRMznDDX57ZWYtl62vtHYdmaQy3daZOaSVuTFwv+52xxnhNOO1UL0b97M2Yo73Plx02erC5l59YKlhckAq+LR0zIFVIhEwRBuEOYaeZVJrIxopuGQinF0LUJx1v10RlMpXihbk0gq8tKrSjaVHWPV7mKGYoVS/MAfJle0Hd1HEtraiuLZ2yEt2zN119vjkzvz4+bPLCiwFuR5K/oTUV4LRIomjr7A8n7u+rShw0EIRfI6iRBEIQ7hHDKe9R6omy4nrS9OIgHVhTwN08/zIJYesMkatXR05srItckNbb3B45dv6qIjZUlvHakPdAWjBmK5x+pTMssm05Y7j58mr3vtaUtGv/C0gRPVpfxx1+r4odH2md0TUHIFTeyOklaloIgCLnEtuDYn8NbW5yfdnoMRLZk02r0Y9ma3YdPU7/vGLsPn8aynbbj7x38mONnB7g6bnGyZ4gfHmmPPN/f0oNJw7sCdm6t8NqUje393rJtgETMwFCKxra+wPngtBebOi/z0mNrOfTcRnbVreHVd09T+5cNbPzLBr7/D7/CivB1HW3vT/Om2RqGrk1gKFgQM7xrvvTY2lsmxqK+Y0G4EaRlKQiCkEuyyRPLkmxajf62pmEoftEzxOiE7ZnnNdBydsATN6MTNkc7+r3craPt/dRWFgNwtK2PokVxxoct3/HOJOUHf/Ioxzovczx1rURM8WBqj6ShFCdT940ZyrsvOBUy/3O/+u5pfvBeu/f+D95r5+etPTxRXQpMLh2vrSwO7MF0ud1G+5km9QtCJkSQCYIg5JJs8sSyJJvMK79A8Ae4jk5YvPZ+OyWL89LafpsqSwLnfdo95DxuhmpP35VxvnHw44CwG0tqb9l2/b5jjE7Y3jWWL81DKadiVV9dFnjud1p78N9F4/jO9r3f4Z1/qneEHVtW89xXKzna0R8QfNl40GbLexeFJPULs4UIMkEQhFySTZ5YlmSTeeUXCGE9NWFpLg6PYSrHjG8oWL9qmZeK78/dmgpHJA2nCbvaimJ2Hz4dyAbLj5s8tb7ce+awOJrqHi6jExZNnZe9dmQ4bX86o30uq1iS1C/MFiLIBEEQckk2eWKziF8gJGIGMUNx9fqkb03jGOFXFuUHzO+bqkoCLcFMy71dlubHSVqasaTtCTsNvO6ftCyYXI/kEhZH//vKpXw+MoY1xb3CQmemYay5rGJJUr8wW4ggEwRByCVuntgsrDPKBlcgNLb3YRoGp3qHg4+jYHtNWZogeaFuDc0+T5gCFi0wuG7ptMBVgOVL89heXZa2e9JfZVtZlJ92n7A4srVm19fWeNET77T2BKIt4qZix5ap88SmI5dVLEnqF2YLEWSCIAg5Yja8S9lcI+oYDWmJ9uBUx/zixn9u79CoVxWztGP4j6qSGQpipsmuujVT7p6MEj7+Y2KGondoDA289WwtAM1nBgKCzElmurnvUapYwp2ACDJBEIQccSPepbC48md+ZbpG1H3CuyjBmXB8srrUEzNu2Kprzo8ZeN6vqJalOy1pazjZM8TehrbAs2QSPv7PVFtZzI4tFRxq7abvynhgR6QGTqYGClyStuZQa48XUPtp9xA/O95FWfEi7NTmgOkEmlSxhDsBEWSCIAg54ka8S2Fx5V9yPTph8XZLN42hSlFjW1/omC7qq8u8SlQQFbiXf1IyacOKgjxWFCboHR7j4vAYdsr8v3CByXjS9tqXoxM2je196NTndJ9nV6o619jej8YRaeHPtHNrBSsL87mQqoS5343WpMVaxAxF/9Vx775JW3NhZJwLI85S8E+7hzjU0s2TNWWzOj0pCLcaEWSCIAg5Yqa5YZuqStJEnH/JdcxQ9F0Z58LwWKBaZoREyKWRcbTWrCst8Dxh4Iay9gNrvb2Q4UnJ+upSms8McCHVSjQU3FeQYPDX1wNesvy4Gbkk3N8qjarWueIr6rvREGhnJuIGYxN2pIfNJWnrQJVNqmDCnYoIMkEQhBwx09ywT7uHSMSNQGTEk9Wl3pqh84Ojnr/KX3HrGRwNXNPW8PMT5xm6NhEQXH5R+Oq7p70Klct9BQkOtfYEXrc1jIxOeLli4KxQemazsy7JL7Qa2/voHRrLSnztyvDd+F8Lr2OKmwqto2M5JANMuNMRQSYIgpAj3OXU4Raev63mrx4lbc3V8fTICNcDFd5l6YqrqCadgkC70hVRrvD5eSiQNW4qBq6OM5YMih1DkXFxtwYvKsNQTqDr51cmxZybyh8lvjL5uvyv+Stm+XGTHVtWOyuZUhOkXQPX6Lsy7olXyQAT7mREkAmCIOSQ6Yz9/uqRS6bIiHCkhSvyfvsrpex7v93L8npo1TIerlgWWADuiih392Lf1fG0Zw2LMXDyxf7m6YfTFne7z9PceZlPzjht0d6hYMXt3iV5U4qv6YiqMLrXAmYcECsIcxml9dxfhFpTU6NbWlpu92MIgiDMmPp9xwJtt/Wrijj03Ebvb1dUvN3S7VV7EjGDB8sKsVJThN96tIrXjrRHTl4mYgbrSgvoHhxF4awmenGbI0zCYsU0VKDK5mdFQYKLI2OBFueKggQffedrUxrlw5/PJT9usnNrhbQQhXmJUqpVa10zk3OkQiYIgpBDMhn7w2b+D/7kUX54pN3xYQ2PeWb8U73DNHde5p+6hxhL2pzoGmTRgpgnqMaSNi3nBlMLvg0+PnOZ7T/qwzQNLxZil69NGhWHAXBpZAzTUNipMlsibrC9pmzaqcVNVSWc6BoMCLlwe1QQhOkRQSYIgjDLROVuNXUG22qZWpka2Ptem28puM2p3smVRraGK+NJz/jvvgaOOPNPVQKBa19P2vQOTw4AKOUGrzpBsJaliRmKe5fkpa08ykQ44T8RN7z2qCAI2WPk6sJKqTeVUp8rpT7zvfbnSqnzSqlPU//861zdXxAEYbZwfVf1+46x+/BprCmWPLphq3vfa6Pl3CBvfNSJofAWY5uG8iInwtOI4FSwwlEUD6woIFyoundJHssLEmmG/vCjude2bM0j/+lIwOe1aIFJIha8gt+/lk2ml2ko/ubph1m/ahmL80weLC3k+UerIr+XbL9DQZiP5EyQAT8B/o+I13drrb+c+ud/5fD+giAIN4UrIjZ/7z1+kBJYe99r43dfb84oKMJhq6MTtie2XMKREzFDUVtRzPf/4Vf8y8UR73WlJk3161ct80SZ205cWZgfmJRcnGeSHw/+az1mKB5eXczvHGhKM91rrXmwrCjt+JlOK752pJ2TPcNcHbc42TPMD4+0px3jVgRbzg1y4MNO9ja0zegegnC3kzNBprX+EBjI1fUFQRByjSsiLgyPeROMtobjZwcyCoqoCldY4IQjJxJxAw3se7+Dq+OT/q6HVi3jpzs2sCBm8NMdG3jha2tYv6qI57ZWsqtuDZuqSsiPm4Bjov/DTavZubWS+5bmBSpnn5y5zPGz6cb7L60sTBOW7mTkTIgKfr2RYwRhPnM7PGR/rJT690AL8B+01un/lgCUUs8CzwKUl5ffwscTBEFwyGSAtzVeCGnYnF9bWRzI5lq/almawAnX1pYm4jR1XE4LPLW19tqGUdERmYJnf9bS7d0jaWt+eWGEMHHTaTW+dqQ9MHSwvaYMgN2HT2e9zHumS8UlM0wQ0rnVgmwf8ArOv49eAb4P/GHUgVrrA8ABcGIvbtUDCoIguPhFhLsWCJyKliso9jS0sf+DDm8Csub+Ip7dWklTZz+1FcVo4Kn9TQFhU19dyr73O0jajom+vroUpRSfdg95oszfOgyLvnAMxlvP1nretK+/3uyl+bvXeWBFAS1nB7wqn6Hgj75axYKYESnqMg0chJ/D/TzZbCTI5hhBmM/cUkGmtb7k/q6Ueh34f2/l/QVBEGaCX0TUVhQDKm1a8mh7f2ACsuXcIBsrSzj03EZ2Hz7N6xHC5qVtazFT65D819Ja805rD+DkiT3/aBW7D5/mZy3dfD7itE1PdA1yqLWbwV9fT01gTl7X9a/5ScQNkpZN9f1FkVll4cpbpoED9/pRQi2b4NcbDYcVhPnCLRVkSqnlWusLqT9/C/hsquMFQRBulkxVnZmc41agHIKCIpzDZWt463gXltb8+OiZtF2PGiIrW3sa2jjWcZkna8q8Z4wKcQ0n4vsFU9i/BjA2YdPaNZR1UOuehjY+vzKZ4u+v1EX5wERgCcLskDNBppT678AjQIlSqgf4LvCIUurLOJX/s8DOXN1fEIQ7FNuC5legqwHK62DDy2CYN3Qpt4XX4gtZDa8uimKqdUdR7cPmzst8fGayMnVpZNxrSbrEDMW/XLzCp93DJG2dVtnyLxg/1NLNkzVlgeXdmfD7sTZVlQT8a4sWxLgyngSiBVSUWD3a3h94br/JX3xggpA7cibItNZfj3j5YK7uJwjCXULzK3D8ryF5DS6dABRs/O4NXSoqgiJcpYqqmE1VCYoSa3/z9MNUv3LYEz8aAqImbiq0JjBB6V53V92aQIswaWt6h8c48GEnRYviU36+xXkmSxNxLK2xbJ3m07J8K5aiBFTUZwmLLn9av/jABCF3SFK/IAhzi64GR4yB87Pr3RsWZFERFKZhBESI1hqlVECghUVJbUWxN3XYOzQaEGtvt3Tzdks3V1NiDEDheKaStiY/blK0MB4w2rvPYijFq++eDrQIXUYnLCZGpq6OXbtucXXc4o2PzmCqSY+Wv5oX5VXzfz9h4fnWs7UZRZf4wAQhd4ggEwRhblFe51TGktcgthDKt2V1WlT7LdzCW79qGZatAyLkUGsPQ9cmAlWiqEqTa86PGQpT4U0sXgot5Aa4b2keT9aUewMA/kqVO61pazjZM0T3wLW0uAuX1KyAh1/oGQpf5S/dn+ZW/qYSUFEtSBFdgnB7EEEmCMLcYsPLgHIqY+XbYMOfZXVaVPstPCWpgXdae7w9kPlxEwVpFa9ddWsCoqR+37FAS3Fxnsm16xa2Tl9VlB83eWp9eerc9ErV+cFRr1o2OmFTmDpndMIiETOIm4bX+vSzJC/G729chaGcSU9DKU72DDE6YZMfN9Mqf9l45aQFKQhzB6V19P8zm0vU1NTolpaW2/0YgiDMYer3HaPl3GTO9PpVRRx6bqP3t39i0b9A29I6YMCPGYo/frQqIGb852ZqQcYMxT2LF1BevAjL1hn9af5rJWIG60oLAnEUmslqmp/w57metPnGwY851TvMAysKSNqa1ik+vyAItw6lVKvWumYm50iFTBCEu4LpJgD9fin/Am3L1vy8tccTWElbB0z8lq2xtaZwYZwi4p5oev3DTi9/bElejG9uWh0QU5+dH6a583KaOHOrUo3tffQOj9FyzonMyI8bGAp21a1FAW8ePeMNAfiDaF3c/ZGjE87+yHWlBV6lTSYgBeHOQwSZIAh3JFHxE1O136Yy6pcuW8jgtete+88vZvY0tAUmFV3R9MmZAY6nJjiTtuNRa2y/7Im+saTtRWF82j2E1ppvP/5Fz6Nlac3e9yaXcLtLyF96bC1KKZIpk5qh4MHSQu/zuJ/7YGNnoNVq2TY7t1ZI+1EQ7lBEkAmCcEeSKSssk29qKqN+nqkoWZLH8OgED6wo4PlHq7zzMkVgOJUzUq87cRpKRQfOJm0ngf/bj3/Re+3nqUR+P/4AVn/6v3+npf9zu+THTTZX3XPDZvwbCc8VBGF2EUEmCMIdyUxT401DsatuDRr4qK2PX1266p0/bmkuDI+lJh+H+eGRdu9amSpr54dGA8MBpmGkrS2airB7N24qGtv70UBtZXHG9mt44fniPJNnNlfcVEVsqiBcQRBuDcbtfgBBEOYXlq3Zffg09fuOsfvwaawMkQ/TsamqhPy4k+Af5ZmKuo8rPFq7hgK5YRCMkHj9ow6272+ift8xbK3ZsaWC9auK2Lm1Ag3s/6CDC8NjJG3NkrwYO7ZUYNnai8II4ywQLwu8Vl9dSixVhVKATu3BPPBhJwrYuXXynn6xFf7cz2yu8PZJ3ihR4lYQhFuLVMgEQbil3Gg1Zqaesaj7hKtLLovzzFQ+mdMm/PV1m09S/q9TvSPs3FrhTSzW7zvmtROdY5MYyhFKn3YPedOapoKaVcuwtfaez/8ZaiuLef6RKpo6w1EYFk2dlzn03MbI7yUXURWyEkkQbj8iyARBuKXc6ILqTPseg4u/HSxbB9YRuffxB8W6JOIGf7hpNYZSvNHYGVhvFPWMUcvE3YR7rR2vGDgRFi9uC3qx/JEXfqEXjtWYShDlIrhV8sgE4fYjgkwQhFknk0ncsjWGobyUeTfOIRtTeTi2ond4jP0fdERGS+xpaAusI4oZyhMazZ2X+eTMAJrJCcYXtzktPw1phnl3xdH1pM1rR9ppbOvjvoIEF1Oes7yYwlCKp/Y3samqhI++87WM7cOwGHXT9Rvb+lhXWoBlazZX3XpBJOn8gnD7EUEmCMKsk6ktuaehjZPdQ9g6GOeQTRvT31ZzGUvaXvSE/7yj7f2BdUT3Lsljl08Uuu/YOpWiH1qe7WaEXfSM/kN84+DHXu5XImawPtWO9Cfmh589LDTDZn1/un5+3GTn1goRRYIwTxFTvyAIs04mk3imOIdsTOUv1K1h59YKlhckPDN8eJ+je17Y+L69pswTXUaoemUoPPP/noY2dtWt4Z0/2sSKgvxArMWJrsFAxpitNYee2xjwnoWf3RWamcz64b2aYqYXhPnLtBUypdQu4G+11oPTHSsIwvwiU6sxyiQebleaBvQMXGP7/qa0CIlMHioNrChIUFaUT/fgKFfGJhibsL3z3EiKqVqAlhXc2P2P3UOc6HLM+Kd6R9Bao5Ti/NCotwgcIGnpyGecyhAfFppNnZcdrxnQ2N6PYSjy40ZkIK0gCPOLbFqWXwCOK6VOAG8C/5++ExZgCoKQczK1GqNM4v52JTgxDxdGxrkw4ni9YoZiRUGC7TVlaan0jW19XBiZbCHGDABHHPnP84e9ZmoBbl5zD7+8cMUTShO+rIrRCYuftXTTf/V6oOUJjjBLxJ2mgj88dipDfJRY839niZjBg2WFgUnM6ZAQV0G4O8lqubhy4qcfB74J1AA/Aw5qrTty+3gOslxcEOYelq3Z8r33Aku2p1poHV7+HcVUC8GzOS98j8V5Jt/ctBqApo7LXlzGD4+0R05UmoqMWWKAVzEzFKxftYyf7tgQOeHpj7ZQKJo6J8XaU/ubplyCPh3hiUzxnQnC3CNny8W11lopdRG4CCSBIuAdpdRhrfX/NfNHFQThTsQvNgxD8fmVSTHmTjJmwl8tioXM9ZnOz5Qb5h6ftDWJmIGhFPX7jgVagABXxy32ve/8/0a3JelW8fwTlTFDkYgZXL2efq+4qbw1Sf5hgONnB9jb0JYmhsJVQ3+GWfh7uJE25Y3GhgiCMLfJxkP2IvDvgX7gDeBPtNYTSikDaANEkAnCnYhtQfMr0NUA5XWw4WUwzClP8YsNv6EeJicZM/GtR6to7rzMqd5hfnP5UroHR7ngq65FnR81WQmw/v4iaitLaOrsD0w5ui3AU73DXvXL33ocnbB4u6Wbj9r6MAxFQX4M04CliTgjYxORz33P4rxAFdDFzR8Li6HpBNPNZn5JiKsg3J1kUyFbBvy21vqc/0Wtta2U+je5eSxBEHJO8ytw/K8heQ0unQAUbPzulKf4xYYbXWHr9EnGKF470u7FRvzi/AjrSgsYujbheanKli1k+4+OYZoGdipXzE3jf+t4F5dGxtE4lbHaymK+/fhaYC31+455FbGxpO0IvhUF/CIl0tyJTNdv1ndlPCAEgbTWpcvDq5fx0OplvPHRGUYnrIDJ381QCzOdYLrZzC8JcRWEu5NpBZnWOuO/obXW/zy7jyMIwi2jq8ERY+D87Hp3WkHmFxt+Q3ptRTGW1tTvO+YZzQFeffc077T2eELGXzlKWhbrSgs41TvM0vy4V+Vy8bcXG9v7uZgy/ydtTVPn5chnAkdcneweYl1pAd2Do6A1ZcsWYmvoHRqNrHZF8dCqIn66YwMAplIc7eintqIYfJ6w5x+tYvfh0wGDfa4Fk4S4CsLdiQTDCsJ8pbzOqYwlr0FsIZRvA6J3Rr52pN0zqe/YUhEwqZuGYvfh0950o7vWqHTZQk6cGyBpp986ETeImaZXMbt23SI01Bho92WK0XAnMF1h51a6xpI23YOjXgVuaDTpLQbPNCQQNfHoVvzSBZDze3gVkvIdGxZMMh0pCMJUiCAThPnKhpcB5VTGyrfBhj8D0k3pzZ2XPeEUZVKH6LVGF0fG0kQWTCb0+0NR/e1P/3HuyiJbawoXxikiTn11WVq6f37cZF1pgfec+XETRbAi93ZLNysKEqwrLSBp2cRMw/vp5pU9nxKfje39aJhWNM3EYH+jS9UFQZgfiCAThPmKYTotylCbMiwyTvUOR4qO8MSlf7oRiBRj7uu21pHtT8u201YW/d4bzV5wq+sHi0r3t2ybHVtWcyjVIl1RmODzK2MkbSeuwvWOTRUVkanilYmZGOwb2/rS9liKIBMEwUUEmSAIAcIi44EVwcqTKzqiAk67Bq7Rd2Xci6NYV1pAz+AoI6FEfbclGPZamYYTX9E75Pi8RidsWs4NeuIuaWveae3m24+ntzE3V92DBq9NeWlkzDPgayanLTOJyk1VJdNWvKLaudn6xUwzuKnONGRznSAIk4ggEwQhQNiU/nwqSNUvOixbc6ilO2234wd/8ijfOPgxp3qHeWBFAX/z9MMsiDktwb0NbWniy++1smzN7sOn01YWhSttw6PXM65Hemp/k/dM4YBX/0RolKg81etMfubHzYwVr0xtx+kqXZat6R64lvaaIAiCS1ZJ/bcbSeoXhNkjW3P5VMd9/x9+xQ/ea/dEU8xQPP9IJc1nBmg5O5ASPgY7t1Zm3ZbLNpV/8QITSxNqdTotUFtr9r3fkbb2KJNhP5zsX3N/IZur7kkTji7h45cXJFhZmD+tSX/34dO8dqTde66YofjjR6ukZSkIdyk5S+oXBOHOJuz3cjO6pvJJTWVCf6e1J5Cyn4gbaPDEGDjtxpmkyE+Vyu9naX7ci64YS9p8fGYAgE+7h3juqxXcuyQY5Bo3FcsWLeCh1ct4advagGiKantOVfEKbxtwfWnT+c2OtvcHROJ0IbqCIMw/xMQgCPMAV1y1nBuk5eyAZ753fVJRhP1Ub7d0e222cB1oaSJOU8fltCnJcMvPbUvW7zvG7sOnA227TVUl5MedTQGJmMECM73a9PDqZTxRXeod5ydpa/Z/2EnpsoXkp5aAxwyF1tA7PMYbH51hb0Nb4JwX6tawc2sF61cVsXNrxbQiyX/8vUvy0nxpmfB/tmxCdAVBmH9IhUwQ5jo3sOIozFQJ+5kmA2sri2k9N+hVwvqujHu7G5+oLvVagzFDUV9dhlLw2flhxpK2t3zb9ZtlU53ze9dqK4p58+gZrluTFbPFeaYX1KqAQ609XApFa0xYmpPdgzxYVoStNecHJ4Ngo0z6U4WsZmrZuseHl3xPNWEp6fqCIEyHCDJBmOvcwIqjMJuqSjyxpHC8T8sL8j0zfCb8bcmkrXm7pZvGVEDs849UBQJicZ4szX/lFy7+rLGwQDINxa66NWjgZy3d/Nq3zshQ8IebKgCn2neotYe+K+OR0RpjSU3XwDVWFuZTumwhg9eue3su3SXk2QSzTpcbNhORJen6giBMhwgyQZjr3MCKozAv1K2hufMyx1Mer4FfX+fJ6rIpIx2aOi6nXefSyBgXhsf4tHuI5x+p8gJiw+f6zfDTVecyVdD83Lc0wYvbgmGwU+F6u/xmfv8S8mwyxqaLwBCRJQjCbCIeMkGY65TXOauNILDiaCaYhsKydZrh3o/fZ3bgw05MQ5GIOf+KMJRjjg/ngYEjxr7+ejN732tLndvhebUsW2MYCrcQlRdTrF+1LODZyuRvc/F7rqKM/zFD4drNDAVL8mKet8uN4zj03MbUZoDpvXMuYd/XVC1JQRCEm0UqZIIw18mw4mimTLUP8mh7P71Do4GK0GfnJxd0K2BkbIIJn6drZGyC60mb3zv4MZ+kJh2dcyenK/c0tHGye8irjH25zFnY7W8VZqqgxQzFvUvy2F5T5rUDw1OO9y7Jo766FP/Cb0tr3vjoTJq3ayap+iC+L0EQbi0iyARhDpJuKP9/MGfYpgwTJTD8LUBTBfdJ/vq6zYmuIcCpiIWHHscmbL5x8GNazg4EXvdPVx5t72cstV3c1mDZdppR3gj5uJYXJFhRmB+ZAxb1Gdzqn2rA87eFF6BnOncqsm1JytJwQRBmAxFkgjAHyfR0zJYAACAASURBVNUiag1oPWnW91enwsn2QCA7y9JO23IidWDS1p4nzU/N/UVM2Da1f9nAlbEJYobyViaZhsH+DzoYS9qc6BqkqaOfrlCC/fKCRNrycpdMIin8fUUtQM+V50uWhguCMBuIh0wQ5iBRhvKbJewR29vQFvBJRRV1YobyFnrnx02+Ul7k/Q3pa40eXr2Mh1YX88MjHVwYHuPquIVla5YvzWNdaQGneocDFbNPzg5yaWQ8cL/NVffM+LPl4vu6E+4tCMLdgwgyQZiDRBnKpwpVzYYo4eAGndbcX8h9BQlPlJkGrChI8PwjlTz/SJVnwv+bpx/m3iV5kddfvjSPh1YvY/+HHYG4DA2gFCd7hrk6nj4d6T/2RhPsb6cBX8z/giDMBtKyFIQ5yHR+r3/sGuRgYydLE3GeqC5NWwkU5WuKMrW7bTwN7P+gwzPV19y/LGS+n2zBPVlTlhY9kR83KS9exBsfnfFamn4UREZVmAYonJZmImZQtmwhT+1vyujFyuTXup0GfDH/C4IwG4ggE4Q5SNjvZNmaQy3dAb/X1XGLq+MW+97vwFTO8a5g+VlLN31Xxkna2vM1hYXD849WsfvwaW/C0t9KtLXOaEx3r9PY3odpGFi2ZnNVCY0ZdlE+tKqIhyuKvclHfzZYbUUx7oRkNjlhmfxatzMTTPLIBEGYDUSQCcJUzMLaotlgT0Mbn18Zj3wvaWuOdvSzq24NX3+9ObDgGyb3UO6qWxMQDv4Efb/0SsSNKdtuUWJxT0Mb54dGPQO/P7Li+Uer2PteG4UL4xQRp766jBe3hatfa6nfdywtJywscqYLaxUEQbhTEUEmCFMxC2uLZoOj7f2BiUc/MUOxqbKEPQ1taWLMxb+HEtIrbu4phoIHSwu9tpsrthrb+jBNA9vWae1Ef9UqZihWFCS87DB3dZI/F8zVYW51bqqWapiZZokJgiDcKYggE4SpmIW1RbNBOKtrRUEepOpa9dWO+Hlqf1OkGINgFc1taV5MLd32Y2voGrjm+bhsX8iqy6fdQ2it+fbjXwSCVaukrVlZlO+1T3cfPs3Bxs60qpaGtNZjNl4s8WsJgnC3kjNBppR6E/g3wOda6y+lXlsGvA2sAs4C27XWg7l6BkG4KWwLlIkzjGzf8NqibJguXNSyguuEVhTm884fbfLOe2p/E0ZqhZDfU+/PAHOraNPtgnT3VZ7qHaFoYTztWGdtUo8nyDJVraLu5b7fmKH1OJ0XS/xagiDcreSyQvYT4AfAf/O99qdAg9b6r5RSf5r6+zs5fAZBuHGaX4ELnwA2YMB9629obdFUYmsqE75fdGxecw+/vHDFEz1uVpdf9CRiBgsXxLgynvTOS8QNfuO+pWxOLfx+an9TmsBKxAyWLV7AxeGxlKHfeX10wqKQOPlxc0oBl6lqFd47uTjP5JnNzv5KDbe09Shp+oIgzHVyJsi01h8qpVaFXv63wCOp3/8r8D4iyIS5SlcDWKOpP2zQ1g0Z+l999zT73u8gaWv+sWuQpo7L2FpnbAlGmdWzET1jSZuiRQsYnbAml2tP2GyuKvGuFbULcntNGY3t/fQOBVuYMUPxxFdWEjMM3m7p4tLIuLdjsr66zDsuU9UqXDl7ZnOFd8ytbj1Kmr4gCHOdW+0h+4LW+kLq94vAFzIdqJR6FngWoLy8/BY8miCEKK9zjPzJazNuV1q25tV3T/NOaw+XRsa8qpOl4ZPU7sdMLcGoilG2oufJ6lLeae2hN+UPc71jUwkh01CBipUfQxm89NhadtWtYW9D24wE1FSi61a3HmU6UxCEuc5tM/VrrbVSKmPUuNb6AHAAoKamZmaR5IIwG2x4GVCOkb9827TtSn9bzDAUJ84NkLQzH++0BGOez0vh7HF0JxSnu09jWx+GoShcGKdQxyhbtohjHZcpXbaQwWvXGZ2wAyn//pbd3z2zgdeOtHvm/ee+Wklz5+XAbsqkrWnq7Mey1wTODS/8zsRc8nvJdKYgCHOdWy3ILimllmutLyillgOf3+L7C0L2GKYzUTnFVKVfHF0YGfN8WIZK3/PoXTb1Xn7cpKxoIX1XrgOOgKmvLp1WwESZ5WOGou/qdS/x3h+8amnN5u+9F/CoNXde5mTPsNfCc//2P3PUIMCd2u6T6UxBEOY6t1qQ/U/g94G/Sv38H7f4/oIwq2SaWpxqzaStYUlejG9uWs2xjsl8Maciddk7zt/2VOCtSAqb5d1zXcaSNrbWHHpuoxP++kGHl8IPpITVcKCF5/8bYNECgy+tLEh5y0YDxza296Fh1g3yuTTez6VqnSAIQhS5jL347zgG/hKlVA/wXRwh9jOl1NPAOWB7ru4vCB45TNuPEkfgJIStX1XELy+MRC7U/vX1JIaaupW2p6HNGwYAvBVJm6pK+LR7KCDCFI7o8EdcuM83FuqbxgzFAysKvApZftxM+/tLKyf/jhkqEJ9hGkZOKmZ3QyVOEAThRsnllOXXM7xVl6t7CkIkGdL2/RWZ2spiAJo6Ls9osbVfULkYCtavcpZzA+xtaOPtlm4u+IJYbe20z/7umQ00d17ms/NDFCyM81G7E5r6Qt2atHT+pK1pbO+jtrIEFSoc3bc0jydrymnqDO6pPD80ShiloGZVEQ+vLuZYh7OPMmnZrCstiNxLmbQ1KwoSrCzKnzJD7GYR470gCPMZSeoX7n4ypO37KzKfdg85b2fIAYPoCk6mRdt+47s7pfi7rzd7pnl3X+RrR9pTlSibX18fo3dojH/qHuJQSzeERJehwDQM3vjoDBO+9NeYoVBKoRS89Wytt67Iv6fS30GdsDRvNp5l59YKNlXd4x2XHzfZudWJprC09qpwbszFtx93vo9cZYiJ8V4QhPmMCDLh7idDfEV45Y/LTBdbZ+NNMg3FT3dsSIuOiApqTdrai63w84WleVi2TqvGuce/dqSNnx3voqx4Eb/0ecKi7GyjExZvNHayNBHPsio1eZVcGeTFeC8IwnxGBJlw95MhviIckgqkebD8TFXByWTA97c9o4zlUX6wKGKG4snqMprPDHhTmqYB/o1KSRsujIxzYWQ8cG6mic+r4xZjE3baeiVwWreZhg1yZZAX470gCPMZEWTC3U8ovsKyNXsOn6axrc/zTW2sLAaU58Ga6WLrTAb8qcSFZWtsrYmuYQW5d0keGjjZPeTFaiyMB9ckRRE3FXkxg7EJ22s/KoXX8gz7w9zPdDPtQ1lTJAiCMHNEkAnzDr8XzO+bcrixxdaZDPiZ4iEsW/M7B5o4fnZw2udNxA1vvZE7MWlrWJyIMWHZ3mthr5h7n6vjTgVwRSp01vKta8qPm9RXl6KUotE3UHAz7UOZlhQEQZg5IsiEeUcupvnCrceYoaaMh9jT0JaVGDMUPFhaGLmQ+8nqUgylONrRT21FMc2dAxw/OxAQZf7U/ZVF+eyqW8Or756mcGGcIuLUV5eh0bwe8Zw32j6UaUlBEISZI4JMmF1ymPk1Hf7UfNM0sG0dqEy5758fGo30TWVz7UxtuBfq1qC15p3WHgDqq8s41hEUJv6KWW9EHEUUtoZTvcPsaWjjW49WRe6hdMVO/b5jGZuf/tR9f3XMUNDYfnlWBZRMSwqCIMwcEWTC7JIh8+tWsKehjf2hVPpTvcNexcffSvO38LJpx4UjMg61dPNkTZknzExD8e3Hv8i3H/+id446DJ+dH2YsaWMo6B0e48CHHYymjPRhli/No7x4EV0D17w1R+CY71870u7d0422CBOViRYzFPcuyfM+p3+q0xVfsy2gZFpSEARh5oggE2aXDJlft4KoVPrRCZu3W7rZlQpa9cdcrCzKz7oSFD63d3iM/R900Nx5GStUiXN5oW5NYGG3u+fSvcbypXmoVMJrfXUZL26brOTtbWjjjcZOL+XfveeBDztQwK66NWkVu2wy0aLE165ZFlAyLSkIgjBzRJAJs0uGzK9bwaaqEk50DaZFPFwYHuM3Xv7/27v34KjOM03gz3tON2pxsS4IX4REhJDsXTuLEyMMQkCChV2ZnanJboLteHZnMzsO4NgBe7y1k0ztpmZ3M7U7U95dPPZgBow9SeVmBzKpTW3NZIIxthGXYEGy2DgZSYiLhLCR0AUxSO3uc7794/Q5Ouf0ad1Q95HUz69qCpC7xaeelP3U+73nff8eixYUoDCqYShhTrgSFFR9Gk6aTtgKal63w5V9HvfScXuYq7vK5n7fMw/eCQVg56E234w0K2C+3tzhWRaeqe/Lf9U62rUnAxQRUXgYyGhqZZj5lQt2RerEud60XqqEodDVP4zFxTGUFxeOuxLk7ktbXlHkuU50z/eyr//8lav6ZQudIBeLaLi3stj5HlbFq90JU0HhaV9zR9qQWPd1pvvvDgpUmZ54ZPgiIppeGMhoavlmfmU0RvP/ZGZZuafhuytIbgNDCRz5xsg61bH+Hv+IjM3rljpPNmoiON3Z76m4+XvNFs2f49kRafdx2Xst3WEqKDw9XFfp9MVpAsybkz57bLRqH594JCKaGRjIKBxjNP9PdpaVff22rbEWf/lGC3a9fdaz9/Ge8iLP68f6e/yB5lj7Vex7Yo1T0RptFVLSVLh8LY7+oaRn1lmmJvqg8PTalnrPFaN7hpi/YT8In3gkIpoZGMgoHGM0//vDyWhDVoPomkABMFxhrLwohu8+vsrzurEqSA01Zc6TkgDwm8uD+F8//0c8s3Fk0bZSIwNZ/a8P+r5PbajB8farONM1gHvKi/Dkhhrnvf7w5O/vMkwFPVWhs0OgYSo89vJxfNA1gLvLi/C9x1dhTkQDwCceiYhmCgYyCscYzf/+cDLakNVMfnyyE6bva3ZQGW0mmXsvJZRC6bw5Th/XYDyJlw614cS5Xk8/mT0K44srKvAvKoo8Q18jmngqUzsPteF0p7X8+3TnAF461IZnHrxzXOEpqAH/sZeP48S5XgDAiXO9+P1XfoHXt9ZnfD0REU0/DGQUjjGa/93hpL56Ifad7JxwL5S/sf/K4DAe2X0MpqmgaYL3Uv1fAmvn4/IKq1rl30vpZyikPThgj6XYe/hc2oyxWFTzhKtMVbnJhqcPugY8fz7j+zMREU1/WtgHoDxlN/9/6bD1q2+avx1O9j2xBiKC7sG488/8FadMNq2o8ISjpGkFqeYLfWg+34uhhFU/U7CewrSrVf69lEEy/dOhhIHhpOH52i2xqOd6taGmDIVR6+edir6uu319cf4+OSIimv4YyGja8wekWxcUjNkLZaRev2hBAaJ6eq+ZPRPMzT253h3kdAHurypBeVEM5UUx3F9VmvHvjUU1fLqy2Hl/RBNsWlEBw1TYcaAFm3YdhakUNq+rxsqqEmxdX+35Wdyv23Ggxfk5RvO9x1dh1dJSzC/QsWppaVqfHBERTX+8sqRpK1Of1yN1lQCAHQdaRh1X4X4a0X6/rUAXlC0owJXBuDO8NRbRoImgqbUb9y0pRkffEATeKfr2ue79r//gTNF3s/+eFZ8o8Yy68D/NuXndUqxZVoamth4owDn/ZJ4unRPRnJ4xIiKamRjIKOsmM1MMwKi7J917K5sv9OHVpnP4g4YqPLPR6sXyrzoqL4qhvDjmrBTSNWuGWMJQiGiC228pQGXpXM9cMfeoCr9/fsctnsZ92/W4gVMX+9Pe7+8b2/X2WSiFtEn7nBtGRJSfeGVJWWcHq+YLfdjzTjtePNg6rvdl2j1pBy73aInBeBK73jrrfG93n5Z9fdhQswjf/vf3Q9ME77p6yOzvbZjK+Zodhtzs68S1f/EmTl1ID2Nu/ve7zwNYPWt2xc792qnuLyMiopmBFTKacv6KWFDVJ2g5tr9qNtpQ06C9lUlT4fXmDjS19aB+2UJsXleNfSc7PGuK9p3sQFe/dxWR/b0VMOoQVXdVzm1+gY4bHxues/jfbz816l4YHvRazg0jIspPDGQ05fzrg2JRzemtsvu01v7Fm4HLsYH0/ZF2L9aTG2qcvrH6ZQtR94lSnDjf6/y9AmvP4+WBYZzpuoat66uxuLjQs6YoPuANQ5rA01g/WhjyV+UAq/r2hw3WOqWmtm7nStTuHbO5F4a7r2H9k/Z1TbCtsRYKSOsvIyKi2YuBjKac/6rxenykB8zdp2UbShhOZauhpgymaz2Quxdrx4GWtMb41dWl1gDXlC7fjkh3lS2iCUQA0zW9f2VVqadHa1tjLQyl8MMTF7DzrTYAwL0VRVhdXYZL/UNpP+uiBQV4euPIDLFMMoXMbQFha7Jro4iIaOZiIKMp5w5BtqA+LVtEE6ey9f6lAUR1LbCxvam12/P1o2d78PpWa05ZU2s3Ll8bhibWSItYVHOqXAI4y8YThoIAiOiC+5aU4LuPr/JcsWqa4NSFXrgLYc0X+nHqYr/nStK2pHTuhB9QGOuBATb2ExHlHwYymnLbXSHoyrVhGMq6CrzUPwSlRkZD2Fd2wEhlazhpeq4F3UNgdd37DErXwDCef6PFqabZNAHurSh2qk/PPHgnmtp6nKtLBeBTlcXOqAh35c0OdH6ZxoEZpn85U7CJhCwuBCciyj98ypKmnBOCvv4A6qpKoYkVgrr6h51QVF4Uw9c21ODw1x/Aw3WVnicQ3dxDYE1fKvpwYBg/dq1UspkKMJUa93R8d1jKFLzsb2XPNLO/z9qaReP4RCb29OT2xlpsXR88OJaIiGYnVshoXCYzS0zXxBm66pY0Fa4NJ5z1Q5meQBQAlaVz8chfH4Wua2k9XKYCrg0nUBjVPNegQYHHvxvTUApffOmI833tqp0u1q5KmybAiiUlWF29EMfPXUV99UIAgmPtE3sKciJPT3IhOBFR/hGlxl7NEra6ujrV3Nwc9jHyQqbg5b7Wc/dAjRXU3O/z8/dSffJPf+YJZJoABRFv2BJ490jqAtRVlcIwzbQnHDMFxqAzBV2fAsAdRTEsLi6c0EBbIiLKbyJyUilVN5H3sEJGHpme8Ms0S+yxl4+j+XwvTAWc6RpIeyLQrgzZIyHOdA04ocvdS2WYCrcURj2BbN6cCAbjSc/5FICoLkikylhG6npy/1cbxv0zun8Wm/3QwZplZZ6xFO4xGuN52nGyWwmIiCi/sYeMAIxMoX+lqT0teAHBPVAvHGx1wpj1ehNHzvZ4FmS/cLAV2xprsf+rDXh9az0eX1sd2Ev1wsFWXB2MO+dZXBzDl9dUpfWWxSIaFi0ocHq67KcpJ8I/NR+wKmSdvTdw9GwPiudGUV4Uw60LCgKn6Y9mslsJiIgov7FCRgC8lTGbOzA9taEGx9uv4kzXAO4pL8KTG2rwey8f9/SHaQInqAVV2ax+MoXiuVGUIIpNK6yBqIapsK+5A3FX81Z5cSH+6ME7oQnwo+YODA4nIGKlsI8Ghp2/N6ppMJRy9lOOp0JlV+0Ot17B5WtxdKcWjF++Fsfla3HnZ19eUYS+G4kJPe3IkRVERDQZDGQEIP0aL6oLNq8becJv56E2nO4cwFDCwOnOAbx0qA0NNWV4/9IAhpMmNLGGrG5rrMWju495QsnepnYowDPwNaIJ9p/ssAa1KoUrruqYLoAmgkd3H4OmCQaGEmmzy2yD8ST2Hj4HXaxG+NHCoD+o2VPzE0Z6H+VQwoBhmti6vnpCa4w4soKIiCaDgYxgmAria3NSyqp42dWloMrPa1vqoZTCvpOdEAD3Ly0FkD4Y9nrcwJ532lEyN+qZ4N81MIydh9o8V4MAUDhHx//r6MNwcnwPnAwlDDS1dUMBgVeumYJaUC+Zc4bUSIuJPu3IXZRERDQZDGSzjLsSVL9sIQDg2Nmradd36dPp+zzfJ2kqz3VbUOVH1wQigv7UtZ5dqQoaYzGUMFCMKAqjetoEf2t0hfX1WESDQEYNYxFNEItqGE6YSJoKhVEduqalXblqqUqbYarAQOlfq7Ro/hwsWTgvcBfleHFkBRERTQYD2SzjX+wNIHCBt/t1QdPp7X4wm7/yYy/6zlSR8i/SLozqeHhFBTQR7HyrzXNNqJRC8dw5KEEUFSVzcepiL/zmzdFwT3kROvqGIAD+9X2LoYvmzANrCqh2mQo43dmPFw+2BgbKbb7ZZAoj4XW0sRlERERTjYFsujMN4Pi3gIsHgSWNwOpvAlrwVHsgfbG3zd9gPtZ0ersfzL0UW9MEnX1D+FFzB5rauvGrjn7Pzkd/z1RQiNt5qA0L583BR9fiULDmisWTCpcHhlNPPt7wfE/AepJy87plUADeSwW8V5vOY+v6aux7Yg0AwFAqdR7vD2M/+fnalvq0q0R3NWvHgRa8zIXeREQUEgay6e74t4B3nwOSN4CPTgEQYM2fZny5/xoOgHOt5w5L7tfFIhqWVxShs8+ahL9pRSWe3jgyEHb322c9+yUB7/BUwHoIYOv6aqdy5m6et4ONeyBrRBOUzYvi6o2EUy3zX2vaw1ofqasMfFgg0xOMAuvq0P1zj3WVyKcjiYgoTAxk2TbBCleaiwetMAZYv158Y9RA5l8R5F/zY1e8DrdcQcm8KPQh4J7yInz38VWYExkZS+eeS+YPY0Hm6AJDKXzmuUPoHowjaVoVq33NHXi4rhLbG2vTqnfXPzY8V5eawLnW9FeygPQ+tvrqhU746+ofcqpjCtYOzMUlhXw6koiIZgSuTsq2o/9lpMIVmQus/ONRA9WUv98naG2Qf4VRpteNxl2V8otogic/WwMRb09Z0jQ9gSyqC37zrd/y9G75H1IQV8A0XGM03CuVYlENT6xfNqEKl2EqvHiwNTAIEhERTcRkVidxUn+2BVW4JmL1N60Qtnit9evq/3xTxwka9RA0hX60kRBBOUUBgWEMqa/va77oDIUtL4ph87pqfKqy2PO6RQsK8OjuY9hxoAVG6nu5J9/vPXwOmgD7nliDZx68E8fOXnXOaP/NmgD3VhRP+AlJ+0rT/t4MY0RElEsMZNm2pNGqbAHWr0s2Tuz9mm5VxL502Pp1Ated7hVGdsgJWhsUdEXXUFPm9KABVgWsvCiGVUtLMUcf+XpQbpGAr9sDXC8PDKPvRgKAwv1LSzG/QEdUF5QXxdD3Tx+nrRwK6u1yn9H/s5ip3ZYMVERENJOwhyzbVn8TgFiVsSUbb7rCNRFBw1D9y74zzdza3liLfc0dTvO+ArC4pBCGqTwzwm6/JYZrwwnPUvDbi2L4wn2LsfvtdiRNhYgmuCUWdb7XUMLAvpOdzvwyO1TZ0/jdTfVBvV3uJz+XVxThYu8Np2+N/V9ERDQThRLIROQ8gEEABoDkRO9ZZxS7wnUTfV8TZQeW0WaEjdVfpWuCh+sqPT1fDcvKoABPQHqkrhJH26/ixLmR2WFLSufi2QfvQlTTnOB3sfcGIq4nHyV1Jvts7qcr3aEqaPK9O2gWRnVsXrc07UEAIiKimSTMCtkGpVTP2C+jicq0KNz9VGKmxdu2TIvAAaQFpMOt3Z73Jg3T6clyD4eNaNbV5CN1lZ6GfPfQWH+oChpX4b/GPNZ+1en9IiIimol4ZTkL+Rvy5xfo+MraahhKjWv4qWEqPPbycTSf74WpgMKo5tlr6Q9Iuu5tRWz56Dp2HGgJHHWxuKTQWfatB4y3GE+o4ogKIiKabcIKZArAz0VEAditlNoT0jlmJX9g+cpaa6TFpl1HPZWlvU3tMFJjT46dversvtx/shOXXYNf7Wn3mcKSYXjnlA3Gk9h5qA1KqYzh6WZ2PmZa4zSeyh8REdF0FFYgW6uUuiQitwI4ICK/UUq9436BiGwBsAUAlixZEsYZQ+WevzXRkBHUd2WYCpomnr2V1+MGdr11FoBVvfrlxT6YamSEhM16alKwadfRwLOsrV2EDy4Ppi0N33+yE4e//kDaWW725/SHOffMNK49IiKimSiUQKaUupT69YqI/ATA/QDe8b1mD4A9gDUYNueHDFnQE5J2yPCHmKdSOyLdoWZbYy0UgKa2HihYoyBOd/Sn7a10zw4zAj5lTaynJk939mMoYQYGnu2NtVBK4a8OtaV9/7EqYaP9nOPFtUdERDTT5TyQicg8AJpSajD1+4cA/Ldcn2O6Gy1k+EPMsbM9aD7fBxNA84U+7GvuwBfuq8ArTeec15TMjXpWINmVMve+S7+oLnjqszVoautBV/9w4FkAK3SJCNx1LU2AL9xXMeZV4lSEKfaUERHRTBdGhew2AD8REfvv/4FS6mchnGPaMEyF599owf6TnRAAX1xRgfplCzPO3/KPs/hlRz/cXVxdA8P4ztHzGcdKxCIa7q0shmGa0ERwsfcGrseTgAJuJAwnqH31MzXOk5JjBZ4jbT2eCtvtt8Q8q5IyVb9GmzM23mvMoCtaIiKimYS7LKeBHQdasPNQm1OlsnY/LoMmgqa2bmgi6OgbwuBwAsMJ01PNKozq0DV4BrMCI5PyTWXtdtyyrjptrETQeIyIJrh1QQEeqat0nnwcz55Hdx+XvRuzqa0HzRf6nNesrCrBvifWeN4X9L39c8b8ezaJiIims8nssuTYixwLqv4caevxhKykqfDqkXN4fG016peVYddbZwOvFO1xFgnTxM5DZz3/TKWa8+3djk9vtAKNu68saF+lezRF0HkzLd0OqlKNp7I2njlj7AkjIqLZjoEsh/zzvc50DUBgXdv9qqPfE7quxw3seacdJXOjgWEsogm+srYa2xpr8fwbLbhtwRz03kgAAAoiunUFCatCdrHXWm7u7z1bXlHkXGPa/MFpvE33QcFqsleJ7AkjIqJ8w0CWQy8cbHXCGDAy3+u1LfVQyhoT0X09jkSqGcvu/bJXDrnduqAAT26oSRvgunX9MijAcwXaPRi3rgV9lSfDNFNXi5n3Wt5MtcoOafY15KO7j7EnjIiIKAADWQ4daevxjIXQBGhYVgZdEzz70F149qG70nqxHl5RAQB49cg5p0+sICKoLJ2Lzzx3KG2Aa1NbD17fWu9ZDJ40FfY2tePu8iIURjUMJUwURnWsrVmUVtXyX1EGPVwwURMdbXEzQ2OJiIhmIgayPVW1NQAADRFJREFUHGqoKcN7nf2Ipypgc+foOJzq57KrRkHVIV0TPL3xTqf5XRNx5oL5aWIFIAV4KmvX4wZOd/Tj3spimEplrDz5w9PmdUuxdX31TVWr2BNGREQ0Om3sl9BU2d5Yi08tKXGegLweN3DyQh/2vNOOFw+2AhipDtnLst1XewpWs35H743AMAYAHX03sOeddqdyFnH9f3g4acIwTeuKFMCju49hx4EWGK6ynT887TvZiaa2HqzJ8HTleDTUlKEwqgNI71EjIiIiVshyyh4h4e/Rt6tGdq9VU2s3dF2DaSqn58pducoUiTQBBOJZ5j2/QPeMxOgaGMbzb7Rg72FraOz7lwZwvP0qjNTf5b6ijGiC7sE4Lg8M39RKIvaEERERjY5zyHLMP3MMsHZF3lEUQ0XpXLznu4rMNNPL/V57vMXKqlLcv7TUCVuFUR0lc6NOLxlSr7ttQQEuX4t7vmY9FKBj87qlzryyS31Dnvfac8RuZs8mERHRbMc5ZNPEaIFle2Otp+EesAJV18Cw52s2u3rmHgXhdkdRDItLCj2VJ901ANZQCjtdOyZNBQymRmLYRp76NHCs/apzXep/wMC+apyK/ZNEREQ0goEsC0YLLLomeLiuMm1CfiZ2ENqWuvZ77d2L+Oha3Gna37SiAs8+dJfnPe4nFA1T4RftV3HivFVdi2iCBbGo5xrTXSFz93dlumpkkz4REdHUYiDLgrECy1MbanC8/SrOdA0gnjSduWOAtdB7+eIiRHTvXDBdE2xrrMWRsz340HXdiIwdZRZdE6yqXohTF0cGz1aWFKL/RsKz1zLoyctM4yc4uJWIiGhqMZBlgTuwRDTBpb4h7DjQ4lxd7jzUhtOdAxhKGNB9eUopYF3tosCK0wsHW3HS1UeWNBWOtfcAGL06dezsVSeMJU0FU6m0URYT6QFjkz4REdHUYiDLAjuwvN7cge7BOLoGhrHnnXYopSAieKVp5LrSUFZVzK6SJU3leeLSv/MyaLDsWPwVraCBsBPBwa1ERERTi4EsC+zA0tTW48wDG0oY2PX2WSgFzxOWEU1QENGglImkqZwrwKA+tIaaMrx/aQDDSRMC4PaimLMofHuqShX0MIG7olVfvRCGUti06yifkCQiIpomGMiyyP9kpLtXDLAqXElT4XrcQEQDyotieKSuEtsaa/Ho7mOePrTXmzvw9n/c4AQre1p/V//IjDAFBD5M4K5o7TjQgpdTr/HPIBsrnHHcBRERUXZwUn8WbW+sxeZ1SxH1N4qluK8fkyZwbTiBvU3teOzl41hdvRARV9j5cGAYn33uEBSA17bUwzCVM6/MDmxBDxPYDFNhx4EWz3XpcNLEu+d70ezbFpDJCwdbsfvts2i+0IcX32zF77183DPln4iIiCaHgSyLdE0gIhjv7N3rcQPX4wZOnOvFiXNXceuCAuef2bPK7ODUUFPmCWzdg3HommRcUWRfgQaNuwDSA1yQI209GE5aIdBUwLvne8cMcURERDQ2BrIsO9LW4+kZs0U0cZ6w1ATw3/x9cPkaHq6rdAKWzQ5O2xtrPYEtaSokDQPLK4owv0DH8ooiPLmhxnMO99yz+QU6VlaVojBq/U9gPOMrGmrKPOc0FcYMcURERDQ2BrIscy/WjkU0rFpaipVVJXjys8vw1IZarKwqwfYHarHiEyWe98WTJpKmic3rqnFHUcyphtnByR4w666IRXQdpzsHcD1u4HTnAF461BZ4jsKojq+srcYPNq/G1vXLsLKqBFvXVzvjK+zrzU27jnqWj29vrMXKqlInlMWiGmeQERERTQHusswyw1R48WBr4Mwvd5P8qupS/OTUJc/6JF2AuqpSJA0zcFCs/3v7913eURTD4uJCNNSU4akNNXjpUNu4Zo/5VyZtXV/tmfyf6echIiKiye2yZCDLkaAnFN2jLYIWgQPetUbuYBTEHaTsipo9SmOs97pt2nXUE+zspeJEREQ0tskEMl5Z5ogdvtxPNPqfirT3U7pNpOl+e2Mttq6vxsqqEty6oMDpXRvPe93815u8liQiIsouziHLEn9FrKm12xO+mtq60VCzyDNB/+EVFQCA/Sc7AQAVJXPx3qV+DCXMcQUj/7wxd/VtIqGKq5GIiIhyi4EsS+yZXcNJEycv9CHim0Wma1pg8NE1wbMP3QUguF9rvG4mVHE1EhERUW6xhyxL/H1YfnWfKMH+r46vL4sT8omIiGaOyfSQsUKWJQ01ZTh1sQ9Bg+w1AdbWjP8KMWivZdDycYY0IiKimYmBLEu2N9biePtVvHu+1xPKNAFWVpVO6AoxaCVSpr2VRERENPMwkGWJrgl+sHk1XjzYiqa2buha+hyxTPxXlPXLFnqa/+2ZY/6QxkBGREQ0MzGQZdFkm+P9V5Sb1y3F1vXVngZ9BaSFNCIiIpqZGMimIf8V5bH2q9j3xBpPsONoCiIiotmDgWwaaqgpG7P6xdEUREREswcDWZbczKgKVr+IiIjyCwNZlgSNqhhvNYvVLyIiovzCXZZZEjSqgoiIiCgIA1mWcEE3ERERjRevLLOEfWBEREQ0XtxlSURERDSFJrPLkleWRERERCFjICMiIiIKGQMZERERUcgYyIiIiIhCxkBGREREFDIGMiIiIqKQhRLIRORzIvKPItImIt8I4wxERERE00XOA5mI6AB2AvgtAHcDeExE7s71OYiIiIimizAqZPcDaFNKtSulPgbwGoDPh3AOIiIiomkhjEC2GECH68+dqa8RERER5aVpu8tSRLYA2JL6Y1xE3g/zPHmoDEBP2IfIM/zMc4+fee7xM889fua5d9dE3xBGILsEoNL154rU1zyUUnsA7AEAEWme6E4oujn8zHOPn3nu8TPPPX7mucfPPPdEZMILuMO4snwXQK2ILBWROQC+BOCnIZyDiIiIaFrIeYVMKZUUka8B+AcAOoBXlVJncn0OIiIioukilB4ypdTfAfi7CbxlT7bOQhnxM889fua5x8889/iZ5x4/89yb8GcuSqlsHISIiIiIxomrk4iIiIhCxkBGREREFLJpHci48zL3RKRSRA6JyAcickZEng77TPlARHQR+aWI/N+wz5IvRKRYRPaLyG9E5NciUh/2mWYzEfmj1L9T3heRH4pILOwzzUYi8qqIXHHP7hSRUhE5ICKtqV9LwjzjbJPhM38u9e+W0yLyExEpHuv7TNtAxp2XoUkC+A9KqbsBrAbwFD/3nHgawK/DPkSe+UsAP1NK/TMA94Kff9aIyGIA2wHUKaU+CesJ+y+Fe6pZ69sAPuf72jcAHFRK1QI4mPozTZ1vI/0zPwDgk0qp5QBaAPzJWN9k2gYycOdlKJRSl5VSp1K/H4T1HymutsoiEakA8NsA9oZ9lnwhIkUA1gN4BQCUUh8rpfrDPdWsFwFQKCIRAHMBdIV8nllJKfUOgF7flz8P4Dup338HwL/K6aFmuaDPXCn1c6VUMvXH47CG4I9qOgcy7rwMmYhUAfg0gF+Ee5JZ73kAfwzADPsgeWQpgG4Af5O6Kt4rIvPCPtRspZS6BOB/ArgI4DKAAaXUz8M9VV65TSl1OfX7DwHcFuZh8tAfAvj7sV40nQMZhUhE5gP4MYBnlFLXwj7PbCUivwPgilLqZNhnyTMRAPcB2KWU+jSAfwKvcbIm1bP0eVhBuBzAPBH5t+GeKj8pa9YV513liIj8J1itQN8f67XTOZCNa+clTT0RicIKY99XSv1t2OeZ5RoA/K6InId1Lf+AiHwv3CPlhU4AnUopu/q7H1ZAo+zYCOCcUqpbKZUA8LcA1oR8pnzykYjcAQCpX6+EfJ68ICJ/AOB3APwbNY6hr9M5kHHnZQhERGD11fxaKfW/wz7PbKeU+hOlVIVSqgrW/8bfVEqxcpBlSqkPAXSIyF2pLzUC+CDEI812FwGsFpG5qX/HNIIPUeTSTwF8OfX7LwP4PyGeJS+IyOdgtaL8rlLqxnjeM20DWaoZzt55+WsAP+LOy5xoAPD7sCo1v0r9378M+1BEWbANwPdF5DSATwH47yGfZ9ZKVSL3AzgF4D1Y/+3hOp8sEJEfAjgG4C4R6RSRxwH8OYAHRaQVVrXyz8M842yT4TP/KwALABxI/Xf0r8f8PlydRERERBSuaVshIyIiIsoXDGREREREIWMgIyIiIgoZAxkRERFRyBjIiIiIiELGQEZEREQUMgYyIiIiopAxkBFRXhGRlSJyWkRiIjJPRM6IyCfDPhcR5TcOhiWivCMifwYgBqAQ1k7L/xHykYgozzGQEVHeSe3HfRfAMIA1Sikj5CMRUZ7jlSUR5aOFAObD2jUXC/ksRESskBFR/hGRnwJ4DcBSAHcopb4W8pGIKM9Fwj4AEVEuici/A5BQSv1ARHQAR0XkAaXUm2GfjYjyFytkRERERCFjDxkRERFRyBjIiIiIiELGQEZEREQUMgYyIiIiopAxkBERERGFjIGMiIiIKGQMZEREREQh+/+2kdoqesDXOQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = x[noise <= 2]\n",
    "y = 2 * x + 1 + noise[noise<=2]\n",
    "\n",
    "plt.figure(figsize=[10, 5])\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim((0, 12))\n",
    "plt.ylim((0, 26))\n",
    "plt.scatter(top_x, top_y, s=12, c=\"darkorange\")\n",
    "plt.scatter(x, y, s=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.3.1.2 抑制过拟合的方法  \n",
    "* 增加数据  \n",
    "* 降维（特征选择、特征提取）  \n",
    "* Regularize正则化 \n",
    "* 降低模型复杂度  \n",
    "* ~~集成学习~~  \n",
    "* ~~Dropout~~    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2 Regularize 正则化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原来的代价函数上加一项正则项（惩罚项penalty）与惩罚率 $\\lambda$ 的乘积: $\\lambda P(w)$ ，即：  \n",
    "  \n",
    "$\\begin{aligned} J(w) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} + P(w) \\end{aligned}$  \n",
    "  \n",
    "$\\begin{aligned}\n",
    "w \\leftarrow w-\\eta \\frac{\\partial J(w)}{\\partial w}\n",
    "\\end{aligned}$\n",
    "  \n",
    "##### 1.3.2.1 范数\n",
    "  \n",
    "$L_p$ 范数：\n",
    "$\\begin{align*}\n",
    "\\| w \\|_{p}=\\sqrt[p]{\\sum_{i}\\left|w_{i}\\right|^{p}} = \\left(\\left|w_{1}\\right|^{p}+\\left|w_{2}\\right|^{p}+\\cdots+\\left|w_{n}\\right|^{p}\\right)^{\\frac{1}{p}}\n",
    "\\end{align*}$             \n",
    "\n",
    "   \n",
    "   \n",
    "$L_2$ 范数：\n",
    "$\\begin{align*}\n",
    "\\| w \\|_{2}=\\sqrt[2]{\\sum_{i} w_i^2} =\\left(\\left|w_{1}\\right|^{2}+\\left|w_{2}\\right|^{2}+\\left|w_{3}\\right|^{2}+\\cdots+\\left|w_{n}\\right|^{2}\\right)^{1 / 2}\n",
    "\\end{align*}$  \n",
    "\n",
    "$L_1$ 范数：\n",
    "$\\begin{align*}\n",
    "\\| w \\|_{1}={\\sum_{i} |w_i|} =\\left|w_{1}\\right|+\\left|w_{2}\\right|+\\left|w_{3}\\right|+\\cdots+\\left|w_{n}\\right| \n",
    "\\end{align*}$  \n",
    "  \n",
    "##### 1.3.2.2 正则化  \n",
    "L1正则(Lasso Regression)： $\\begin{aligned} J(w, b) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} + ||w||_1 \\end{aligned}$   \n",
    "L2正则(Ridge Regression)： $\\begin{aligned} J(w, b) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} + ||w||_2^2 \\end{aligned}$ \n",
    "\n",
    "**结论**\n",
    "* 原 loss function 加上 $L_1$ 正则项就是Lasso回归（索套回归）， 加上$L_2$ 正则项就是Ridge回归（岭回归）  \n",
    "* Lasso和Ridge都可以一定程度抑制过拟合。  \n",
    "* 岭回归用的多一点，效果好一点，且方便求解。 Lasso不方便求解  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.3 Ridge求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ridge Regression 岭回归，也叫权值衰减。  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{aligned} \n",
    "J(w, b) &= \\sum_{i=1}^{m}\\left(w^T x_{i}-y_{i}\\right)^{2} + \\lambda ||w||_2^2 \\\\ &= \\sum_{i=1}^{m}\\left(w^T x_{i}-y_{i}\\right)^{2} + \\lambda w^T w \\\\ &= \\left(w^T X^T-y^T\\right)(X w-y)+\\lambda w^T w \\\\ &= w^T X^T X w -2w^T X^T y + y^T y + \\lambda w^T w \\\\ &= w^T\\left(X^T X+\\lambda I\\right) w-2 w^T X^T y^T y\n",
    "\\end{aligned}$   \n",
    "  \n",
    "$\\begin{align}\n",
    "\\hat{w} &=\\arg \\min _{w} J(w) \\\\ &= w^T\\left(X^T X+\\lambda I\\right) w-2 w^T X^T y^T y\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{aligned}\n",
    "\\frac{\\partial J(w)}{\\partial w} = 2\\left(X^T X+\\lambda I\\right) w-2 X^T y=0\n",
    "\\end{aligned}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解得 $\\begin{align}\n",
    "\\hat{w^*}=\\left(X^T X+\\lambda I\\right)^{-1} X^T Y\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.4 岭回归的特点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 可以抑制过拟合。  \n",
    "* 引入单位矩阵，半逆矩阵加上单位矩阵，**一定可逆**。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.5 问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 1. 岭回归的“岭”指的是什么？  \n",
    "> 岭回归中引入了单位矩阵，单位矩阵也叫对角矩阵。  \n",
    "> 单位矩阵是一个方阵，对角线上的值全为 $1$ , 其他值全为 $0$。  \n",
    "> 对角线就像山岭一样，岭回归因此得名。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 2. 正则化为什么能够抑制过拟合？\n",
    "> **正则化的本质是给参数加上约束。**  \n",
    ">  \n",
    "> 如何求带约束的优化问题？  \n",
    "    >> $\\begin{aligned} J(w) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} \\end{aligned}$   \n",
    "    >>  \n",
    "    >> $s.t: ||w||_2 ^2 < m $  \n",
    "    >> \n",
    "    >> 构造拉格朗日函数：  \n",
    "    >>  $\\begin{aligned} L(w) = \\sum_{i=1}^{m}\\left(f\\left(x_{i}\\right)-y_{i}\\right)^{2} + \\lambda (||w||_2^2 - m) \\end{aligned}$\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 3. 为什么L1正则会产生稀疏解？  \n",
    "> $w = (1.2, 0, 0, 3.2, 0, 0, 6.3, 0)$  \n",
    ">>  \n",
    "> * 从解空间形状的角度看：   \n",
    ">> <img src=\"./imgs/2.regularize1.png\" width=\"550px\">  \n",
    ">> \n",
    ">> 假设只有两个特征，且没有偏置 $b$ 。   \n",
    ">> 给 $w$ 加上约束： $w_1^2 + w_2^2 \\leq m$\n",
    "> * 贝叶斯先验角度\n",
    ">> L1正则化相当于对模型参数 $w$ 引入了**拉普拉斯先验分布**，L2正则化引入了**高斯先验分布**。  \n",
    ">>  \n",
    ">> 而**拉普拉斯先验使参数为0的概率更大**。  \n",
    ">>  \n",
    ">> \n",
    ">> <img src=\"./imgs/2.Gaussian_distribution.png\" width=\"500px\"> \n",
    ">>  \n",
    ">> 高斯分布(Gaussian distribution)：均值为 $\\mu$ ，方差为 $\\sigma$ ，记作：\n",
    "$\\begin{align}\n",
    "X \\sim N\\left(\\mu, \\sigma^{2}\\right)\n",
    "\\end{align}$  \n",
    ">>  \n",
    ">> 概率密度函数：\n",
    ">> $\\begin{align}\n",
    "f(x)=\\frac{1}{\\sigma \\sqrt{2 \\pi}} e^{-\\frac{(x-\\mu)^{2}}{2 \\sigma^{2}}}\n",
    "\\end{align}$\n",
    ">>  \n",
    ">> <img src=\"./imgs/2.Laplace_distribution.png\" width=\"500px\">  \n",
    ">>  \n",
    ">>  \n",
    ">> 拉普拉斯分布(Laplace distribution): 均值为 $\\mu$ ，b > 0 是尺度参数  \n",
    ">>  \n",
    ">> 概率密度函数：\n",
    ">> $\\begin{align}\n",
    "f(x \\mid \\mu, b)=\\frac{1}{2 b} e^ {-\\frac{|x-\\mu|}{b}}\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 4. Lasso有哪些求解方式？  \n",
    "> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.6 L1 vs L2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### L1\n",
    "* 产生稀疏解，可以做特征选择  \n",
    "* 梯度分情况讨论，不能直接得到闭式解，也不能用梯度下降求解  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### L2  \n",
    "* 抑制过拟合效果更好  \n",
    "* 可以直接得到闭式解，且**一定可以求逆**   \n",
    "* 可以用梯度下降法求解  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 概率视角看线性回归和岭回归  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.1 极大似然估计MLE（Maximum Likelihood Estimate）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{align}\n",
    "y=f(w)+\\varepsilon\n",
    "\\end{align}$    \n",
    "  \n",
    "$\\begin{align}\n",
    "\\varepsilon \\sim N\\left(0,\\sigma^{2}\\right)\n",
    "\\end{align}$  \n",
    "  \n",
    "$\\begin{align}\n",
    "f(w)=w^{\\top} X\n",
    "\\end{align}$  \n",
    "\n",
    "$\\begin{align}\n",
    "y=w^{\\top} X+\\varepsilon\n",
    "\\end{align}$  \n",
    "\n",
    "$\\begin{align}\n",
    "y \\mid X ; w \\sim N\\left(w^{\\top} X, \\varepsilon ^2\\right)\n",
    "\\end{align}$  \n",
    "  \n",
    "$\\begin{align} P(y|X;w) = \\frac{1}{\\sigma \\sqrt{2 \\pi}} e^{-\\frac{(y-w^{\\top} X)^{2}}{2 \\sigma^{2}}} \\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "写出似然函数：  \n",
    "$\\begin{aligned}\n",
    "L(w) &= log\\left( P(y|X; w) \\right) \n",
    "\\\\ &= log \\prod_{i=1}^{m}P(y_i|x_i; w)  \n",
    "\\\\ &= \\sum_{i=1}^{m} log P(y_i|x_i; w) \n",
    "\\\\ &= \\sum_{i=1}^{m} (log \\frac{1}{\\sigma \\sqrt{2 \\pi}}  - {\\frac{(y_i - w^{\\top} x_i)^{2}}{2 \\sigma^{2}}})\n",
    "\\end{aligned}$  \n",
    "  \n",
    "$\\begin{align}\n",
    "\\hat{w} &= \\underset{w}{\\arg \\max} L(w) \\\\\n",
    "&= \\underset{w}{\\arg \\max} \\sum_{i=1}^{m} (log \\frac{1}{\\sigma \\sqrt{2 \\pi}}  - {\\frac{(y_i-w^{\\top} x_i)^{2}}{2 \\sigma^{2}}}) \\\\\n",
    "&= \\underset{w}{\\arg \\max} \\sum_{i=1}^{m} -\\frac{1}{2 \\sigma^{2}}\\left(y_{i}-w^{\\top} x_{i}\\right)^{2} \\\\\n",
    "&= \\underset{w}{\\arg \\min} \\sum_{i=1}^{m} \\left(y_{i}-w^{\\top} x_{i}\\right)^{2}\n",
    "\\end{align}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 结论  \n",
    "**最小二乘估计LSE等价于噪声服从高斯分布的MLE**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.2 贝叶斯角度 最大后验概率估计MAP（maximum a posteriori probability estimate）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ridge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{align}\n",
    "y=f(w)+\\varepsilon\n",
    "\\end{align}$    \n",
    "  \n",
    "$\\begin{align}\n",
    "\\varepsilon \\sim N\\left(0,\\sigma^{2}\\right)\n",
    "\\end{align}$   \n",
    "\n",
    "$\\begin{align}\n",
    "y \\mid X ; w \\sim N\\left(w^{\\top} X, \\varepsilon ^2\\right)\n",
    "\\end{align}$  \n",
    "\n",
    "$\\begin{align} P(y \\mid w) = \\frac{1}{\\sigma \\sqrt{2 \\pi}} e^{-\\frac{(y-w^{\\top} X)^{2}}{2 \\sigma^{2}}} \\end{align}$\n",
    "\n",
    "$\\begin{align}\n",
    "w \\sim N\\left(0, \\sigma_{0}^{2}\\right)\n",
    "\\end{align}$  \n",
    "\n",
    "$\\begin{align} P(w) = \\frac{1}{\\sigma_0 \\sqrt{2 \\pi}} e^{-\\frac{w^{2}}{2 \\sigma_0 ^{2}}} \\end{align}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{align}\n",
    "P(w \\mid y) &= \\frac{P(y|w) \\cdot P(w)}{P(y)} \n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MAP： \n",
    "  \n",
    "$\\begin{align}\n",
    "\\begin{aligned}\n",
    "\\hat{w} &=\\underset{w}{\\arg \\max}\\  log P(w \\mid y) \\\\\n",
    "&=\\underset{w}{\\arg \\max} \\ log P(y \\mid w) \\cdot P(w) \\\\ \n",
    "&=\\underset{w}{\\arg \\max} \\ log \\prod_{i=1}^{m} P(y_i \\mid w) \\cdot P(w) \\\\  \n",
    "&=\\underset{w}{\\arg \\max} \\sum_{i=1}^{m} \\ log\\ P(y_i \\mid w) + log\\ P(w) \\\\\n",
    "&=\\underset{w}{\\arg \\max} \\sum_{i=1}^{m} \\ log \\left(\\frac{1}{\\sigma \\sqrt{2 \\pi}} \\cdot \\frac{1}{\\sigma_0 \\sqrt{2 \\pi}}\\right) + ({-\\frac{(y_i-w^{\\top} x_i)^{2}}{2 \\sigma^{2}}} {-\\frac{w^{2}}{2 \\sigma_0 ^{2}}}) \\\\\n",
    "&=\\underset{w}{\\arg \\max} \\sum_{i=1}^{m}  ({-\\frac{(y_i-w^{\\top} x_i)^{2}}{2 \\sigma^{2}}} {-\\frac{w^{2}}{2 \\sigma_0 ^{2}}}) \\\\\n",
    "&=\\underset{w}{\\arg \\min} \\sum_{i=1}^{m}  ({\\frac{(y_i-w^{\\top} x_i)^{2}}{2 \\sigma^{2}}} + {\\frac{w^{2}}{2 \\sigma_0 ^{2}}}) \\\\\n",
    "&=\\underset{w}{\\arg \\min} \\sum_{i=1}^{m}  ({(y_i-w^{\\top} x_i)^{2}} + {\\frac{2 \\sigma^{2}}{2 \\sigma_0 ^{2}}} \\cdot {w^{2}})\n",
    "\\end{aligned} \n",
    "\\end{align}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{align}\n",
    "\\hat{w}_{MAP} = \\underset{w}{\\arg \\min} \\sum_{i=1}^{m}  ({(y_i-w^{\\top} x_i)^{2}} + {\\frac{2 \\sigma^{2}}{2 \\sigma_0 ^{2}}} \\cdot {w^{2}})\n",
    "\\end{align}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 结论  \n",
    "**L2正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验也服从高斯分布的MAP**  \n",
    "**同理可推，L1正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验服从拉普拉斯分布的MAP**  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 算法复杂度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 时间复杂度  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.1 线性回归闭式解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " $X$ 为 $ m \\times (d + 1)$ 的矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{aligned}\n",
    "\\hat{\\boldsymbol{w}}^{*}=\\left(\\mathbf{X}^{\\mathrm{T}} \\mathbf{X}\\right)^{-1} \\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}\n",
    "\\end{aligned}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|操作|input|output|时间复杂度|\n",
    "|-|:-:|:-:|:-:|\n",
    "|矩阵乘法|(n, m) * (m, p) |(n, p)|$O(nmp)$|\n",
    "|求逆矩阵|(n, n)| (n, n)|$O(n^3)$| "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ O(m \\cdot (d+1)^2) = O(m \\cdot (d+1)^2) + O((d+1)^3) + O((d+1)^2 \\cdot m) + O((d+1) \\cdot m) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.3 Ridge闭式解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " $\\begin{align}\n",
    "\\hat{w}=\\left(X^T X+\\lambda I\\right)^{-1} X^T Y\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ O(m \\cdot (d+1)^2) = O(m \\cdot (d+1)^2) +  O((d+1) ^ 2) + O((d+1)^3) + O((d+1)^2 \\cdot m) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 空间复杂度  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.1 线性回归闭式解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$O(m \\cdot (d+1)) = O(X) + O(y) + O(w) =  O(m \\cdot (d+1)) + O(m) + O(d+1)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.2 Ridge闭式解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$O(m \\cdot (d+1)) = O(X) + O(y) + O(w) + O(I) =  O(m \\cdot (d+1)) + O(m) + O(d+1) + O((d+1)^2)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 优缺点及其他特点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 优点  \n",
    "1. 实现简单  \n",
    "2. 效果稳定  \n",
    "  \n",
    "### 3.2 缺点：\n",
    "1. 计算复杂度高  \n",
    "2. 只能实现线性拟合  \n",
    "  \n",
    "### 3.3 其他\n",
    "1. 属于监督学习  \n",
    "2. 是一种线性模型  \n",
    "3. 是一种参数模型  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 优化空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 多项式回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin {aligned}\n",
    "f(x) = a x^2 + b x + c\n",
    "\\end {aligned}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAFACAYAAAAMDVCyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGDtJREFUeJzt3X+wpXd9F/D3hw2FlDpK3R1+JKwJJdFCVCpbppARnSaxcaSNVGrp1P6AjkscIMRpB0lTx061o1KnUYoyxBbtjLQpYlsYyhQ2qVoNpLIpHSVEkk0oJfyQBIpxbIRk9+Mf926y2T03e/buOed5znNer5k7uee5d+/97j7n+X4/eT7fz+ep7g4AAOPzpKEHAADAbAI1AICREqgBAIyUQA0AYKQEagAAIyVQAwAYKYEaAMBICdQAAEZqtIFaVV1ZVZ+sqiNV9eahxwMAsGo1xicTVNWeJHcluSLJfUk+muT7uvsTs75/7969fcEFF6xugAAAu3T77bc/0N375vnec5Y9mF16cZIj3X1vklTVTUmuSjIzULvgggty+PDhFQ4PAGB3qurT837vWFOf5yX5zAmv79s+BgCwMcYaqJ1WVR2sqsNVdfj+++8fejgAAAs31kDts0mec8Lr87ePPaq7b+zuA919YN++udK8AABrZayB2keTXFRVF1bV1yV5VZL3DTwmAICVGmUxQXc/UlWvT/LBJHuSvLO77xh4WAAAKzXKQC1JuvsDST4w9DgAAIYy1tQnAMDGE6gBAIzUaFOfY3L0WOett9ydW488kEuftzfXXHZR9jyphh4WADBxArU5vPWWu3Pjb9+bhx4+mjs+92AqybVXXDz0sACAiZP6nMOtRx7IQw8fTZI89PDR3HrPAwOPCADYBAK1OVz6vL0598l7kiTnPnlPLv2mvQOPCADYBFKfc7jmsotSSW6954Fc+k1784bLLhp6SADABqjuHnoMZ+3AgQN9+PDhoYcBAHBaVXV7dx+Y53ulPgEARkqgBgAwUgI1AICREqgBAIyUqs9d8rQCAJiOsa7rArVd8rQCAJiOsa7rUp+75GkFADAdY13XBWq75GkFADAdY13XpT53ydMKAGA6xrquezIBAMAKeTIBAMAECNQAAEZKoAYAMFICNQCAkVL1uUBj7WoMADxmndZrgdoCjbWrMQDwmHVar6U+F2isXY0BgMes03otUFugsXY1BgAes07rtdTnAo21qzEA8Jh1Wq89mQAAYIU8mQAAYAIEagAAIyVQAwAYKcUES7ZOTfUAYGrWfR0WqC3ZOjXVA4CpWfd1WOpzydapqR4ATM26r8MCtSVbp6Z6ADA1674OS30u2To11QOAqVn3dVjDWwCAFdLwFgBgAgRqAAAjJVADABgpxQQDWPfmewAwNlNdW0cXqFXVTyb520nu3z704939geFGtHjr3nwPAMZmqmvrWFOfN3T3C7c/JhWkJevffA8Axmaqa+tYA7VJW/fmewAwNlNdW0eX+tz2+qr6wSSHk/xod//hyd9QVQeTHEyS/fv3r3h4Z2fdm+8BwNhMdW0dpOFtVd2c5JkzvnR9ktuSPJCkk/zDJM/q7tc80c/T8BYAWBdn0vB2kDtq3X35PN9XVf86yfuXPBwAgFEa3R61qnrWCS9fkeTjQ40FAGBIY9yj9paqemG2Up+/n+S1ww4HAGAYowvUuvsHhh7DEKbaqA8AFm2T1szRBWqbaqqN+gBg0TZpzRzdHrVNNdVGfQCwaJu0ZgrURmKqjfoAYNE2ac2U+hyJqTbqA4BF26Q1c5CGt4um4S0AsC7OpOGt1CcAwEgJ1AAARkqgBgAwUooJRm6TmvoBwMk2fR0UqI3cJjX1A4CTbfo6KPU5cpvU1A8ATrbp66BAbeQ2qakfAJxs09dBqc+R26SmfgBwsk1fBzW8BQBYIQ1vAQAmQKAGADBSAjUAgJFSTLCGNr35HwDTZH07lUBtDW168z8Apsn6diqpzzW06c3/AJgm69upBGpraNOb/wEwTda3U0l9rqFNb/4HwDRZ306l4S0AwAppeAsAMAECNQCAkbJHbSL0ngFgnVi35iNQmwi9ZwBYJ9at+Uh9ToTeMwCsE+vWfARqE6H3DADrxLo1H6nPidB7BoB1Yt2ajz5qAAArpI8aAMAECNQAAEbKHrUJ06MGgDGwHu2eQG3C9KgBYAysR7sn9TlhetQAMAbWo90TqE2YHjUAjIH1aPekPidMjxoAxsB6tHv6qAEArJA+agAAEzBIoFZV31NVd1TVsao6cNLXrquqI1X1yar6jiHGBwAwBkPtUft4ku9O8o4TD1bV85O8KskLkjw7yc1VdXF3H139EKdJLxsAlsk6s1iDBGrdfWeSVJ1y4q5KclN3fzXJp6rqSJIXJ/nIakc4XXrZALBM1pnFGtsetfOSfOaE1/dtHztFVR2sqsNVdfj+++9fyeCmQC8bAJbJOrNYSwvUqurmqvr4jI+rFvHzu/vG7j7Q3Qf27du3iB+5EfSyAWCZrDOLtbTUZ3dfvos/9tkkzznh9fnbx1gQvWwAWCbrzGIN2ketqv5Tkh/r7sPbr1+Q5JeytS/t2UluSXLR6YoJ9FEDANbF6PuoVdUrquq+JC9J8htV9cEk6e47krw7ySeS/GaS16n4BAA21VBVn7+W5Nd2+NpPJ/np1Y4IAGB8POsTPW8A2BXrx/IJ1NDzBoBdsX4s39j6qDEAPW8A2A3rx/IJ1NDzBoBdsX4sn9Qnet4AsCvWj+UbtI/aouijBgCsi9H3UQMA4PQEagAAI2WPGjPpjQPAcdaE4QjUmElvHACOsyYMR+qTmfTGAeA4a8JwBGrMpDcOAMdZE4Yj9clMeuMAcJw1YTj6qAEArJA+agAAEyD1ydyUZwNMn7l+XARqzE15NsD0mevHReqTuSnPBpg+c/24CNSYm/JsgOkz14+L1CdzU54NMH3m+nHRngMAYIW05wAAmACpT86aUm6A9WT+Hj+BGmdNKTfAejJ/j5/UJ2dNKTfAejJ/j59AjbOmlBtgPZm/x0/qk7OmlBtgPZm/x097DgCAFVpoe46qekNVPf3shwUAwJmYZ4/aM5J8tKreXVVXVpW6XU7r6LHODYfuyivf/uHccOiuHD22/nduAdaZeXk9nXaPWnf/RFX9/SR/Jcmrk7ytqt6d5Be6+55lD5D1pOQbYFzMy+tprqrP3trI9oXtj0eSPD3Je6rqLUscG2tMyTfAuJiX19M8e9TeWFW3J3lLkluT/Nnu/jtJXpTkbyx5fKwpJd8A42JeXk/ztOf4xiTf3d2fPvFgdx+rqpcvZ1isOyXfAONiXl5P2nMAAKzQQttzAAAwDE8mYGWOHuu89Za7c+uRB3Lp8/bmmssuyp4n6fYCsGjm2+kQqLEySsMBVsN8Ox1Sn6yM0nCA1TDfTodAjZVRGg6wGubb6Rgk9VlV35PkJ5N8c5IXd/fh7eMXJLkzySe3v/W27r56gCGyBErDAVbDfDsdg7TnqKpvTnIsyTuS/NhJgdr7u/uSM/l52nMAAOviTNpzDHJHrbvvTBLPdwcA2NkY96hdWFUfq6r/XFV/cejBsFxHj3VuOHRXXvn2D+eGQ3fl6LH1b8AMsErm0Wlb2h21qro5yTNnfOn67n7vDn/s80n2d/eXqupFSX69ql7Q3Q/O+PkHkxxMkv379y9q2KyYEnKAs2Menbal3VHr7su7+5IZHzsFaenur3b3l7Y/vz3JPUlmvtu6+8buPtDdB/bt27ecvwRLp4Qc4OyYR6dtVKnPqtpXVXu2P39ukouS3DvsqFgmJeQAZ8c8Om1Dted4RZKfS7IvyW9U1e9193ckeVmSn6qqh7NVFXp1d395iDGyGkrIAc6OeXTaBmnPsWjacwAA62L07TngiXiYMMBs5sfNI1BjdFQwAcxmftw8oyomgEQFE8BOzI+bR6DG6KhgApjN/Lh5pD4ZHRVMALOZHzePqk8AgBVS9cnkqHQCNo15j0SgxppQ6QRsGvMeiWIC1oRKJ2DTmPdIBGqsCZVOwKYx75FIfbImVDoBm8a8R6LqEwBgpVR9shFURAFTYT5jJwI11paKKGAqzGfsRDEBa0tFFDAV5jN2IlBjbamIAqbCfMZOpD5ZWyqigKkwn7ETVZ8AACuk6pONpXIKGDNzFGdKoMakqJwCxswcxZlSTMCkqJwCxswcxZkSqDEpKqeAMTNHcaakPpkUlVPAmJmjOFOqPgEAVkjVJ5xEpRWwauYdFkGgxkZQaQWsmnmHRVBMwEZQaQWsmnmHRRCosRFUWgGrZt5hEaQ+2QgqrYBVM++wCKo+2Vg2+gKLYj7hTKj6hDnY6AssivmEZbFHjY1loy+wKOYTlkWgxsay0RdYFPMJyyL1ycay0RdYFPMJy6KYAABghRQTwC6p3AJOxzzBKgnU4AQqt4DTMU+wSooJ4AQqt4DTMU+wSgI1OIHKLeB0zBOsktQnnEDlFnA65glWSdUnnIaNw7C5XP8sw+irPqvqZ5J8Z5KvJbknyau7+yvbX7suyY8kOZrkmu7+4BBjhONsHIbN5fpnaEPtUTuU5JLu/nNJ7kpyXZJU1fOTvCrJC5JcmeRfVdWegcYISWwchk3m+mdogwRq3f2h7n5k++VtSc7f/vyqJDd191e7+1NJjiR58RBjhONsHIbN5fpnaGMoJnhNkl/Z/vy8bAVux923fewUVXUwycEk2b9//zLHx4abtXHYvhWYnlnXtcIBhra0QK2qbk7yzBlfur6737v9PdcneSTJu87053f3jUluTLaKCc5iqPCE9jypcu0VFz9uX8oNh+6ybwUmZqf9aCdf/7BKS0t9dvfl3X3JjI/jQdoPJ3l5ku/vx0pPP5vkOSf8mPO3j8Go2LcC0+O6ZowG2aNWVVcmeVOS7+ruPzrhS+9L8qqqekpVXZjkoiT/bYgxwhOxbwWmx3XNGA21R+1tSZ6S5FBVJclt3X11d99RVe9O8olspURf191HBxoj7Mi+FZge1zVjpOEtLIgCA1gfrleGNPqGtzBFGmPC+nC9si48lB0WxEZkWB+uV9aFQA0WxEZkWB+uV9aF1CcsiI3IsD5cr6wLxQSwRDYsw/Bch4yNYgIYCRuWYXiuQ9aZPWqwRDYsw/Bch6wzgRoskQ3LMDzXIetM6hOWaNaGZftlYDl2urYUDrDOFBPAit1w6K5H98uc++Q9ee3Lnmu/DCyAa4t1cSbFBFKfsGL2y8ByuLaYIoEarJj9MrAcri2myB41WDH71uDszbpm7EVjiuxRgxGwtwbOjGuGdWaPGqwZe2vgzLhm2BQCNRgBe2vgzLhm2BT2qMEI7LS3xt41sB+NzWaPGoyYfTjgOmB67FGDibAPB1wHbDaBGozYrH04R491bjh0V1759g/nhkN35eix9b8rDsfNen/bj8Yms0cNRmzWPpy33nL3o2mgOz73YCqRBmIyZr2/7Udjk9mjBmvmlW//cA5/+g8fff2tFzw9//7qlw44Ilgc7282gT1qMGHSQEyZ9zc8ntQnrBmPoGIqtN2A05P6hAnQvoB15H3LppL6hA2jfQHryPsWTk+gBhOgjQdjp+0G7I49ajAB2ngwdtpuwO7YowYTpc0BY+L9CI+xRw2QDmUw0pywOFKfMFHSoQxFmhMWR+oTNoj0E6vgfQZPTOoTmEk6lEWT5oTlkvqEDSIdyqJJc8JySX3ChpOm4mx4/8CZk/oE5iYdyjx2ek9Ic8JySX3ChpMOZR47vSekOWG5pD6BU8xKZ9108CV56y1359YjD+TS5+3NNZddlD1PqgFHybIcPdannOvvfcdHpDhhQc4k9emOGnCKS5+3N3d87sE89PDRR9NZ7rJtjlnnetZ7Alg+gRpwilnprO99x0fy0MNHkyQPPXw0t97zQK694uKZd1/caVsfs87frUceOOVc33TwJVKcMIBBArWq+pkk35nka0nuSfLq7v5KVV2Q5M4kn9z+1tu6++ohxgibbM+TKtdecfHj7pjtdEfFnbb1Nu/ds1nvCWD5hrqjdijJdd39SFX90yTXJfl721+7p7tfONC4gB3stGl81t0Xi/n6cPcMxm2QQK27P3TCy9uSvHKIcQDz2+mOyqy7L9Kh4zTrvLh7BuM2hj1qr0nyKye8vrCqPpbkwSQ/0d3/ZZhhAfOYt73H8eOCt9WYFZR5igCsn6UFalV1c5JnzvjS9d393u3vuT7JI0netf21zyfZ391fqqoXJfn1qnpBdz844+cfTHIwSfbv37+MvwIwh1l3X2al0zqxl22FZgVlO6Wp3T2D8Vrakwm6+/LuvmTGx/Eg7YeTvDzJ9/d2M7fu/mp3f2n789uzVWgwc/bo7hu7+0B3H9i3b9+y/hrALszqVj8rSPAEhMWY9e8469/bUwRg/QxV9Xllkjcl+Uvd/UcnHN+X5MvdfbSqnpvkoiT3DjFGYPdmpdM60ZttSeat3HyDNCesnaH2qL0tyVOSHKqq5LE2HC9L8lNV9XCSY0mu7u4vDzRGYJdmpUPn7c1mL9sTO5u+Z4oEYP14hBQwmBsO3fXonaBzn7wnr33Zcx+3l+34sU0N3k5XEPBE/2aCMRgvj5AC1sK8d9lmFSJMKXjbqZ3JvAUB+p7BdAnUgMHM+wSE/zpnFelOwdtQfd1m/d4kc7XNuPaKi3csCND3DDaHQA0YlXkLEeYN3q694uK5+7olpwZRZ3Ns1u+dNcad2mYoCAAEasCozFuIMG/wttOdqVkB06KPzfy9nbnuku3093b3DDaLQA0YvbMJ3pIzSKfOCKLO5tis3ztrjDvdJROUAQI1YC3NG7ztdHxWwLToYzsFYO6SAfPSngPYSEePdX7ulrtPCaIWeWxdq1CB5TqT9hwCNQCAFTqTQG1pz/oEAODsCNQAAEZKoAYAMFICNQCAkRKoAQCMlEANAGCkBGoAACMlUAMAGCmBGgDASE3iyQRVdX+ST6/gV+1N8sAKfg9nxnkZL+dmnJyX8XJuxmnR5+VPdfe+eb5xEoHaqlTV4Xkf+cDqOC/j5dyMk/MyXs7NOA15XqQ+AQBGSqAGADBSArUzc+PQA2Am52W8nJtxcl7Gy7kZp8HOiz1qAAAj5Y4aAMBICdQAAEZKoDaHqrqyqj5ZVUeq6s1Dj4ctVfWcqvqPVfWJqrqjqt449Jh4TFXtqaqPVdX7hx4Lj6mqP1FV76mq/1lVd1bVS4YeE0lV/d3teezjVfXLVfXUoce0qarqnVX1xar6+AnHvrGqDlXV3dv/ffqqxiNQO42q2pPkXyb5q0men+T7qur5w46KbY8k+dHufn6Sb0vyOudmVN6Y5M6hB8Ep/kWS3+zuP5Pkz8c5GlxVnZfkmiQHuvuSJHuSvGrYUW20f5vkypOOvTnJLd19UZJbtl+vhEDt9F6c5Eh339vdX0tyU5KrBh4TSbr78939u9uf/59sLTjnDTsqkqSqzk/y15L8/NBj4TFV9ceTvCzJLyRJd3+tu78y7KjYdk6Sc6vqnCRfn+RzA49nY3X3byf58kmHr0ryi9uf/2KSv76q8QjUTu+8JJ854fV9EQyMTlVdkORbkvzOsCNh2z9P8qYkx4YeCI9zYZL7k/yb7bT0z1fV04Ye1Kbr7s8m+WdJ/iDJ55P87+7+0LCj4iTP6O7Pb3/+hSTPWNUvFqix9qrqG5L8hyTXdveDQ49n01XVy5N8sbtvH3osnOKcJH8hydu7+1uS/N+sMIXDbNv7na7KViD97CRPq6q/Neyo2Elv9TVbWW8zgdrpfTbJc054ff72MUagqp6crSDtXd39q0OPhyTJpUm+q6p+P1tbBb69qv7dsENi231J7uvu43ee35OtwI1hXZ7kU919f3c/nORXk7x04DHxeP+rqp6VJNv//eKqfrFA7fQ+muSiqrqwqr4uWxs83zfwmEhSVZWtvTZ3dvfPDj0etnT3dd19fndfkK3r5be6292BEejuLyT5TFX96e1DlyX5xIBDYssfJPm2qvr67XntsijyGJv3Jfmh7c9/KMl7V/WLz1nVL1pX3f1IVb0+yQezVYnzzu6+Y+BhseXSJD+Q5H9U1e9tH/vx7v7AgGOCsXtDkndt/4/nvUlePfB4Nl53/05VvSfJ72armv1j8SipwVTVLyf5y0n2VtV9Sf5Bkn+S5N1V9SNJPp3kb65sPB4hBQAwTlKfAAAjJVADABgpgRoAwEgJ1AAARkqgBgAwUgI1AICREqgBAIyUQA3gJFX1rVX136vqqVX1tKq6o6ouGXpcwObR8BZghqr6R0memuTcbD0f8x8PPCRgAwnUAGbYfsTSR5P8vyQv7e6jAw8J2EBSnwCz/ckk35Dkj2XrzhrAyrmjBjBDVb0vyU1JLkzyrO5+/cBDAjbQOUMPAGBsquoHkzzc3b9UVXuSfLiqvr27f2vosQGbxR01AICRskcNAGCkBGoAACMlUAMAGCmBGgDASAnUAABGSqAGADBSAjUAgJH6/8Egzl3ZhH+xAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_func_points(func, x_start=0, x_stop=10, x_step=0.01):\n",
    "    x = np.arange(x_start, x_stop, x_step)\n",
    "    y = func(x)\n",
    "    plt.figure(figsize=[10, 5])\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(\"y\")\n",
    "    plt.scatter(x, y, s=12)\n",
    "    plt.show()\n",
    "\n",
    "show_func_points(func=lambda x: np.square(x) - 10 * x - 1, x_step=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 局部加权线性回归 Locally Weighted Linear Regression，LWLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.1 欠拟合underfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型在训练集上拟合的不好，在测试集上也拟合的不好。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.2 原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测过程，给每个点一个权值，离待预测点越近的点权值越大，优先优化权值大的点。  \n",
    "LWLR使用“核”来对附近的点赋予更高的权重，一般采用高斯核（Gaussian kernel，也叫径向基函数核 Radial basis function kernel，RBF kernel）计算，  \n",
    "公式： $\\begin{align}\n",
    "w(i, i)=\\exp \\left(\\frac{||x^{(i)}-x||_{2} ^ 2}{-2 k^{2}}\\right)\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.3 求解"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "J(a, b)= \\sum_{i=1}^{N} w_{i}\\left(y_{i}-a x_{i}-b\\right)^{2}\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{align}\n",
    "\\hat{a}^*=\\left( X ^{ T } W X \\right)^{-1} X ^{ T } W y\n",
    "\\end{align}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 最小二乘法\n",
    "    > * 使残差平方和最小   \n",
    "    > * 几何意义：使预测值和真实值的欧式距离之和最小   \n",
    "    > * 目标函数：$\\begin{aligned} J(w, b) = \\sum_{i=1}^{m}\\left(y_{i} - w x_{i}-b\\right)^{2} \\end{aligned}$   \n",
    "    > * $\\begin{aligned} (w^{*}, b^{*}) =\\underset{(w, b)}{\\arg \\min } \\sum_{i=1}^{m}\\left(y_{i} - w x_{i}-b\\right)^{2} \\end{aligned}$  \n",
    "    >  \n",
    "    > * 闭式解：$\\begin{aligned}\n",
    "{w^*}=\\frac{\\sum_{{i}=1}^{{m}} {y}_{{i}}\\left({x}_{{i}}-\\overline{{x}}\\right)}{\\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\\frac{1}{{m}}\\left(\\sum_{{i}=1}^{{m}} {x}_{{i}}\\right)^{2}}\n",
    "\\end{aligned}$   \n",
    "> $\\begin{aligned} {b^*}=\\frac{1}{{m}} \\sum_{{i}=1}^{{m}}\\left({y}_{{i}}-{w} {x}_{{i}}\\right) \\end{aligned}$   \n",
    "    >  \n",
    "    > * 矩阵形式，将 $b$ 纳入 $w$ 记作 $\\hat{w}$ ，将 $X$ 添加一列，全部填充为 $1$ \n",
    "    > * 矩阵形式闭式解：$\\begin{aligned}\n",
    "\\hat{\\boldsymbol{w}}^{*}=\\left(\\mathbf{X}^{\\mathrm{T}} \\mathbf{X}\\right)^{-1} \\mathbf{X}^{\\mathrm{T}} \\boldsymbol{y}\n",
    "\\end{aligned}$  \n",
    "  \n",
    "2. 梯度下降法：\n",
    "    > * 梯度：关于目标函数  𝐽(𝑤,𝑏)  的偏导数组成的向量   \n",
    "    > * 目标函数：$\\begin{aligned}\n",
    "J(w, b) = \\sum_{{i}=1}^{{m}}(wx_i + b - y_i)^2\n",
    "\\end{aligned}$    \n",
    "    > * 梯度：$\\begin{aligned}\n",
    "\\frac{\\partial J({w}, {b})}{\\partial {w}} = 2\\sum_{{i}=1}^{m}(wx_i + b - y_i)x_i\n",
    "\\end{aligned}$   \n",
    "    > $\\begin{aligned}\n",
    "\\frac{\\partial J({w}, {b})}{\\partial {b}} = 2\\sum_{{i}=1}^{m}(wx_i + b - y_i)\n",
    "\\end{aligned}$  \n",
    "    > * 更新梯度：$\\begin{aligned}\n",
    "w \\leftarrow w-\\eta \\frac{\\partial J(w, b)}{\\partial w}\n",
    "\\end{aligned}$  \n",
    "    >  \n",
    "    > $\\begin{aligned}\n",
    "b \\leftarrow b-\\eta \\frac{\\partial J(w, b)}{\\partial b}\n",
    "\\end{aligned}$   \n",
    "    >  \n",
    "    > * BGD、SGD、MBGD\n",
    "\n",
    "3. 正则化：\n",
    "    > * 正则化公式：$\\begin{aligned}\n",
    "J(w) = L(w) + \\lambda P(w)\n",
    "\\end{aligned}$  \n",
    "    > $L(w)$是Loss function，$P(w)$ 是penalty function   \n",
    "    >   \n",
    "    > * 范数公式：$\\begin{align*}\n",
    "\\| w \\|_{p}=\\sqrt[p]{\\sum_{i}\\left|w_{i}\\right|^{p}} = \\left(\\left|w_{1}\\right|^{p}+\\left|w_{2}\\right|^{p}+\\cdots+\\left|w_{n}\\right|^{p}\\right)^{\\frac{1}{p}}\n",
    "\\end{align*}$   \n",
    "    > * Lasso索套回归: $P(w) = ||w||_1$  \n",
    "    > * Ridge岭回归（权值衰减）：  $P(w) = ||w||_2 ^ 2$  \n",
    "    > \n",
    "    >  \n",
    "    > * Ridge目标函数：$\\begin{aligned} \n",
    "J(w, b) &= \\sum_{i=1}^{m}\\left(w^T x_{i}-y_{i}\\right)^{2} + \\lambda ||w||_2^2 \\\\ &= w^T\\left(X^T X+\\lambda I\\right) w-2 w^T X^T y^T y\n",
    "\\end{aligned}$  \n",
    "    > * Ridge矩阵形式闭式解：$\\begin{align}\n",
    "\\hat{w^*}=\\left(X^T X+\\lambda I\\right)^{-1} X^T Y\n",
    "\\end{align}$   \n",
    "    > \n",
    "    > * Lasso vs. Ridge：  \n",
    "    > Lasso不能求闭式解也不能直接梯度下降法求解，但是Lasso可以产生稀疏解，可用于特征选择；\n",
    "    > Ridge都可以，且一定可逆，且效果更好。\n",
    "    > \n",
    "    > * 为什么正则化能够抑制过拟合？\n",
    "    > 正则化的本质是给参数加上约束。降低了模型的复杂度。\n",
    "    > * 为什么l1正则可以产生稀疏解？  \n",
    "    > 答：从解空间的形状看，l2的解空间是圆形，l1的解空间是多边形，多边形的解空间更容易在尖角处与等位线碰撞出稀疏解。从贝叶斯先验看，l2的参数w服从高斯分布，l1的参数w服从拉普拉斯分布，拉普拉斯分布在0点处是一个尖峰，所以参数w取值为0的可能性更高。\n",
    "\n",
    "4. 概率视角：\n",
    "    > * 线性回归 极大似然估计：最小二乘估计LSE等价于噪声服从高斯分布的MLE   \n",
    "    > * Lasso、Ridge贝叶斯视角 最大后验估计：  \n",
    "    > L2正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验也服从高斯分布的MAP  \n",
    "    > L1正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验服从拉普拉斯分布的MAP  \n",
    "\n",
    "5. 算法复杂度\n",
    "    > * $X$ 为 $ m \\times (d + 1)$ 的矩阵\n",
    "    > * 时间复杂度（闭式解，线性回归和Ridge同）：  \n",
    "    > $ O(m(d+1)^2)$\n",
    "    > * 空间复杂度（闭式解，线性回归和Ridge同）：\n",
    "    > $ O(m(d+1))$\n",
    "\n",
    "5. 优化\n",
    "    > * 多项式回归\n",
    "    >> * 转化为特征工程问题\n",
    "    > * 局部加权线性回归LWLR  \n",
    "    >>  * 求距离一般采用高斯核：$\\begin{align}\n",
    "w(i, i)=\\exp \\left(\\frac{||x^{(i)}-x||_{2} ^ 2}{-2 k^{2}}\\right)\n",
    "\\end{align}$\n",
    "    >> * 目标函数：$\n",
    "\\begin{align}\n",
    "J(a, b)= \\sum_{i=1}^{N} w_{i}\\left(y_{i}-a x_{i}-b\\right)^{2}\n",
    "\\end{align}\n",
    "$  \n",
    "    >> * 向量形式解： $\\begin{align}\n",
    "\\hat{a}^*=\\left( X ^{ T } W X \\right)^{-1} X ^{ T } W y\n",
    "\\end{align}$\n",
    "\n",
    "6. 名词  \n",
    "    > 过拟合、欠拟合  \n",
    "    > 梯度、学习率learning_rate、超参数、epochs、mini-batch、batch-size、step、正则系数  \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 相关链接   \n",
    "  \n",
    "<a href=\"./02.2.LinearRegression-sklearn.ipynb\" style=\"\"> 2.2 sklearn中使用线性回归、岭回归、Lasso、SGD、多项式回归  </a>  \n",
    "<a href=\"./02.3.LinearRegression-numpy.ipynb\" style=\"\"> 2.3 numpy实现线性回归、岭回归、SGD回归、LWLR</a>  \n",
    "<a href=\"./02.4.LinearRegression-tf2.ipynb\"> 2.4 TensorFlow2实现线性回归、岭回归、SGD回归、LWLR</a>  \n",
    "<a href=\"./02.5.LinearRegression-torch1.ipynb\"> 2.5 PyTorch1实现线性回归、岭回归、SGD回归、LWLR </a>  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考文献"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://book.douban.com/subject/24703171/\">[1] [M]Peter Harrington[美]. 机器学习实战. 北京: 人民邮电出版社, 2013.6.</a>  \n",
    "<a href=\"https://book.douban.com/subject/26708119/\">[2] [M]周志华. 机器学习. 北京: 清华大学出版社, 2016.1.</a>  \n",
    "<a href=\"https://book.douban.com/subject/30285146/\">[3] [M]诸葛越, 葫芦娃. 百面机器学习. 北京: 人民邮电出版社, 2018.8.1.</a>\n",
    "\n",
    "https://www.experfy.com/blog/gradient-descent/  \n",
    "https://datawhalechina.github.io/pumpkin-book/#/chapter3/chapter3  \n",
    "https://blog.csdn.net/pxhdky/article/details/82960659  \n",
    "https://www.cnblogs.com/guoyaohua/p/8542554.html  \n",
    "https://zh.wikipedia.org/wiki/最小二乘法  \n",
    "https://zh.wikipedia.org/wiki/正态分布  \n",
    "https://zh.wikipedia.org/wiki/拉普拉斯分布   \n",
    "https://github.com/dragen1860/Deep-Learning-with-TensorFlow-book  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}