Update documentation

_sources/basic_sm/ls_regression.ipynb

@@ -9,16 +9,16 @@
     "As discussed in the lecture, one of the ways to create a model of $y(x)$ is by considering a linearly weighted combination of basis functions. Mathematically, this can be represented as:\n",
     "\n",
     "$$\n",
-    "    y(x) \\sim \\hat{y}(x) = \\mathbf{w}^T\\Psi(x) = \\sum w_i \\Psi_i(x)\n",
+    "    y(x) \\sim \\hat{y}(x) = \\mathbf{w}^T\\pmb{\\psi} = \\sum w_i \\psi_i(x)\n",
     "$$\n",
     "\n",
-    "where $\\Psi(x)$ is the vector of basis functions and $\\mathbf{w}$ is the vector of weights. The model is linear in terms of weights $w_i$ but basis function $\\Psi_i(x)$ can be non-linear. For example, the basis functions vector $\\Psi(x)$ can be $[1 \\text{  } x \\text{  } x^2 \\text{  } \\sin(x) \\text{  } e^{x}]^T$. The weights $w_i$ are determined by minimizing the sum of squared differences between the predictions and actual values. The weights obtained after minimization are given by (refer lecture notes for derivation):\n",
+    "where $\\mathbf{\\pmb{\\psi}}$ is the vector of basis functions and $\\mathbf{w}$ is the vector of weights. The model is linear in terms of weights $w_i$ but basis function $\\psi_i(x)$ can be non-linear. For example, the basis functions vector $\\pmb{\\psi}$ can be $[1 \\text{  } x \\text{  } x^2 \\text{  } \\sin(x) \\text{  } e^{x}]^T$. The weights $w_i$ are determined by minimizing the sum of squared differences between the predictions and actual values. The weights obtained after minimization are given by (refer lecture notes for derivation):\n",
     "\n",
     "$$\n",
     "    \\mathbf{w} = (\\Psi^T\\Psi)^{-1}\\Psi^T\\mathbf{y} = \\Psi^{\\dagger}\\mathbf{y}\n",
     "$$\n",
     "\n",
-    "where $\\mathbf{y}$ is the vector of target values and $\\Psi^{\\dagger}$ is the (Moore-Penrose) pseudo-inverse of $\\Psi$. The pseudo-inverse will be regular inverse if $\\Psi$ is invertible.\n",
+    "where $\\mathbf{y}$ is the vector of target values and $\\Psi^{\\dagger}$ is the (Moore-Penrose) pseudo-inverse of $\\Psi$ matrix. The pseudo-inverse will be regular inverse if $\\Psi$ is invertible.\n",
     "\n",
     "The choice of basis functions is crucial in determining the performance of the model. In this notebook, the entire process of creating a linear least squares regression model is demonstrated using simple basis functions. Later sections will include more complex basis functions.\n",
     "\n",
@@ -101,7 +101,7 @@
     "Assuming that we only have training data (and no knowledge of the true function), we will try to fit a linear model to the training data. Thus, the basis function vector will be:\n",
     "\n",
     "$$\n",
-    "    \\Psi(x) = [1 \\quad x]^T\n",
+    "    \\pmb{\\psi} = [1 \\quad x]^T\n",
     "$$\n",
     "\n",
     "Below block of code defines couple of functions: one to fit a line to given data and another to predict the output for given input. Read comments in the code for more details."
@@ -251,7 +251,7 @@
    "source": [
     "We are able to obtain a line that minimizes the sum of squared differences. However, the line does not fit the data well. This is because the model is too simple for the given data. We need to use more complex basis functions to create a better model.\n",
     "\n",
-    "Since we know what true function is composed of, we can set basis function vector as $\\Psi(x) = [\\sqrt x \\quad \\cos(x)]^T$ and see if we can get correct weight vector. Below block of code updates fit and predict function to accomodate new basis function vector."
+    "Since we know what true function is composed of, we can set basis function vector as $\\pmb{\\psi} = [\\sqrt x \\quad \\cos(x)]^T$ and see if we can get correct weight vector. Below block of code updates fit and predict function to accomodate new basis function vector."
    ]
   },
   {
@@ -378,7 +378,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The obtained weights are $[0.5 \\quad 1]$ which exactly matchs the true function. As a result, the prediction is same as true function. This shows that if there is some prior knowledge about the true function, we can incorporate that into basis function to get more accurate results."
+    "The obtained weights are $[1 \\quad 0.5]$ which exactly matchs the true function. As a result, the prediction is same as true function. This shows that if there is some prior knowledge about the true function, we can incorporate that into basis function to get more accurate results."
    ]
   }
  ],

_sources/basic_sm/poly.ipynb

@@ -87,10 +87,10 @@
    "source": [
     "## Fitting $m^{th}$ order polynomial\n",
     "\n",
-    "A one variable 2nd order polynomial will be fit to above data. Later, techniques like cross-validation will be used to find order of the polynomial. Basis set consists of:\n",
+    "A one variable 2nd order polynomial will be fit to above data. Later, techniques like cross-validation will be used to find order of the polynomial. Basis function vector will be:\n",
     "\n",
     "$$\n",
-    "\\{1\\text{, } x\\text{, } x^2\\}\n",
+    "\\pmb{\\psi} = [1\\quad x \\quad x^2]^T\n",
     "$$\n",
     "\n",
     "Below block of code defines two functions - `fit_polynomial` and `predict`. First function is used to get a least square estimate for fitting a $m^{th}$ order polynomial to the given data. Second function is used for getting the prediction at given input `x` based on the parameter values. Please read comments in the code for more details."
@@ -491,10 +491,10 @@
     "    f(x_1,x_2) = (1 - x_1)^2 + (1 - x_2)^2 + \\frac{1}{2}(2x_2 - x_1^2)^2\n",
     "$$\n",
     "\n",
-    "Since there are only two variables, following will be the basis set:\n",
+    "Since there are only two variables, following will be the basis function vector:\n",
     "\n",
     "$$\n",
-    "\\text{Basis set: } \\{ 1, x_1, x_2, x_1x_2, x_1^2, x_2^2 \\}\n",
+    "\\pmb{\\psi} =  [1 \\quad x_1 \\quad x_2 \\quad x_1x_2 \\quad x_1^2 \\quad x_2^2 ]^T\n",
     "$$\n",
     "\n",
     "> __*NOTE*__: There is no cross-validation here since basis function vector is fixed.\n",

basic_sm/ls_regression.html

@@ -356,14 +356,14 @@ <h1>Linear Least Squares Regression Model<a class="headerlink" href="#linear-lea
 <p>As discussed in the lecture, one of the ways to create a model of <span class="math notranslate nohighlight">\(y(x)\)</span> is by considering a linearly weighted combination of basis functions. Mathematically, this can be represented as:</p>
 <div class="math notranslate nohighlight">
 \[
-    y(x) \sim \hat{y}(x) = \mathbf{w}^T\Psi(x) = \sum w_i \Psi_i(x)
+    y(x) \sim \hat{y}(x) = \mathbf{w}^T\pmb{\psi} = \sum w_i \psi_i(x)
 \]</div>
-<p>where <span class="math notranslate nohighlight">\(\Psi(x)\)</span> is the vector of basis functions and <span class="math notranslate nohighlight">\(\mathbf{w}\)</span> is the vector of weights. The model is linear in terms of weights <span class="math notranslate nohighlight">\(w_i\)</span> but basis function <span class="math notranslate nohighlight">\(\Psi_i(x)\)</span> can be non-linear. For example, the basis functions vector <span class="math notranslate nohighlight">\(\Psi(x)\)</span> can be <span class="math notranslate nohighlight">\([1 \text{  } x \text{  } x^2 \text{  } \sin(x) \text{  } e^{x}]^T\)</span>. The weights <span class="math notranslate nohighlight">\(w_i\)</span> are determined by minimizing the sum of squared differences between the predictions and actual values. The weights obtained after minimization are given by (refer lecture notes for derivation):</p>
+<p>where <span class="math notranslate nohighlight">\(\mathbf{\pmb{\psi}}\)</span> is the vector of basis functions and <span class="math notranslate nohighlight">\(\mathbf{w}\)</span> is the vector of weights. The model is linear in terms of weights <span class="math notranslate nohighlight">\(w_i\)</span> but basis function <span class="math notranslate nohighlight">\(\psi_i(x)\)</span> can be non-linear. For example, the basis functions vector <span class="math notranslate nohighlight">\(\pmb{\psi}\)</span> can be <span class="math notranslate nohighlight">\([1 \text{  } x \text{  } x^2 \text{  } \sin(x) \text{  } e^{x}]^T\)</span>. The weights <span class="math notranslate nohighlight">\(w_i\)</span> are determined by minimizing the sum of squared differences between the predictions and actual values. The weights obtained after minimization are given by (refer lecture notes for derivation):</p>
 <div class="math notranslate nohighlight">
 \[
     \mathbf{w} = (\Psi^T\Psi)^{-1}\Psi^T\mathbf{y} = \Psi^{\dagger}\mathbf{y}
 \]</div>
-<p>where <span class="math notranslate nohighlight">\(\mathbf{y}\)</span> is the vector of target values and <span class="math notranslate nohighlight">\(\Psi^{\dagger}\)</span> is the (Moore-Penrose) pseudo-inverse of <span class="math notranslate nohighlight">\(\Psi\)</span>. The pseudo-inverse will be regular inverse if <span class="math notranslate nohighlight">\(\Psi\)</span> is invertible.</p>
+<p>where <span class="math notranslate nohighlight">\(\mathbf{y}\)</span> is the vector of target values and <span class="math notranslate nohighlight">\(\Psi^{\dagger}\)</span> is the (Moore-Penrose) pseudo-inverse of <span class="math notranslate nohighlight">\(\Psi\)</span> matrix. The pseudo-inverse will be regular inverse if <span class="math notranslate nohighlight">\(\Psi\)</span> is invertible.</p>
 <p>The choice of basis functions is crucial in determining the performance of the model. In this notebook, the entire process of creating a linear least squares regression model is demonstrated using simple basis functions. Later sections will include more complex basis functions.</p>
 <p>Below block of code imports required packages:</p>
 <div class="cell docutils container">
@@ -412,7 +412,7 @@ <h1>Linear Least Squares Regression Model<a class="headerlink" href="#linear-lea
 <p>Assuming that we only have training data (and no knowledge of the true function), we will try to fit a linear model to the training data. Thus, the basis function vector will be:</p>
 <div class="math notranslate nohighlight">
 \[
-    \Psi(x) = [1 \quad x]^T
+    \pmb{\psi} = [1 \quad x]^T
 \]</div>
 <p>Below block of code defines couple of functions: one to fit a line to given data and another to predict the output for given input. Read comments in the code for more details.</p>
 <div class="cell docutils container">
@@ -518,7 +518,7 @@ <h1>Linear Least Squares Regression Model<a class="headerlink" href="#linear-lea
 </div>
 </div>
 <p>We are able to obtain a line that minimizes the sum of squared differences. However, the line does not fit the data well. This is because the model is too simple for the given data. We need to use more complex basis functions to create a better model.</p>
-<p>Since we know what true function is composed of, we can set basis function vector as <span class="math notranslate nohighlight">\(\Psi(x) = [\sqrt x \quad \cos(x)]^T\)</span> and see if we can get correct weight vector. Below block of code updates fit and predict function to accomodate new basis function vector.</p>
+<p>Since we know what true function is composed of, we can set basis function vector as <span class="math notranslate nohighlight">\(\pmb{\psi} = [\sqrt x \quad \cos(x)]^T\)</span> and see if we can get correct weight vector. Below block of code updates fit and predict function to accomodate new basis function vector.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">fit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
@@ -610,7 +610,7 @@ <h1>Linear Least Squares Regression Model<a class="headerlink" href="#linear-lea
 <img alt="../_images/a77ac997e9d150734765c7e4b7d466679c1ed6b32da7641857ba7404c9ab3494.png" src="../_images/a77ac997e9d150734765c7e4b7d466679c1ed6b32da7641857ba7404c9ab3494.png" />
 </div>
 </div>
-<p>The obtained weights are <span class="math notranslate nohighlight">\([0.5 \quad 1]\)</span> which exactly matchs the true function. As a result, the prediction is same as true function. This shows that if there is some prior knowledge about the true function, we can incorporate that into basis function to get more accurate results.</p>
+<p>The obtained weights are <span class="math notranslate nohighlight">\([1 \quad 0.5]\)</span> which exactly matchs the true function. As a result, the prediction is same as true function. This shows that if there is some prior knowledge about the true function, we can incorporate that into basis function to get more accurate results.</p>
 </section>
 
     <script type="text/x-thebe-config">

basic_sm/poly.html

@@ -421,10 +421,10 @@ <h1>Polynomial Model<a class="headerlink" href="#polynomial-model" title="Permal
 </div>
 <section id="fitting-m-th-order-polynomial">
 <h2>Fitting <span class="math notranslate nohighlight">\(m^{th}\)</span> order polynomial<a class="headerlink" href="#fitting-m-th-order-polynomial" title="Permalink to this heading">#</a></h2>
-<p>A one variable 2nd order polynomial will be fit to above data. Later, techniques like cross-validation will be used to find order of the polynomial. Basis set consists of:</p>
+<p>A one variable 2nd order polynomial will be fit to above data. Later, techniques like cross-validation will be used to find order of the polynomial. Basis function vector will be:</p>
 <div class="math notranslate nohighlight">
 \[
-\{1\text{, } x\text{, } x^2\}
+\pmb{\psi} = [1\quad x \quad x^2]^T
 \]</div>
 <p>Below block of code defines two functions - <code class="docutils literal notranslate"><span class="pre">fit_polynomial</span></code> and <code class="docutils literal notranslate"><span class="pre">predict</span></code>. First function is used to get a least square estimate for fitting a <span class="math notranslate nohighlight">\(m^{th}\)</span> order polynomial to the given data. Second function is used for getting the prediction at given input <code class="docutils literal notranslate"><span class="pre">x</span></code> based on the parameter values. Please read comments in the code for more details.</p>
 <div class="cell docutils container">
@@ -718,10 +718,10 @@ <h2>General 2nd order Polynomial<a class="headerlink" href="#general-2nd-order-p
 \[
     f(x_1,x_2) = (1 - x_1)^2 + (1 - x_2)^2 + \frac{1}{2}(2x_2 - x_1^2)^2
 \]</div>
-<p>Since there are only two variables, following will be the basis set:</p>
+<p>Since there are only two variables, following will be the basis function vector:</p>
 <div class="math notranslate nohighlight">
 \[
-\text{Basis set: } \{ 1, x_1, x_2, x_1x_2, x_1^2, x_2^2 \}
+\pmb{\psi} =  [1 \quad x_1 \quad x_2 \quad x_1x_2 \quad x_1^2 \quad x_2^2 ]^T
 \]</div>
 <blockquote>
 <div><p><strong><em>NOTE</em></strong>: There is no cross-validation here since basis function vector is fixed.</p>