Notes/Week_1-intro_to_ML_linear_algebra/Notes_for_Chapter_1-Linear_Algebra.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aca2b3c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open notebook from D drive: jupyter notebook --notebook-dir=D:/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93f14ca",
   "metadata": {},
   "source": [
    "## 1. Python/numpy multiplication operators:\n",
    "\n",
    "- If matrices defined by np.matrix, * would give matrix multiplication. Now all matrices defined as ndarray/numpy array, so * will do element wise. \n",
    "\n",
    "- np.matrix is deprecated.\n",
    "\n",
    "- **@** is called infix operator for matrix multiplication in **python**. The matmul function implements the semantics of the @ operator introduced in Python 3.5 following PEP465. @ is same as np.matmul without method parameters.\n",
    "\n",
    "- Using **a.dot(b)** or **np.matmul(a,b)** is better IMO. Both are numpy operators. I would prefer the second.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "182a5619",
   "metadata": {},
   "source": [
    "### > `a.dot(b):`\n",
    "https://numpy.org/doc/stable/reference/generated/numpy.dot.html\n",
    "\n",
    "**`numpy.dot(a, b, out=None)`**\n",
    "\n",
    "Dot product of two arrays. Specifically,\n",
    "\n",
    "1. If both a and b are 1-D arrays, it is inner product of vectors (without complex conjugation).\n",
    "\n",
    "2. If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a @ b is preferred.\n",
    "\n",
    "3. If either a or b is 0-D (scalar), it is equivalent to multiply and using numpy.multiply(a, b) or a * b is preferred.\n",
    "\n",
    "4. If a is an N-D array and b is a 1-D array, it is a sum product over the last axis of a and b.\n",
    "\n",
    "5. If a is an N-D array and b is an M-D array (where M>=2), it is a sum product over the last axis of a and the second-to-last axis of b:\n",
    "\n",
    "    dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e85781",
   "metadata": {},
   "source": [
    "### > `numpy.matmul`\n",
    "https://numpy.org/doc/stable/reference/generated/numpy.matmul.html\n",
    "\n",
    "numpy.matmul(x1, x2, /, out=None, *, casting='same_kind', order='K', dtype=None, subok=True[, signature, extobj, axes, axis]) = <ufunc 'matmul'>\n",
    "Matrix product of two arrays.\n",
    "\n",
    "**Parameters**\n",
    "x1, x2array_like\n",
    "- Input arrays, scalars not allowed.\n",
    "\n",
    "outndarray, optional\n",
    "- A location into which the result is stored. If provided, it must have a shape that matches the signature (n,k),(k,m)->(n,m). If not provided or None, a freshly-allocated array is returned.\n",
    "\n",
    "**kwargs**\n",
    "For other keyword-only arguments, see the ufunc docs.\n",
    "New in version 1.16: Now handles ufunc kwargs\n",
    "\n",
    "**Returns**\n",
    "yndarray\n",
    "The matrix product of the inputs. This is a scalar only when both x1, x2 are 1-d vectors.\n",
    "\n",
    "**Raises**\n",
    "ValueError\n",
    "- If the last dimension of x1 is not the same size as the second-to-last dimension of x2.\n",
    "- If a scalar value is passed in.\n",
    "\n",
    "\n",
    "**The behavior depends on the arguments in the following way.**\n",
    "\n",
    "1. If both arguments are 2-D they are multiplied like conventional matrices.\n",
    "\n",
    "2. If either argument is N-D, N > 2, it is treated as a stack of matrices residing in the last two indexes and broadcast accordingly.\n",
    "\n",
    "3. If the first argument is 1-D, it is promoted to a matrix by prepending a 1 to its dimensions. After matrix multiplication the prepended 1 is removed.\n",
    "\n",
    "4. If the second argument is 1-D, it is promoted to a matrix by appending a 1 to its dimensions. After matrix multiplication the appended 1 is removed.\n",
    "\n",
    "**matmul differs from dot in two important ways:**\n",
    "\n",
    "- Multiplication by scalars is not allowed, use * instead.\n",
    "\n",
    "- Stacks of matrices are broadcast together as if the matrices were elements, respecting the signature (n,k),(k,m)->(n,m):broadcast together as if the matrices were elements, respecting the signature (n,k),(k,m)->(n,m):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4524a6a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e08e53",
   "metadata": {},
   "source": [
    "## Different combinations of vector and matrix multiplication:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4622fcb2",
   "metadata": {},
   "source": [
    "### 1. Vector x Vector dot product\n",
    "\n",
    "**Basic operations:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "acfbba21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 18, 28])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "a = np.array([2,3,4])\n",
    "b = np.array([5,6,7])\n",
    "\n",
    "a*b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "825aef20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a@b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2b2d3d8d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.dot(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cb586aee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.matmul(a,b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc42b4c3",
   "metadata": {},
   "source": [
    "Matmul has same results as dot product. Going to use dot product for this practice notebook. \n",
    "\n",
    "Also, taking transpose to convert vector dot product to vector multiplication is theoretic notation to understand how multiplication is done. Numpy functions undertand how to proceed, so no need to manually transpose while calling dot or matmul. Therefore, unless otherwise needed, like in the normal equation, don't take transpose, a.dot(b) or np.matmul(a,b) should take care of it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0efb6ca",
   "metadata": {},
   "source": [
    "**Commutative property:**\n",
    "\n",
    "- For dot prdouct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "25b3a96a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is vector dot product commutative?   True\n",
      "56\n"
     ]
    }
   ],
   "source": [
    "print(\"Is vector dot product commutative?  \", np.array_equal(a.dot(b), b.dot(a)))\n",
    "print (a.dot(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d950f3b",
   "metadata": {},
   "source": [
    "- For transpose of vectors within dot product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3d2d838c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transpose in vector dot product commutative?   True\n",
      "56\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transpose in vector dot product commutative?  \", \n",
    "      np.array_equal(a.T.dot(b),a.dot(b.T)))\n",
    "\n",
    "print(a.T.dot(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abb49d85",
   "metadata": {},
   "source": [
    "**Is regular vs transoposed (either one of the vectors or both) in vector multiplication equal:**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3fa009a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is vector dot product with a.dot(b) and a.T.dot(b) equal?   True\n",
      "Is vector dot product with b.dot(a) and b.T.dot(a) equal?   True\n"
     ]
    }
   ],
   "source": [
    "print(\"Is vector dot product with a.dot(b) and a.T.dot(b) equal?  \", \n",
    "      np.array_equal(a.T.dot(b),a.dot(b.T)))\n",
    "\n",
    "print(\"Is vector dot product with b.dot(a) and b.T.dot(a) equal?  \", \n",
    "      np.array_equal(b.dot(a),b.dot(a.T)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "133b7229",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transposing both vector and taking dot product commutative?   True\n",
      "Does transposing both vector and taking dot product give same result as a.dot(b)?   True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "56"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Is transposing both vector and taking dot product commutative?  \", \n",
    "      np.array_equal(a.T.dot(b.T),b.T.dot(a.T)))\n",
    "\n",
    "print(\"Does transposing both vector and taking dot product give same result as a.dot(b)?  \", \n",
    "      np.array_equal(a.T.dot(b.T),a.dot(b)))\n",
    "\n",
    "a.T.dot(b.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "053a0740",
   "metadata": {},
   "source": [
    "**Conclusion:** *Vector dot product is a scaler, so either way, a single number. Vector dot product has lot's of flexibility.*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9018fba6",
   "metadata": {},
   "source": [
    "### 2. Matrix x vector or vector x matrix:\n",
    "\n",
    "Vector is just a 1D matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e7b08f32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of vector is:  (3,)\n",
      "shape of matrix is:  (3, 3)\n"
     ]
    }
   ],
   "source": [
    "vector = np.array([2,3,4])\n",
    "matrix = np.array([[5,6,7],[5,6,7],[5,6,7]])\n",
    "\n",
    "print(\"shape of vector is: \", vector.shape)\n",
    "print(\"shape of matrix is: \", matrix.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "629828c4",
   "metadata": {},
   "source": [
    "vector is a column vector, with 3 rows and 1 column,\\\n",
    "matrix is a 3x3 matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c881a3",
   "metadata": {},
   "source": [
    "#### Basic operations:\n",
    "\n",
    "- vector into matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b4778b65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([45, 54, 63])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# By normal linear algebra, this should give an error, as number of columns of vector are 1,\n",
    "# and number of rows of matrix is 3.However, dot does some padding to make both compatible by inserting \n",
    "# columns of 1s, and then discarding their result after multiplication\n",
    "\n",
    "vector_into_matrix = vector.dot(matrix)\n",
    "vector_into_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1ece9916",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[15, 18, 21],\n",
       "       [15, 18, 21],\n",
       "       [45, 54, 63]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Explanation:\n",
    "\n",
    "vector_padded = np.array ([[1,1,1],[1,1,1], [2,3,4]])\n",
    "vector_padded_into_matrix = vector_padded.dot(matrix)\n",
    "vector_padded_into_matrix\n",
    "\n",
    "# As you can see, the result we get is the last row only, and first two rows are discarded."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a09bbe60",
   "metadata": {},
   "source": [
    "- matrix into vector: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b3f7ec8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([56, 56, 56])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this one is obvious, as number of columns of matrix are equal to number of rows of vector. \n",
    "# This is also most common, multipying matrix by vector in linear algebra\n",
    "matrix_into_vector = matrix.dot(vector)\n",
    "matrix_into_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aec618ca",
   "metadata": {},
   "source": [
    "#### Commutative property:\n",
    "\n",
    "- For dot product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "501e9d5a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is matrix-vector multiplication commutative?   False\n",
      "\n",
      "vector.dot(matrix) is:\n",
      " [45 54 63] \n",
      "\n",
      "matrix.dot(vector) is:\n",
      " [56 56 56]\n"
     ]
    }
   ],
   "source": [
    "# we've already seen it is not commutative\n",
    "\n",
    "print(\"Is matrix-vector multiplication commutative?  \", np.array_equal(vector.dot(matrix), matrix.dot(vector)))\n",
    "      \n",
    "print (\"\\nvector.dot(matrix) is:\\n\", vector.dot(matrix), \"\\n\\nmatrix.dot(vector) is:\\n\", matrix.dot(vector))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1143979f",
   "metadata": {},
   "source": [
    "- for transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7b5f45fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transpose in vector-matrix multiplication commutative?   False\n",
      "\n",
      "vector.dot(matrix.T) is:\n",
      " [56 56 56] \n",
      "\n",
      "vector.T.dot(matrix) is:\n",
      " [45 54 63]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transpose in vector-matrix multiplication commutative?  \", np.array_equal(vector.dot(matrix.T), vector.T.dot(matrix)))\n",
    "      \n",
    "print (\"\\nvector.dot(matrix.T) is:\\n\", vector.dot(matrix.T), \"\\n\\nvector.T.dot(matrix) is:\\n\", vector.T.dot(matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "f4b6f062",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transpose in matrix-vector multiplication commutative?   False\n",
      "\n",
      "matrix.dot(vector.T) is:\n",
      " [56 56 56] \n",
      "\n",
      "matrix.T.dot(vector) is:\n",
      " [45 54 63]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transpose in matrix-vector multiplication commutative?  \", np.array_equal(matrix.dot(vector.T), matrix.T.dot(vector)))\n",
    "      \n",
    "print (\"\\nmatrix.dot(vector.T) is:\\n\",matrix.dot(vector.T), \"\\n\\nmatrix.T.dot(vector) is:\\n\", matrix.T.dot(vector))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00219ac6",
   "metadata": {},
   "source": [
    "**Is regular vs transposed (either one of the vectors or both) in vector-matrix or matrix-vector multiplication equal?**\n",
    "\n",
    "This is interesting. While the commutative property does not hold for dot product or transpose when order of multiplication is kept same, it holds when second element in multiplication is transposed, or order of multiplication is reveresed along with transpose:\n",
    "\n",
    "`vector.dot(matrix) = vector.T.dot(matrix) = matrix.T.dot(vector) = [45 54 63] `\\\n",
    "`matrix.dot(vector) = matrix.dot(vector.T) = vector.dot(matrix.T)= [56 56 56] `\n",
    "\n",
    "A little weird, but point is to follow equations and multiply accordingly, in correct order. \n",
    "for the sake of completeness, taking transpose of both vector and matrix and checking result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0b0c54f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transposing both matrices and multiplying commutative?   False\n",
      "\n",
      "vector.T.dot(matrix.T) is:\n",
      " [56 56 56] \n",
      "\n",
      "matrix.T.dot(vector.T) is:\n",
      " [45 54 63]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transposing both matrices and multiplying commutative?  \", np.array_equal(vector.T.dot(matrix.T), matrix.T.dot(vector.T)))\n",
    "\n",
    "print (\"\\nvector.T.dot(matrix.T) is:\\n\", vector.T.dot(matrix.T), \"\\n\\nmatrix.T.dot(vector.T) is:\\n\", matrix.T.dot(vector.T))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b1a6af",
   "metadata": {},
   "source": [
    "**Conclusion:** *add checks in code to see if multiplication was done correctly.*\n",
    "\n",
    "\n",
    "`vector.dot(matrix) = vector.T.dot(matrix) = matrix.T.dot(vector) = matrix.T.dot(vector.T) = [45 54 63] `\n",
    "`matrix.dot(vector) = matrix.dot(vector.T) = vector.dot(matrix.T)= vector.T.dot(matrix.T) = [56 56 56] `\n",
    "\n",
    "A vector and its transpose seem to be pretty much the same, probably due to dimensions being adjusted to facilitate multiplication. Not sure.\n",
    "\n",
    "I would narrow it down as:\n",
    "\n",
    "`vector.dot(matrix) = matrix.T.dot(vector) = [45 54 63] `\n",
    "`matrix.dot(vector) = vector.dot(matrix.T) = [56 56 56] `\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a075439",
   "metadata": {},
   "source": [
    "### 3. Matrix x Matrix Multiplication:\n",
    "\n",
    "**Is matrix multiplication same as dot product?**\n",
    "\n",
    "In short yes, each entry in the product matrix is the dot product of a row in the first matrix and a column in the second matrix.\n",
    "\n",
    "\n",
    "#### Basic operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "557d1f6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[10, 18, 28],\n",
       "       [10, 18, 28],\n",
       "       [10, 18, 28]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x= np.array([[2,3,4],[2,3,4],[2,3,4]])\n",
    "y= np.array([[5,6,7],[5,6,7],[5,6,7]])\n",
    "\n",
    "x*y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e4d7318d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[45, 54, 63],\n",
       "       [45, 54, 63],\n",
       "       [45, 54, 63]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.dot(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2388b13",
   "metadata": {},
   "source": [
    "#### Commutative property:\n",
    "\n",
    "- For dot product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "0632ec31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is matrix multiplication commutative?   False\n",
      "\n",
      "x.dot(y) is:\n",
      " [[45 54 63]\n",
      " [45 54 63]\n",
      " [45 54 63]] \n",
      "\n",
      "y.dot(x) is:\n",
      " [[36 54 72]\n",
      " [36 54 72]\n",
      " [36 54 72]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is matrix multiplication commutative?  \", np.array_equal(x.dot(y), y.dot(x)))\n",
    "      \n",
    "print (\"\\nx.dot(y) is:\\n\", x.dot(y), \"\\n\\ny.dot(x) is:\\n\", y.dot(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac9945e",
   "metadata": {},
   "source": [
    "- For transpose of vectors within dot product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "2db70c3f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transpose in matrix multiplication commutative?   False\n",
      "\n",
      "x.T.dot(y) is:\n",
      " [[30 36 42]\n",
      " [45 54 63]\n",
      " [60 72 84]] \n",
      "\n",
      "y.T.dot(x) is:\n",
      " [[30 45 60]\n",
      " [36 54 72]\n",
      " [42 63 84]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transpose in matrix multiplication commutative?  \", np.array_equal(x.T.dot(y), y.T.dot(x)))\n",
    "      \n",
    "print (\"\\nx.T.dot(y) is:\\n\", x.T.dot(y), \"\\n\\ny.T.dot(x) is:\\n\", y.T.dot(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2085f4f",
   "metadata": {},
   "source": [
    "**Is regular vs transposed (either one of the vectors or both) in vector multiplication equal?**\n",
    "\n",
    "None of the transposed multiplication is equal to any of the x.dot(y) or y.dot(x): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "f3b23168",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print (np.array_equal(x.dot(y), x.T.dot(y))) \n",
    "\n",
    "print (np.array_equal(y.dot(x), y.T.dot(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6320f26b",
   "metadata": {},
   "source": [
    "Also, transposing both matrices is also completely different from each other or any of the calculations before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "a176c58d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is transposing both matrices and multiplying commutative?   False\n",
      "\n",
      "x.T.dot(y.T) is:\n",
      " [[36 36 36]\n",
      " [54 54 54]\n",
      " [72 72 72]] \n",
      "\n",
      "y.T.dot(x.T) is:\n",
      " [[45 45 45]\n",
      " [54 54 54]\n",
      " [63 63 63]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Is transposing both matrices and multiplying commutative?  \", np.array_equal(x.T.dot(y.T), y.T.dot(x.T)))\n",
    "\n",
    "print (\"\\nx.T.dot(y.T) is:\\n\", x.T.dot(y.T), \"\\n\\ny.T.dot(x.T) is:\\n\", y.T.dot(x.T))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92609b2b",
   "metadata": {},
   "source": [
    "**Conclusion:** *Matrix multiplication has completely different rules from vector multiplication. Matrix multiplication strictly depends on order of multiplication, and taking transpose of either or both matrices changes the results completely.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a04cf8",
   "metadata": {},
   "source": [
    "### 4. Normal Equation:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1495d24",
   "metadata": {},
   "source": [
    "`X` = feature matrix\n",
    "`w` = weights of the feature matrix\n",
    "`y` = traget variable to be predicted\n",
    "\n",
    "`X.w = y`\n",
    "Now, to find w which are the weights, is general LA, we'd do:\n",
    "\n",
    "`w =  inv(X).y`\n",
    "\n",
    "However, since X is a matrix, and it could be a singular matrix (meaning it's determinant is zero, because  its rows are linearly dependent vectors, or its columns are linearly dependent vectors). On top of it, X has to be square to find determinant in the first place. So we multiply X with X.T in left hand order, to make sure we have a square matrix and creating a Gram matrix. Found this online: \"The column Gram matrix (A.T.A) is not invertible only if columns of A are linearly dependent. Thus if columns of A are linearly independent then the Gram matrix is invertible.\"  Anyways, now we have a matrix we can possibly invert:\n",
    "\n",
    "\n",
    "`(X.T.X).w = y\n",
    "inv(X.T.X).(X.T.X).w = inv(X.T.X).X.T.y\n",
    "I.w = inv(X.T.X).X.T.y\n",
    "w = inv(X.T.X).X.T.y`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f0f28f",
   "metadata": {},
   "source": [
    "Overall euqation looks like this:\n",
    "\n",
    "`inv(X.T.X).(X.T.X).w = inv(X.T.X).(X.T).y`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (mlzoomcamp)",
   "language": "python",
   "name": "mlzoomcamp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}