Regression
A regression is a statistical technique that relates a dependent variable to one or more independent (explanatory) variables. You predict a continuous value, rather than discrete classes such as in Classification.
We can use L1 and L2 distance to solve regression problems. In Stanford CS231n, it was a Classification problem, where you used the SVM loss to predict the class.
Linear Regression
Method 1: Using MLE
Simple linear regression model: $Y_{i}βΌN(Ξ±+Ξ²x_{i},Ο_{2})$ Alternate Formulation $Y_{i}=Ξ±+Ξ²x_{i}+R_{i},R_{i}βΌN(0,Ο_{2})$
We use data $(x_{1},y_{1}),β¦,(x_{n},y_{n})$ to estimate $Ξ±,Ξ²,Ο_{2}$
The Likelihood Function is given by ?? i am too lazy to put this
We come up with the line of best fit using MLEs. We get the following results (derivation is at page 402) for the estimates of the parameters $Ξ±,Ξ²β,Ο_{2}$: $Ξ±=yββΞ²βx$ $Ξ²β=S_{xx}S_{xy}β=β(x_{i}βx)_{2}β[(x_{i}βx)(y_{i}βyβ)]β$ $Ο_{2}=n1ββ(y_{i}β(Ξ±βΞ²βx_{i}))_{2}$
The line of best fit is given by $y=Ξ±+Ξ²βX$
$Ξ²=0βΉ$ $x$ has no predictive power for $Y$.
Method 2: Least Squares
I donβt think the teacher went too in depth for thisβ¦ They both end up with the same final equation.
We are making the GaussMarkov Theorem
We want to ask if $Ξ²=0βΉ$ if $Ξ²=0$ then $x$ has no predictive power for $Y$ Suppose H_0$:$\beta = 0H_1$:$\beta \neq 0$
You do hypothesis testing, where it is given by $s_{e}/S_{xx}ββ£Ξ²ββΞ²_{0}β£β$

Note that $Ξ²_{0}=0$ (since this is the hypothesis we are testing) And then you use your ttable, where your Degrees of Freedom is $nβ2$.