Regression

A regression is a statistical technique that relates a dependent variable to one or more independent (explanatory) variables. You predict a continuous value, rather than discrete classes such as in Classification.

We can use L1 and L2 distance to solve regression problems. In Stanford CS231n, it was a Classification problem, where you used the SVM loss to predict the class.

Linear Regression

Method 1: Using MLE

Simple linear regression model: $Y_i\sim N(\alpha +\beta x_i,{\sigma }^2)$ Alternate Formulation $Y_i=\alpha +\beta x_i+R_i,R_i\sim N(0,{\sigma }^2)$

We use data $(x_1,y_1),\dots ,(x_n,y_n)$ to estimate $\alpha ,\beta ,{\sigma }^2$

The Likelihood Function is given by ?? i am too lazy to put this

We come up with the line of best fit using MLEs. We get the following results (derivation is at page 402) for the estimates of the parameters $\hat{\alpha },\hat{\beta },{\hat{\sigma }}^2$: $\hat{\alpha }=\overline{y}-\hat{\beta }\overline{x}$ $\hat{\beta }=\frac{S_{xy}}{S_{xx}}=\frac{\sum [(x_i-\overline{x})(y_i-\overline{y})]}{\sum (x_i-\overline{x}{)}^2}$ ${\hat{\sigma }}^2=\frac{1}{n}\sum (y_i-(\hat{\alpha }-\hat{\beta }{x}_i){)}^2$

The line of best fit is given by $y=\hat{\alpha }+\hat{\beta }X$

$\beta =0⟹$ $x$ has no predictive power for $Y$.

Method 2: Least Squares

I don’t think the teacher went too in depth for this… They both end up with the same final equation.

We are making the Gauss-Markov Theorem

We want to ask if $\beta =0⟹$ if $\beta =0$ then $x$ has no predictive power for $Y$ Suppose H_0$:$\beta = 0H_1$:$\beta \neq 0$

You do hypothesis testing, where it is given by $\frac{|\hat{\beta }-{\beta }_0|}{s_e/\sqrt{S_{xx}}}$

• Note that ${\beta }_0=0$ (since this is the hypothesis we are testing) And then you use your t-table, where your Degrees of Freedom is $n-2$.

• Least Squares