Regression

A regression is a statistical technique that relates a dependent variable to one or more independent (explanatory) variables. You predict a continuous value, rather than discrete classes such as in Classification.

We can use L1 and L2 distance to solve regression problems. In Stanford CS231n, it was a Classification problem, where you used the SVM loss to predict the class.

Practice implementing it: - https://www.deep-ml.com/problems/14

Linear Regression

Method 1: Using MLE

Simple linear regression model: $Y_i\sim N(\alpha +\beta x_i,{\sigma }^2)$ Alternate Formulation $Y_i=\alpha +\beta x_i+R_i,R_i\sim N(0,{\sigma }^2)$

We use data $(x_1,y_1),\dots ,(x_n,y_n)$ to estimate $\alpha ,\beta ,{\sigma }^2$

The Likelihood Function is given by ?? i am too lazy to put this

We come up with the line of best fit using MLEs. We get the following results (derivation is at page 402) for the estimates of the parameters $\hat{\alpha },\hat{\beta },{\hat{\sigma }}^2$: $\hat{\alpha }=\overline{y}-\hat{\beta }\overline{x}$ $\hat{\beta }=\frac{S_{xy}}{S_{xx}}=\frac{\sum [(x_i-\overline{x})(y_i-\overline{y})]}{\sum (x_i-\overline{x}{)}^2}$ ${\hat{\sigma }}^2=\frac{1}{n}\sum (y_i-(\hat{\alpha }-\hat{\beta }{x}_i){)}^2$

The line of best fit is given by $y=\hat{\alpha }+\hat{\beta }X$

$\beta =0⟹$ $x$ has no predictive power for $Y$.

Method 2: Least Squares

I don’t think the teacher went too in depth for this… They both end up with the same final equation.

We are making the Gauss-Markov Theorem

We want to ask if $\beta =0⟹$ if $\beta =0$ then $x$ has no predictive power for $Y$ Suppose H_0$:$\beta = 0H_1$:$\beta \neq 0$

You do hypothesis testing, where it is given by $\frac{|\hat{\beta }-{\beta }_0|}{s_e/\sqrt{S_{xx}}}$

• Note that ${\beta }_0=0$ (since this is the hypothesis we are testing) And then you use your t-table, where your Degrees of Freedom is $n-2$.

Other

From the deep-ml practice, they just use some formula?

In linear regression, you’re trying to find parameters ($\theta$) that make predictions: $\hat{y}=X\theta$

• $X$ is your design matrix (rows = training examples, columns = features, often with a column of ones for the bias term).
• $y$ is the vector of actual target values.
• $\theta$ is the vector of parameters (weights) you’re solving for.

The normal equation is derived by minimizing the cost function:

$J(\theta )=\frac{1}{2m}(X\theta -y{)}^T(X\theta -y)$

You take the derivative with respect to $\theta$, set it to zero, and solve. That derivation gives you:

$\theta =(X^{T}X{)}^{-1}{X}^{T}y$

So in short:

• $\theta$ is the closed-form solution for the weights of your linear regression model that minimize the squared error.
• It “came from” solving for the best parameters that make your prediction line (or hyperplane) fit the data as best as possible.

Normally, a linear regression model looks like this: $h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n$

To express this in matrix form, we add a column of ones to $X$:

$$X=\left[\begin{array}{ccccc}1& x_{11}& x_{12}& \cdots & x_{1n}\\ 1& x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& ⋮& & ⋮\\ 1& x_{m1}& x_{m2}& \cdots & x_{mn}\end{array}\right]$$

In the deep ML problem, they don’t have a set of features equal to 1, which does not allow us to encode a bias value.

“A practical implementation involves augmenting ( X ) with a column of ones to account for the intercept term and then applying the normal equation directly to compute ( \theta ).“.