Server Help :: View topic - Least Squares vs. Least Absolute

Server Help

Trash Talk - Least Squares vs. Least Absolute

Muskrat - Sat Oct 11, 2008 7:34 pm
Post subject: Least Squares vs. Least Absolute

Here's a question for math people:

The common practice in function approximation is to the use the minimized sum of squares to find error. Why not absolute value?

Using squares gives more weight to big screw-ups and less to small screw-ups. Why not just give the amount of weight due to them? I know this will depend on someone's definition of "best fit," but I need some ideas.

Dr Brain - Sat Oct 11, 2008 9:48 pm
Post subject:

MMSE ususally results in a nicer closed form. Especially with normally (aka Gaussian) distributed functions.

SamHughes - Mon Oct 13, 2008 9:22 pm
Post subject: Re: Least Squares vs. Least Absolute

Let's step down a dimension, from functions in R->R to real numbers themselves, i.e. functions in {0} -> R, you could say.

The analogue to a least squares approximation is the average. The average, m = (a_1 + ... + a_n) / n, is the value that minimizes the sum, ((a_1 - m)^2 + ... + (a_n - m)^2).

Suppose that, instead, we wanted the value for m that minimizes the expression (|a_1 - m| + ... + |a_n - m|).

What value is that? Well, if n is odd, the value's the median. If n is even, you can use any value between the two closest to the median. Is the median what we want? It depends on what we want.

Now let's go back to the least squares approximation. It turns out that the same thing happens: if you want to find the line such that the absolute value distance is minimized, you'll have half the points above the line, and half the points below the line, and there isn't a unique solution.

But why not something in between? Why not minimize (|y_1 - f(x_1)|^1.5 + ... + |y_n - f(y_n)|^1.5) or something else? People prefer minimizing the sum of squares because that makes the computations easy and fast -- you can use simple principles to compute the solution -- you're really just finding the minimum of some paraboloid. And there is a huge mathematical framework built around this.

So people use least squares a lot because it makes the math easy, and beacuse it fits with other assumptions (that are preferred because they make the math easy), and people _definitely_ don't use minimal sum of absolute distances because there is not one solution. If you have sample points (0, 0), (100, -1), (100, 1), least squares makes you split the uprights, while sum of absolute distances only cares if you get it in.

Cheese - Wed Oct 15, 2008 11:23 pm
Post subject:

|3| = 3
|-3| = 3

Samapico - Wed Oct 15, 2008 11:39 pm
Post subject:

Cheese wrote:

|3| = 3
|-3| = 3

rootbear75 - Wed Oct 22, 2008 3:40 am
Post subject:

the perfect number: 6

Samapico - Wed Oct 22, 2008 12:56 pm
Post subject:

SamHughes - Wed Oct 22, 2008 10:42 pm
Post subject:

Why was 6 afraid of 20? Because 28 496.

Samapico - Wed Oct 22, 2008 11:48 pm
Post subject:

...

What?

What does 20 have to do with that?