Let us say that we expect a linear relation, say y=ax+b, between two observable variables x and y and where a and b are not known. It is not necessary that the relation should be linear, but here we have assumed this only for the purpose of simplification.
However, actual observed data may be accompanied by errors or noise for various reasons and when we plot the observed data on a graph, it may not be a linear and may be some thing as shown below.
Observe that each point denotes observed value of x and y and n points provides us with n data points given by (x_i,y_i), where i ranges from 1 to n.
We can draw many lines through these points, with varying slopes i.e. a's and intercepts b's, but how do we know which one is the best fit. This is done by the method of Least squares. In this method we define the relationship between observed data and expected data - by minimizing the sum of squares of the deviation between observed and expected values.
In other words, best line has minimum error between line and data points. Note that had we not squared, positive and negative errors would have almost cancelled out. We will talk more about this later.@
For a particular x_i observed value is y_i, expected value of y_i for is ax_i+b and difference between them d_i is given by y_i-ax_i-b and in least square method we seek to
minimise error E=sum_(i=1)^nd_i^2 i.e. sum_(i=1)^n(y_i-ax_i-b)^2.
To get this minimum, we use calculus. For this, we must have first derivatives of a and b yielding zero. Differentiating sum_(i=1)^n(y_i-ax_i-b)^2 w.r.t. a and b, we get
(delE)/(dela)=-2sum_(i=1)^nx_i(y_i-ax_i-b)=0
and (delE)/(delb)=-2sum_(i=1)^n(y_i-ax_i-b)=0
To solve for a and b, we rewrite them as
asumx_i^2+bsumx_i=sumx_iy_i and
asumx_i+bn=sumy_i
and solving them for a and b we get
a=(sumy_isumx_i^2-sumx_isumx_iy_i)/(nsumx_i^2-(sumx_i)^2)
and b=(nsumx_iy_i-sumx_isumy_i)/(nsumx_i^2-(sumx_i)^2)
@ - Note that asumx_i+bn=sumy_i can be expressed as asumx_i/n+b=sumy_i/n is just a fit between averages of x_i and y_i. Hence, this alone may not give the best fit. The latter is arrived due to asumx_i^2+bsumx_i=sumx_iy_i.
