Simple Linear Regression: Predicting Values with a Line

Simple linear regression is a statistical technique for modeling the relationship between a dependent variable (Y, what we want to predict) and an independent variable (X, what we use to predict). The idea is to find the line that best fits the data by minimizing prediction errors.

The classic example is predicting the price of a house (Y) based on its area in square meters (X). Intuitively, we expect larger houses to cost more — regression quantifies that relationship precisely and allows predictions for new houses.

The regression line is described by the equation ŷ = a + bx, where ŷ is the predicted value of Y for a given X, b is the slope coefficient and a is the intercept (where the line crosses the Y-axis when X = 0).

The slope b indicates how much Y changes on average for each unit increase in X. If a house price regression gives b = 3,500, this means each additional square meter is associated with an average increase of $3,500 in price.

The intercept a is the predicted value of Y when X = 0. In many practical contexts, this value has no direct interpretation — a house with 0 m² does not exist. But mathematically it is necessary to complete the line equation.

R² (the coefficient of determination) measures the proportion of the variation in Y that is explained by the regression model. It ranges from 0 to 1 (or 0% to 100%). An R² of 0.85 means that 85% of the variation in house prices is explained by the variation in area, while the remaining 15% is due to other factors not included in the model.

High R² does not mean the model is good in all contexts. In social sciences, R² of 0.4 may already be considered robust. In experimental physics, R² above 0.99 is expected. R² also does not indicate whether the relationship is causal — only how well the line fits the data.

Note that R² is literally the square of Pearson's r correlation coefficient in simple regression. If the correlation between area and price is r = 0.92, then R² = 0.85 — the same 85% mentioned above.

Linear regression assumes that the relationship between X and Y is approximately linear (check with a scatter plot), that residuals are approximately normal with constant variance (homoscedasticity), and that observations are independent of each other.

The ordinary least squares method — which minimizes the sum of squared residuals — is the most widely used to estimate a and b. It is sensitive to outliers: a single point far from the general trend can pull the line significantly. In practical analyses, always check for influential points.

Extrapolation — using the model to predict Y with X values far beyond the range of the data — is risky. The line may behave in a completely different way outside the observed limits. The model for temperature and ice cream sales, for example, cannot reliably predict sales at −30°C.