When creating a predictive model, we need to compute coefficients, as they are rarely known beforehand. To estimate regression coefficients, we use a standard ordinary least squares (OLS) matrix coefficient estimator:

Understanding this formula’s operations requires familiarity with matrix notation. But at present, all we need to understand is that the size and contents of the X matrix are determined by the independent variables chosen as the model’s parameters. Moreover, the degree of correlation between predictor variables—known as correlation coefficients and represented by —are used in calculating regression coefficients between X and Y.³

As independent variables are included or excluded from the model, the estimated coefficients for any one predictor can change drastically, making coefficient estimates unreliable and imprecise. Correlation between two or more predictors creates difficulty in determining any one variable’s individual impact on the model output. Remember that a regression coefficient measures a given predictor variable’s effect on the output assuming other predictors remain constant. But if predictors are correlated, it may not be possible to isolate predictors. Thus, estimated regression coefficients for multicollinear variables do not reflect any one predictor’s effect on the output but rather the predictor’s partial effect, depending on which covariates are in the model.⁴

Additionally, different data samples, or even small changes in data, with the same multicollinear variables can produce widely different regression coefficient. This is perhaps the most widely known problem of multicollinearity: overfitting. Overfitting denotes models with low training error and high generalization error. As mentioned, statistical significance of any one multicollinear variable remains unclear amidst its relational noise with the others. This prevents precise calculation of any one variable’s statistical significance on the model’s output, which is what the coefficient estimate largely indicates. Because multicollinearity prevents calculating precise coefficient estimates, multicollinear models fail to generalize onto unseen data. In this way, estimated coefficients for multicollinear variables possess a large variability, also known as a large standard error.⁵

Statistics textbooks and articles sometimes divide between extreme and perfect multicollinearity. Perfect multicollinearity signifies when one independent variable has a perfect linear correlation with one or more independent variables. Extreme multicollinearity is when one predictor is highly correlated with one or more additional independent variables.⁶ These are the two principal degrees of multicollinearity.

There are not so much discrete forms of multicollinearity as different potential causes. These causes can range from the nature of data under consideration to poorly designed experiments. Some common causes are:

- Data collection This data-based multicollinearity can result if one samples a non-representative subspace of the data in question. For instance, Montgomery et al. supply the example of a supply chain delivery dataset in which order distance and size are independent variables of a predictive model. In the data they provide, order inventory size appears to increase with delivery distance. The solution to this correlation is straightforward: collect and include data samples for short distance deliveries with large inventories, or vice-versa.⁷

- Model constraints This is similar to the data collection cause, albeit not identical. Multicollinearity can result due to the nature of the data and predictive model variables in question. Imagine we are creating a predictive model to measure employee satisfaction in the workplace, with hours worked per week and reported stress being two of several predictors. There may very well be a correlation between these predictors due to the nature of the data—i.e. people who work more will likely report higher stress. A similar situation may occur if education and salary are model predictors—employees with more education will likely earn more. In this case, collecting more data may not alleviate the issue, as multicollinearity is inherent to the data itself.

- Overdefined model Multicollinearity can occur when there are more model predictors than data observation points. This issue can arise particularly in biostatistics or other biological studies. Resolving overdefined models requires eliminating select predictors from the model altogether. But how to determine which models to remove? One can conduct a several preliminary studies using subsets of regressors (i.e. predictors) or utilize principal component analysis (PCA) to combine multicollinear variables.⁸

Select types of data can especially lead to multicollinearity. Time series data is chief among these. Growth and trends factors, notably in economics, often move in the same direction over time, readily producing multicollinearity. Additionally, observational studies in social sciences are readily conducive to multicollinearity, since many socioeconomic variables (e.g. income, education, political affiliation, etc.) are often interrelated and uncontrolled by researchers.⁹

Multicollinearity can also result from manipulation of predictor variables. In some cases, one may use the squared or lagged values of independent variables as new model predictors. Of course, these new predictors will share a high correlation with the independent variables from whence they were derived.¹⁰ This is structural multicollinearity.

Large estimated coefficients in themselves can indicate the presence of multicollinearity, as well as massive changes in estimated coefficients when a single predictor (or even data point) is added or removed from the model. Coefficients with large confidence intervals are also indicative of multicollinearity. Occasionally, coefficients possessing signs or magnitudes contrary to expectations derived from preliminary data analysis can indicate multicollinearity. Of course, none of these definitively confirm multicollinearity nor provide quantitative measurements of multicollinearity.¹¹ Several diagnostic methods help do so however.

Two relatively simple tools for measuring multicollinearity are a scatter plot and correlation matrix of independent variables. When using a scatter plot, one plots independent variable values for each data point against one another. If the scatter plot reveals a linear correlation between the chosen variables, then some degree of multicollinearity may be present. This figure illustrates multicollinear data in a scatter plot using the Montgomery et al. delivery dataset example.

Another diagnostic method is to calculate a correlation matrix for all the independent variables. The elements of the matrix are the correlation coefficients between each predictor in a model. The correlation coefficient is a value between -1 and 1 that measures the degree of correlation between two predictors. Note how the matrix contains a diagonal of 1s because each variable has a perfect correlation with itself. The higher a given matrix element, the greater the degree of correlation between them.¹²

Variance inflation factor (VIF) is the most common method for determining the degree of multicollinearity in linear regression models. Each model predictor has a VIF value, which measures how much the variance of that predictor is inflated by the model’s other predictors.

The VIF algorithm contains several steps. A full explanation of this algorithm is beyond the scope of this article however. Suffice it to say, VIF measures a chosen variable’s proportion of the variance as determined by the model’s other independent variables. The equation representing VIF is:

R-squared (R²) signifies the coefficient of multiple determination obtained by regressing one independent variable against all the others.¹³ The bottom term of the VIF equation is tolerance, a concept distinct from tolerance intervals. Tolerance is the inverse of VIF. Though much less discussed in literature, it is nevertheless another viable means for calculating multicollinearity.¹⁴

The higher the VIF value, the greater degree of multicollinearity. There is no VIF cutoff value determining a “bad” or “good” model. Nevertheless, a widely repeated rule of thumb is that a VIF value greater than or equal to ten indicates severe multicollinearity.¹⁵

Note that R and Python contain functions for calculating VIF. Respectively, the vif() function in R’s car package and the variance_inflation_factor() function in Python’s statsmodels.stats module can compute VIF for a designated model.¹⁶