In statistics, an autoregressive (AR) model is a modelled representation of a type of random process. It can be used to describe time-varying processes from many natural and artificial sources. The model specifies output variables that are dependent linearly on their own previous values on a stochastic basis. The model is in the form of a stochastic difference equation (or recurrence relation) which should not be confused with a differential equation. Together with the moving-average (MA) model, it is a special case and key component of the more general autoregressive–moving-average (ARMA) and autoregressive integrated moving average (ARIMA) models of time series, which have a more complicated stochastic structure; it is also a special case of the vector autoregressive model (VAR), which consists of a system of more than one interlocking stochastic difference equation in more than one evolving random variable.
Another important extension is the time-varying autoregressive (TVAR) model, where the autoregressive coefficients are allowed to change over time to model evolving or non-stationary processes. TVAR models are widely applied in cases where the underlying dynamics of the system are not constant, such as in sensors time series modelling, climate science, economics and finance (as econometrics), signal processing, telecommunications, radar systems, and biological signals.
Unlike the moving-average (MA) model, the autoregressive model is not always stationary; non-stationarity can arise either due to the presence of a unit root or due to time-varying model parameters, as in time-varying autoregressive models.
Large language models are called autoregressive, but they are not a classical autoregressive model in this sense because they are not linear.
Definition
The notation indicates an autoregressive model of order p. The AR(p) model is defined as
where are the parameters of the model, and is white noise. This can be equivalently written using the backshift operator B as
so that, moving the summation term to the left side and using polynomial notation, we have
An autoregressive model can thus be viewed as the output of an all-pole infinite impulse response filter whose input is white noise.
Some parameter constraints are necessary for the model to remain weak-sense stationary. For example, processes in the AR(1) model with are not stationary. More generally, for an AR(p) model to be weak-sense stationary, the roots of the polynomial must lie outside the unit circle, i.e., each (complex) root must satisfy (see pages 89,92 ).
Intertemporal effect of shocks
In an AR process, a one-time shock affects values of the evolving variable infinitely far into the future. For example, consider the AR(1) model . A non-zero value for at say time t=1 affects by the amount . Then by the AR equation for in terms of , this affects by the amount . Then by the AR equation for in terms of , this affects by the amount . Continuing this process shows that the effect of never ends, although if the process is stationary then the effect diminishes toward zero in the limit.
Because each shock affects X values infinitely far into the future from when they occur, any given value Xt is affected by shocks occurring infinitely far into the past. This can also be seen by rewriting the autoregression
(where the constant term has been suppressed by assuming that the variable has been measured as deviations from its mean) as
When the polynomial division on the right side is carried out, the polynomial in the backshift operator applied to has an infinite order—that is, an infinite number of lagged values of appear on the right side of the equation.
Characteristic polynomial The autocorrelation function of an AR(p) process can be expressed as
where are the roots of the polynomial
where B is the backshift operator, where is the function defining the autoregression, and where are the coefficients in the autoregression. The formula is valid only if all the roots have multiplicity 1.
The autocorrelation function of an AR(p) process is a sum of decaying exponentials. Each real root contributes a component to the autocorrelation function that decays exponentially. Similarly, each pair of complex conjugate roots contributes an exponentially damped oscillation.
Graphs of AR(p) processes
The simplest AR process is AR(0), which has no dependence between the terms. Only the error/innovation/noise term contributes to the output of the process, so in the figure, AR(0) corresponds to white noise.
For an AR(1) process with a positive , only the previous term in the process and the noise term contribute to the output. If is close to 0, then the process still looks like white noise, but as approaches 1, the output gets a larger contribution from the previous term relative to the noise. This results in a "smoothing" or integration of the output, similar to a low pass filter.
For an AR(2) process, the previous two terms and the noise term contribute to the output. If both and are positive, the output will resemble a low pass filter, with the high frequency part of the noise decreased. If is positive while is negative, then the process favors changes in sign between terms of the process. The output oscillates. This can be linked to edge detection or detection of change in direction.
Example: An AR(1) process
An AR(1) process is given by:where is a white noise process with zero mean and constant variance . (Note: The subscript on has been dropped.) The process is weak-sense stationary if since it is obtained as the output of a stable filter whose input is white noise. (If then the variance of depends on time lag t, so that the variance of the series diverges to infinity as t goes to infinity, and is therefore not weak-sense stationary.) Assuming , the mean is identical for all values of t by definition of weak sense stationarity. If the mean is denoted by , it follows fromthatand hence
The variance is
where is the standard deviation of . This can be shown by noting that
and then by noticing that the quantity above is a stable fixed point of this relation.
The autocovariance is given by
It can be seen that the autocovariance function decays with a decay time (also called time constant) of .Lai, Dihui; and Lu, Bingfeng; "Understanding Autoregressive Model for Time Series as a Deterministic Dynamic System" , in Predictive Analytics and Futurism, June 2017, number 15, June 2017, pages 7-9
The spectral density function is the Fourier transform of the autocovariance function. In discrete terms this will be the discrete-time Fourier transform:
This expression is periodic due to the discrete nature of the , which is manifested as the cosine term in the denominator. If we assume that the sampling time () is much smaller than the decay time (), then we can use a continuum approximation to :
which yields a Lorentzian profile for the spectral density:
where is the angular frequency associated with the decay time .
An alternative expression for can be derived by first substituting for in the defining equation. Continuing this process N times yields
For N approaching infinity, will approach zero and:
It is seen that is white noise convolved with the kernel plus the constant mean. If the white noise is a Gaussian process then is also a Gaussian process. In other cases, the central limit theorem indicates that will be approximately normally distributed when is close to one.
For , the process will be a geometric progression (exponential growth or decay). In this case, the solution can be found analytically: whereby is an unknown constant (initial condition).
Explicit mean/difference form of AR(1) process
The AR(1) model is the discrete-time analogy of the continuous Ornstein-Uhlenbeck process. It is therefore sometimes useful to understand the properties of the AR(1) model cast in an equivalent form. In this form, the AR(1) model, with process parameter , is given by
, where , is the model mean, and is a white-noise process with zero mean and constant variance .
By rewriting this as and then deriving (by induction) , one can show that
and
Choosing the maximum lag
The partial autocorrelation of an AR(p) process equals zero at lags larger than p, so the appropriate maximum lag p is the one after which the partial autocorrelations are all zero.
Calculation of the AR parameters
There are many ways to estimate the coefficients, such as the ordinary least squares procedure or method of moments (through Yule–Walker equations).
The AR(p) model is given by the equation
It is based on parameters where i = 1, ..., p. There is a direct correspondence between these parameters and the covariance function of the process, and this correspondence can be inverted to determine the parameters from the autocorrelation function (which is itself obtained from the covariances). This is done using the Yule–Walker equations.
Yule–Walker equations
The Yule–Walker equations, named for Udny Yule and Gilbert Walker,Yule, G. Udny (1927) "On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers" , Philosophical Transactions of the Royal Society of London, Ser. A, Vol. 226, 267–298.]Walker, Gilbert (1931) "On Periodicity in Series of Related Terms" , Proceedings of the Royal Society of London, Ser. A, Vol. 131, 518–532. are the following set of equations.
where , yielding equations. Here is the autocovariance function of Xt, is the standard deviation of the input noise process, and is the Kronecker delta function.
Because the last part of an individual equation is non-zero only if , the set of equations can be solved by representing the equations for in matrix form, thus getting the equation
which can be solved for all The remaining equation for m = 0 is
which, once are known, can be solved for
An alternative formulation is in terms of the autocorrelation function. The AR parameters are determined by the first p+1 elements of the autocorrelation function. The full autocorrelation function can then be derived by recursively calculating
Examples for some Low-order AR(p) processes p=1
Hence p=2 The Yule–Walker equations for an AR(2) process are
Remember that Using the first equation yields Using the recursion formula yields
Estimation of AR parameters
The above equations (the Yule–Walker equations) provide several routes to estimating the parameters of an AR(p) model, by replacing the theoretical covariances with estimated values. Some of these variants can be described as follows:
Estimation of autocovariances or autocorrelations. Here each of these terms is estimated separately, using conventional estimates. There are different ways of doing this and the choice between these affects the properties of the estimation scheme. For example, negative estimates of the variance can be produced by some choices. Formulation as a least squares regression problem in which an ordinary least squares prediction problem is constructed, basing prediction of values of Xt on the p previous values of the same series. This can be thought of as a forward-prediction scheme. The normal equations for this problem can be seen to correspond to an approximation of the matrix form of the Yule–Walker equations in which each appearance of an autocovariance of the same lag is replaced by a slightly different estimate. Formulation as an extended form of ordinary least squares prediction problem. Here two sets of prediction equations are combined into a single estimation scheme and a single set of normal equations. One set is the set of forward-prediction equations and the other is a corresponding set of backward prediction equations, relating to the backward representation of the AR model: