Search Results for “gaussian process”

Source: Gaussian process

In probability theory and statistics, a Gaussian process is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.
The concept of Gaussian processes is named after Carl Friedrich Gauss because it is based on the notion of the Gaussian distribution (normal distribution). Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions.
Gaussian processes are useful in statistical modelling, benefiting from properties inherited from the normal distribution. For example, if a random process is modelled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times. While exact models often scale poorly as the amount of data increases, multiple approximation methods have been developed which often retain good accuracy while drastically reducing computation time.

Definition

process

Gaussian

process

Variance

Gaussian

process

Stationarity

Example

Gaussian

Covariance functions

= Usual covariance functions

Gaussian

process

Continuity

= Stationary case

Gaussian

process

In these two cases the function

σ

{\displaystyle \sigma }

is increasing on

[
0
,
∞
)
,

{\displaystyle [0,\infty ),}

but generally it is not. Moreover, the condition

does not follow from continuity of

σ

{\displaystyle \sigma }

and the evident relations

σ
(
h
)
≥
0

{\displaystyle \sigma (h)\geq 0}

(for all

h

{\displaystyle h}

) and

σ
(
0
)
=
0.

{\displaystyle \sigma (0)=0.}

Some history.: 424
Sufficiency was announced by Xavier Fernique in 1964, but the first proof was published by Richard M. Dudley in 1967.: Theorem 7.1
Necessity was proved by Michael B. Marcus and Lawrence Shepp in 1970.: 380
There exist sample continuous processes

X

{\displaystyle X}

such that

I
(
σ
)
=
∞
;

{\displaystyle I(\sigma )=\infty ;}

they violate condition (*). An example found by Marcus and Shepp : 387 is a random lacunary Fourier series

X

t

=

∑

n
=
1

∞

c

n

(

ξ

n

cos
⁡

λ

n

t
+

η

n

sin
⁡

λ

n

t
)
,

{\displaystyle X_{t}=\sum _{n=1}^{\infty }c_{n}(\xi _{n}\cos \lambda _{n}t+\eta _{n}\sin \lambda _{n}t),}

where

ξ

1

,

η

1

,

ξ

2

,

η

2

,
…

{\displaystyle \xi _{1},\eta _{1},\xi _{2},\eta _{2},\dots }

are independent random variables with standard normal distribution; frequencies

0
<

λ

1

<

λ

2

<
…

{\displaystyle 0<\lambda _{1}<\lambda _{2}<\dots }

are a fast growing sequence; and coefficients

c

n

>
0

{\displaystyle c_{n}>0}

satisfy

∑

n

c

n

<
∞
.

{\textstyle \sum _{n}c_{n}<\infty .}

The latter relation implies

E

∑

n

c

n

(

|

ξ

n

|

+

|

η

n

|

)
=

∑

n

c

n

E

[

|

ξ

n

|

+

|

η

n

|

]
=

const

⋅

∑

n

c

n

<
∞
,

{\textstyle {\mathbb {E} }\sum _{n}c_{n}(|\xi _{n}|+|\eta _{n}|)=\sum _{n}c_{n}{\mathbb {E} }[|\xi _{n}|+|\eta _{n}|]={\text{const}}\cdot \sum _{n}c_{n}<\infty ,}

whence

∑

n

c

n

(

|

ξ

n

|

+

|

η

n

|

)
<
∞

{\textstyle \sum _{n}c_{n}(|\xi _{n}|+|\eta _{n}|)<\infty }

almost surely, which ensures uniform convergence of the Fourier series almost surely, and sample continuity of

X
.

{\displaystyle X.}

Its autocovariation function

E

[

X

t

X

t
+
h

]
=

∑

n
=
1

∞

c

n

2

cos
⁡

λ

n

h

{\displaystyle {\mathbb {E} }[X_{t}X_{t+h}]=\sum _{n=1}^{\infty }c_{n}^{2}\cos \lambda _{n}h}

is nowhere monotone (see the picture), as well as the corresponding function

σ
,

{\displaystyle \sigma ,}

σ
(
h
)
=

2

E

[

X

t

X

t

]
−
2

E

[

X

t

X

t
+
h

]

=
2

∑

n
=
1

∞

c

n

2

sin

2

⁡

λ

n

h

2

.

{\displaystyle \sigma (h)={\sqrt {2{\mathbb {E} }[X_{t}X_{t}]-2{\mathbb {E} }[X_{t}X_{t+h}]}}=2{\sqrt {\sum _{n=1}^{\infty }c_{n}^{2}\sin ^{2}{\frac {\lambda _{n}h}{2}}}}.}

Brownian motion as the integral of Gaussian processes

A Wiener process (also known as Brownian motion) is the integral of a white noise generalized Gaussian process. It is not stationary, but it has stationary increments.
The Ornstein–Uhlenbeck process is a stationary Gaussian process.
The Brownian bridge is (like the Ornstein–Uhlenbeck process) an example of a Gaussian process whose increments are not independent.
The fractional Brownian motion is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.

RKHS structure and Gaussian process

Let

f

{\displaystyle f}

be a mean-zero Gaussian process

{

X

t

;
t
∈
T

}

{\displaystyle \left\{X_{t};t\in T\right\}}

with a non-negative definite covariance function

K

{\displaystyle K}

and let

R

{\displaystyle R}

be a symmetric and positive semidefinite function. Then, there exists a Gaussian process

X

{\displaystyle X}

which has the covariance

R

{\displaystyle R}

. Moreover, the Reproducing kernel Hilbert space associated to

R

{\displaystyle R}

coincides with the Cameron–Martin theorem associated space

R
(
H
)

{\displaystyle R(H)}

of

X

{\displaystyle X}

, and all the spaces

R
(
H
)

{\displaystyle R(H)}

,

H

X

{\displaystyle H_{X}}

, and

H

(
K
)

{\displaystyle {\mathcal {H}}(K)}

are isometric. From now on, let

H

(
R
)

{\displaystyle {\mathcal {H}}(R)}

be a Reproducing kernel Hilbert space with positive definite kernel

R

{\displaystyle R}

.
Driscoll's zero-one law is a result characterizing the sample functions generated by a Gaussian process:

lim

n
→
∞

tr
⁡
[

K

n

R

n

−
1

]
<
∞
,

{\displaystyle \lim _{n\to \infty }\operatorname {tr} [K_{n}R_{n}^{-1}]<\infty ,}

where

K

n

{\displaystyle K_{n}}

and

R

n

{\displaystyle R_{n}}

are the covariance matrices of all possible pairs of

n

{\displaystyle n}

points, implies

Pr
[
f
∈

H

(
R
)
]
=
1.

{\displaystyle \Pr[f\in {\mathcal {H}}(R)]=1.}

Moreover,

lim

n
→
∞

tr
⁡
[

K

n

R

n

−
1

]
=
∞

{\displaystyle \lim _{n\to \infty }\operatorname {tr} [K_{n}R_{n}^{-1}]=\infty }

implies

Pr
[
f
∈

H

(
R
)
]
=
0.

{\displaystyle \Pr[f\in {\mathcal {H}}(R)]=0.}

This has significant implications when

K
=
R

{\displaystyle K=R}

, as

lim

n
→
∞

tr
⁡
[

R

n

R

n

−
1

]
=

lim

n
→
∞

tr
⁡
[
I
]
=

lim

n
→
∞

n
=
∞
.

{\displaystyle \lim _{n\to \infty }\operatorname {tr} [R_{n}R_{n}^{-1}]=\lim _{n\to \infty }\operatorname {tr} [I]=\lim _{n\to \infty }n=\infty .}

As such, almost all sample paths of a mean-zero Gaussian process with positive definite kernel

K

{\displaystyle K}

will lie outside of the Hilbert space

H

(
K
)

{\displaystyle {\mathcal {H}}(K)}

.

Linearly constrained Gaussian processes

For many applications of interest some pre-existing knowledge about the system at hand is already given. Consider e.g. the case where the output of the Gaussian process corresponds to a magnetic field; here, the real magnetic field is bound by Maxwell's equations and a way to incorporate this constraint into the Gaussian process formalism would be desirable as this would likely improve the accuracy of the algorithm.
A method on how to incorporate linear constraints into Gaussian processes already exists:
Consider the (vector valued) output function

f
(
x
)

{\displaystyle f(x)}

which is known to obey the linear constraint (i.e.

F

X

{\displaystyle {\mathcal {F}}_{X}}

is a linear operator)

F

X

(
f
(
x
)
)
=
0.

{\displaystyle {\mathcal {F}}_{X}(f(x))=0.}

Then the constraint

F

X

{\displaystyle {\mathcal {F}}_{X}}

can be fulfilled by choosing

f
(
x
)
=

G

X

(
g
(
x
)
)

{\displaystyle f(x)={\mathcal {G}}_{X}(g(x))}

, where

g
(
x
)
∼

G
P

(

μ

g

,

K

g

)

{\displaystyle g(x)\sim {\mathcal {GP}}(\mu _{g},K_{g})}

is modelled as a Gaussian process, and finding

G

X

{\displaystyle {\mathcal {G}}_{X}}

such that

F

X

(

G

X

(
g
)
)
=
0

∀
g
.

{\displaystyle {\mathcal {F}}_{X}({\mathcal {G}}_{X}(g))=0\qquad \forall g.}

Given

G

X

{\displaystyle {\mathcal {G}}_{X}}

and using the fact that Gaussian processes are closed under linear transformations, the Gaussian process for

f

{\displaystyle f}

obeying constraint

F

X

{\displaystyle {\mathcal {F}}_{X}}

becomes

f
(
x
)
=

G

X

g
∼

G
P

(

G

X

μ

g

,

G

X

K

g

G

X
′

T

)
.

{\displaystyle f(x)={\mathcal {G}}_{X}g\sim {\mathcal {GP}}({\mathcal {G}}_{X}\mu _{g},{\mathcal {G}}_{X}K_{g}{\mathcal {G}}_{X'}^{\mathsf {T}}).}

Hence, linear constraints can be encoded into the mean and covariance function of a Gaussian process.

Applications

A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference. Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian. For solution of the multi-output prediction problem, Gaussian process regression for vector-valued function was developed. In this method, a 'big' covariance is constructed, which describes the correlations between all the input and output variables taken in N points in the desired domain. This approach was elaborated in detail for the matrix-valued Gaussian processes and generalised to processes with 'heavier tails' like Student-t processes.
Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging. Gaussian processes are thus useful as a powerful non-linear multivariate interpolation tool. Kriging is also used to extend Gaussian process in the case of mixed integer inputs.
Gaussian processes are also commonly used to tackle numerical analysis problems such as numerical integration, solving differential equations, or optimisation in the field of probabilistic numerics.
Gaussian processes can also be used in the context of mixture of experts models, for example. The underlying rationale of such a learning framework consists in the assumption that a given mapping cannot be well captured by a single Gaussian process model. Instead, the observation space is divided into subsets, each of which is characterized by a different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.
In the natural sciences, Gaussian processes have found use as probabilistic models of astronomical time series and as predictors of molecular properties. They are also being increasingly used as surrogate models for force field optimization .

= Gaussian process prediction, or Kriging

=

When concerned with a general Gaussian process regression problem (Kriging), it is assumed that for a Gaussian process

f

{\displaystyle f}

observed at coordinates

x

{\displaystyle x}

, the vector of values ⁠

f
(
x
)

{\displaystyle f(x)}

⁠ is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates ⁠

n

{\displaystyle n}

⁠. Therefore, under the assumption of a zero-mean distribution, ⁠

f
(

x
′

)
∼
N
(
0
,
K
(
θ
,
x
,

x
′

)
)

{\displaystyle f(x')\sim N(0,K(\theta ,x,x'))}

⁠, where ⁠

K
(
θ
,
x
,

x
′

)

{\displaystyle K(\theta ,x,x')}

⁠ is the covariance matrix between all possible pairs ⁠

(
x
,

x
′

)

{\displaystyle (x,x')}

⁠ for a given set of hyperparameters θ.
As such the log marginal likelihood is:

log
⁡
p
(
f
(

x
′

)
∣
θ
,
x
)
=
−

1
2

(

f
(
x

)

T

K
(
θ
,
x
,

x
′

)

−
1

f
(

x
′

)
+
log
⁡
det
(
K
(
θ
,
x
,

x
′

)
)
+
n
log
⁡
2
π

)

{\displaystyle \log p(f(x')\mid \theta ,x)=-{\frac {1}{2}}\left(f(x)^{\mathsf {T}}K(\theta ,x,x')^{-1}f(x')+\log \det(K(\theta ,x,x'))+n\log 2\pi \right)}

and maximizing this marginal likelihood towards θ provides the complete specification of the Gaussian process f. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified θ, making predictions about unobserved values ⁠

f
(

x

∗

)

{\displaystyle f(x^{*})}

⁠ at coordinates x* is then only a matter of drawing samples from the predictive distribution

p
(

y

∗

∣

x

∗

,
f
(
x
)
,
x
)
=
N
(

y

∗

∣
A
,
B
)

{\displaystyle p(y^{*}\mid x^{*},f(x),x)=N(y^{*}\mid A,B)}

where the posterior mean estimate A is defined as

A
=
K
(
θ
,

x

∗

,
x
)
K
(
θ
,
x
,

x
′

)

−
1

f
(
x
)

{\displaystyle A=K(\theta ,x^{*},x)K(\theta ,x,x')^{-1}f(x)}

and the posterior variance estimate B is defined as:

B
=
K
(
θ
,

x

∗

,

x

∗

)
−
K
(
θ
,

x

∗

,
x
)
K
(
θ
,
x
,

x
′

)

−
1

K
(
θ
,

x

∗

,
x

)

T

{\displaystyle B=K(\theta ,x^{*},x^{*})-K(\theta ,x^{*},x)K(\theta ,x,x')^{-1}K(\theta ,x^{*},x)^{\mathsf {T}}}

where ⁠

K
(
θ
,

x

∗

,
x
)

{\displaystyle K(\theta ,x^{*},x)}

⁠ is the covariance between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector θ, ⁠

K
(
θ
,
x
,

x
′

)

{\displaystyle K(\theta ,x,x')}

⁠ and ⁠

f
(
x
)

{\displaystyle f(x)}

⁠ are defined as before and ⁠

K
(
θ
,

x

∗

,

x

∗

)

{\displaystyle K(\theta ,x^{*},x^{*})}

⁠ is the variance at point x* as dictated by θ. It is important to note that practically the posterior mean estimate of ⁠

f
(

x

∗

)

{\displaystyle f(x^{*})}

⁠ (the "point estimate") is just a linear combination of the observations ⁠

f
(
x
)

{\displaystyle f(x)}

⁠; in a similar manner the variance of ⁠

f
(

x

∗

)

{\displaystyle f(x^{*})}

⁠ is actually independent of the observations ⁠

f
(
x
)

{\displaystyle f(x)}

⁠. A known bottleneck in Gaussian process prediction is that the computational complexity of inference and likelihood evaluation is cubic in the number of points |x|, and as such can become unfeasible for larger data sets. Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue. The kriging method can be used in the latent level of a nonlinear mixed-effects model for a spatial functional prediction: this technique is called the latent kriging.
Often, the covariance has the form

K
(
θ
,
x
,

x
′

)
=

1

σ

2

K
~

(
θ
,
x
,

x
′

)

{\textstyle K(\theta ,x,x')={\frac {1}{\sigma ^{2}}}{\tilde {K}}(\theta ,x,x')}

, where

σ

2

{\displaystyle \sigma ^{2}}

is a scaling parameter. Examples are the Matérn class covariance functions. If this scaling parameter

σ

2

{\displaystyle \sigma ^{2}}

is either known or unknown (i.e. must be marginalized), then the posterior probability,

p
(
θ
∣
D
)

{\displaystyle p(\theta \mid D)}

, i.e. the probability for the hyperparameters

θ

{\displaystyle \theta }

given a set of data pairs

D

{\displaystyle D}

of observations of

x

{\displaystyle x}

and

f
(
x
)

{\displaystyle f(x)}

, admits an analytical expression.

= Bayesian neural networks as Gaussian processes

=

Bayesian neural networks are a particular type of Bayesian network that results from treating deep learning and artificial neural network models probabilistically, and assigning a prior distribution to their parameters. Computation in artificial neural networks is usually organized into sequential layers of artificial neurons. The number of neurons in a layer is called the layer width. As layer width grows large, many Bayesian neural networks reduce to a Gaussian process with a closed form compositional kernel. This Gaussian process is called the Neural Network Gaussian Process (NNGP). It allows predictions from Bayesian neural networks to be more efficiently evaluated, and provides an analytic tool to understand deep learning models.

Computational issues

In practical applications, Gaussian process models are often evaluated on a grid leading to multivariate normal distributions. Using these models for prediction or parameter estimation using maximum likelihood requires evaluating a multivariate Gaussian density, which involves calculating the determinant and the inverse of the covariance matrix. Both of these operations have cubic computational complexity which means that even for grids of modest sizes, both operations can have a prohibitive computational cost. This drawback led to the development of multiple approximation methods.

References

External links

= Literature

=
The Gaussian Processes Web Site, including the text of Rasmussen and Williams' Gaussian Processes for Machine Learning
Ebden, Mark (2015). "Gaussian Processes: A Quick Introduction". arXiv:1505.02965 [math.ST].
A Review of Gaussian Random Fields and Correlation Functions
Efficient Reinforcement Learning using Gaussian Processes

= Software

=

GPML: A comprehensive Matlab toolbox for GP regression and classification
STK: a Small (Matlab/Octave) Toolbox for Kriging and GP modeling
Kriging module in UQLab framework (Matlab)
CODES Toolbox: implementations of Kriging, variational kriging and multi-fidelity models (Matlab)
Matlab/Octave function for stationary Gaussian fields
Yelp MOE – A black box optimization engine using Gaussian process learning
ooDACE Archived 2020-08-09 at the Wayback Machine – A flexible object-oriented Kriging Matlab toolbox.
GPstuff – Gaussian process toolbox for Matlab and Octave
GPy – A Gaussian processes framework in Python
GSTools - A geostatistical toolbox, including Gaussian process regression, written in Python
Interactive Gaussian process regression demo
Basic Gaussian process library written in C++11
scikit-learn – A machine learning library for Python which includes Gaussian process regression and classification
[1] - The Kriging toolKit (KriKit) is developed at the Institute of Bio- and Geosciences 1 (IBG-1) of Forschungszentrum Jülich (FZJ)

= Video tutorials

=
Gaussian Process Basics by David MacKay
Learning with Gaussian Processes by Carl Edward Rasmussen
Bayesian inference and Gaussian processes by Carl Edward Rasmussen

Definition

Variance

Stationarity

Example

Covariance functions

= Usual covariance functions

Continuity

= Stationary case

Brownian motion as the integral of Gaussian processes

RKHS structure and Gaussian process

Linearly constrained Gaussian processes

Applications

= Gaussian process prediction, or Kriging

= Bayesian neural networks as Gaussian processes

Computational issues

See also

References

External links

= Literature

= Software

= Video tutorials

Kata Kunci Pencarian:

Recent Movies

Recent Movies

Categories

Recent Movies