Reference

Biwhitening

Biwhitening.biwhiten — Function
bw = biwhiten(Y, model::QuadraticVariance=QuadraticVariance(); kwargs...)
bw = biwhiten(Y, V::AbstractMatrix; kwargs...)

Biwhiten the data matrix Y: find row scales a and column scales b such that the noise of the scaled matrix Y ./ (a .* b') has average variance one in every row and every column. Then compute all singular values of the scaled matrix, and the singular vectors of those that exceed noise_edge. The result is a Biwhitened.

The noise is described either by a model that gives the variance of an entry as a function of its expected value, or by a matrix V of variance estimates of the same size as Y. The default model is the Poisson distribution, for which V = Y; this is the method of Landa, Zhang, and Kluger (2022). Each estimate in V must be nonnegative, should be unbiased, and must be computed from Y itself or from the measurements from which Y was derived. fit_variance chooses a model from the data.

The scales are computed by MatrixCovers.noise_cover, to which all keywords are passed. Rows and columns of V that are entirely zero receive a zero scale and are left out of the spectrum. For some patterns of zeros in V no scaling exists, and noise_cover throws an error that names the rows or columns responsible. Removing those rows or columns resolves the error, as does passing the keyword shrink=δ with δ > 0. (This keyword is unrelated to the function shrink.)

rank(bw) assumes that the variance estimates have the right overall scale. Check that noise_variance(bw) is close to one and that ks_distance(bw) is small before relying on it. When the two differ, the display of bw also gives the number of singular values above sqrt(noise_variance(bw)) * noise_edge(bw), the count for noise of the estimated variance.

Examples

julia> Y = [3 0 1 2; 0 2 5 1; 4 1 0 3];

julia> bw = biwhiten(Y);

julia> Vhat = Y ./ (bw.a .* bw.b') .^ 2;    # variance estimates for the scaled matrix

julia> round.(sum(Vhat; dims=2); digits=8)
3×1 Matrix{Float64}:
 4.0
 4.0
 4.0

julia> length(bw.S)
3
source
Biwhitening.Biwhitened — Type
Biwhitened{T}

The result of biwhiten for an m × n data matrix Y. Its fields are

  • a, b: the row and column scales. The scaled matrix, scaled(bw, Y), is Y ./ (a .* b'), with the rows and columns whose scale is zero set to zero.
  • S: all singular values of the scaled matrix after removing those rows and columns, in decreasing order.
  • U, V: the left and right singular vectors of the k = rank(bw) largest singular values, as the columns of an m × k and an n × k matrix.

rank(bw) is the number of singular values above noise_edge. noise_variance and ks_distance compare the spectrum with MarchenkoPastur(bw). shrink, scores, and denoise estimate the signal.

source
Biwhitening.scaled — Function
Yh = scaled(bw::Biwhitened, Y)

The scaled matrix Y ./ (bw.a .* bw.b') of the data Y given to biwhiten, with the rows and columns whose scale is zero set to zero. This is the matrix that biwhiten analyzes, and it can be passed to other algorithms, such as other factorization or clustering methods. A sparse Y gives a sparse result.

Examples

julia> Y = [3 0 1 2; 0 2 5 1; 4 1 0 3; 0 0 0 0];

julia> bw = biwhiten(Y);

julia> Yh = scaled(bw, Y);

julia> iszero(Yh[4, :])                     # the empty row has scale zero
true

julia> all(isnan, (Y ./ (bw.a .* bw.b'))[4, :])
true
source
Biwhitening.noise_edge — Function
noise_edge(bw::Biwhitened)

The largest singular value of the scaled matrix expected of noise alone, √m′ + √n′, where m′ and n′ are the numbers of rows and columns of the data with a nonzero scale. rank(bw) is the number of singular values in bw.S that exceed it.

source
Biwhitening.noise_variance — Function
noise_variance(bw::Biwhitened)

The noise variance of the scaled matrix, estimated as the ratio of the median of its eigenvalues to the median of the Marchenko–Pastur law. The median, unlike the mean, is barely affected by the few eigenvalues that come from the signal. The estimate is close to one when the variance estimates given to biwhiten have the right overall scale.

source
Biwhitening.ks_distance — Function
ks_distance(bw::Biwhitened)

The Kolmogorov–Smirnov distance between the eigenvalues of the scaled matrix and the Marchenko–Pastur law: the largest difference between the fraction of eigenvalues below a point and the probability the law assigns below it. It is small when the variance estimates given to biwhiten describe the noise and the rank of the signal is small compared with the size of the matrix.

source
Biwhitening.spectrumplot — Function
fig = spectrumplot(bw::Biwhitened; figure=(;))

Plot the singular values of the scaled matrix in two panels and return the Makie Figure. This function needs a Makie backend, such as CairoMakie or GLMakie, to be loaded.

The left panel shows every positive singular value against its index, on logarithmic axes, with noise_edge as a dashed line and the rank(bw) values above it marked. The right panel shows a histogram of the singular values up to a little beyond the edge, together with the density that the Marchenko–Pastur law predicts for noise of unit variance.

The plot shows what the numbers rank(bw), noise_variance, and ks_distance summarize. A histogram that follows the curve means that the variance model describes the noise. In the left panel, singular values that descend into the bulk without a gap mean that the count of components depends on exactly where the edge lies, and a change of slope among them may separate components of different origin.

figure holds keyword arguments for the Figure, such as (; size=(1000, 450)).

source

Variance models

Biwhitening.QuadraticVariance — Type
QuadraticVariance(a, b, c)
QuadraticVariance(; a=0, b=1, c=0)

A noise model in which the variance of each entry y of the data is a quadratic function of its expected value x:

Var(y) = a + b*x + c*x^2

The default, QuadraticVariance(), is the Poisson distribution. Other choices of the coefficients describe the negative binomial distribution with dispersion θ (c = 1/θ), the binomial with n trials (c = -1/n), the gamma distribution with shape k (b = 0, c = 1/k), and noise of constant variance a (b = c = 0). c = -1 is excluded, because the variance cannot then be estimated from a single observation.

variance_estimate estimates the variance of an entry from its value, and fit_variance chooses b and c from the data.

source
Biwhitening.variance_estimate — Function
variance_estimate(model::QuadraticVariance, y)

An unbiased estimate of the variance of an observation whose value is y:

(a + b*y + c*y^2) / (1 + c)

Its expected value is a + b*x + c*x^2, where x is the expected value of y.

Examples

julia> variance_estimate(QuadraticVariance(), 3)
3.0

julia> variance_estimate(QuadraticVariance(c=0.5), 3)
5.0
source
Biwhitening.fit_variance — Function
fit = fit_variance(Y; grid=0:0.2:1, refine=2, kwargs...)

Choose a QuadraticVariance for the nonnegative data matrix Y by the method of Landa, Zhang, and Kluger (2022, section 5.1), and return a VarianceFit; fit.model is the chosen model.

For each candidate β, Y is biwhitened with the variance estimate (1 - β) * y + β * y^2, which interpolates between a variance proportional to the expected value (β = 0) and one proportional to its square (β = 1). An overall scale α is then set so that the median eigenvalue of the scaled matrix equals the median of the Marchenko–Pastur law. The fit chooses the β whose eigenvalues have the smallest Kolmogorov–Smirnov distance to the law, and the middle one of several that share that distance. The chosen estimate, α * ((1 - β) * y + β * y^2), is the variance_estimate of the model with

a = 0,    b = α * (1 - β) / (1 - α * β),    c = α * β / (1 - α * β)

No such model exists when α * β ≥ 1, and the fit then throws an error.

The candidates are the elements of grid, followed by refine rounds of refinement. Each round adds the midpoints between the best candidate so far and its neighbors on either side, which halves the spacing around the minimum. The defaults examine at most ten candidates and locate β to within 0.05. The refinement finds the best β only if the distance has a single minimum at the spacing of grid; with refine=0 the candidates are exactly those of grid. fit.scan holds the results for every candidate in order of β; a distance that changes little across them means that the spectrum does not determine β.

The remaining keywords are those of biwhiten. The fit costs one biwhitening for each candidate.

Because the model is selected for its small distance, fit.ks is biased downward. To assess the fit without this bias, fit the model on some of the rows of Y, biwhiten the remaining rows with it, and compute their ks_distance.

source
Biwhitening.VarianceFit — Type
VarianceFit{T}

The result of fit_variance. Its fields are

  • model: the fitted QuadraticVariance;
  • β, α: the weight of the quadratic term and the overall scale of the chosen variance estimate α * ((1 - β) * y + β * y^2);
  • ks: the Kolmogorov–Smirnov distance between the eigenvalues of the scaled matrix and the Marchenko–Pastur law for that estimate;
  • scan: β, α, ks, and the number of singular values above the noise (rank) for every candidate β, in order of β.
source

The Marchenko–Pastur law

Biwhitening.MarchenkoPastur — Type
MarchenkoPastur(γ)

The Marchenko–Pastur distribution with aspect ratio 0 < γ ≤ 1: the limiting distribution of the eigenvalues of E * E' / q for a p × q matrix E of independent entries with mean zero and variance one, as p and q grow with p / q → γ.

For d = MarchenkoPastur(γ), the support is extrema(d) == ((1 - √γ)^2, (1 + √γ)^2). Biwhitening.pdf, Biwhitening.cdf, quantile, and median evaluate it.

Examples

julia> d = MarchenkoPastur(0.25);

julia> extrema(d)
(0.25, 2.25)

julia> Biwhitening.cdf(d, 2.25)
1.0
source
Biwhitening.MarchenkoPastur — Method
MarchenkoPastur(bw::Biwhitened)

The distribution of the eigenvalues bw.S .^ 2 ./ q when the data consist only of noise with the modeled variance. q is the larger of the numbers of rows and columns with a nonzero scale.

source
Biwhitening.cdf — Function
Biwhitening.cdf(d::MarchenkoPastur, x)

The probability that a variable with distribution d is at most x.

source
Statistics.quantile — Method
quantile(d::MarchenkoPastur, p)

The point x with Biwhitening.cdf(d, x) == p, for 0 ≤ p ≤ 1.

source

Shrinkage

Biwhitening.Shrinker — Type
Shrinker

A rule for estimating the singular values of the signal from those singular values of the scaled matrix that exceed noise_edge. The rules are HardThreshold, FrobeniusShrinker, OperatorShrinker, NuclearShrinker, and GramShrinker; shrink applies one.

Noise inflates the singular values of the signal and tilts its singular vectors. Let q be the larger dimension of the scaled matrix. In the limit of a large matrix, a component of the signal with singular value x√q appears in the scaled matrix with a larger singular value y√q. The angles between its observed singular vectors and those of the signal have cosines c₁ for the left singular vector (in bw.U) and c₂ for the right (in bw.V). x, c₁, and c₂ are functions of y and of the aspect ratio of the matrix, and each rule is a function of them (Gavish and Donoho, 2017).

source
Biwhitening.OperatorShrinker — Type
OperatorShrinker()

The Shrinker x, which estimates the singular value of the signal itself. The resulting estimate of the signal has the smallest error in the operator norm.

source
Biwhitening.NuclearShrinker — Type
NuclearShrinker()

The Shrinker for which the estimate of the signal has the smallest error in the nuclear norm, x * (c₁ * c₂ - sqrt((1 - c₁^2) * (1 - c₂^2))) where that is positive and zero elsewhere.

source
Biwhitening.GramShrinker — Type
GramShrinker(dim)

The Shrinker x * c₁ (dim = 1) or x * c₂ (dim = 2). With Z the scores of the rows (dim = 1) or columns (dim = 2) under this rule, Z * Z' has the smallest expected squared error as an estimate of the matrix of inner products between the rows, or columns, of the signal. It applies to scores and not to denoise.

source
Biwhitening.shrink — Function
η = shrink(bw::Biwhitened, shrinker::Shrinker=FrobeniusShrinker())

Estimates of the singular values of the signal in the scaled matrix, one for each of the rank(bw) singular values above the noise, under the rule shrinker. They assume that the noise of the scaled matrix has variance one (see noise_variance).

Examples

julia> Y = [40 1 0 2; 38 0 1 1; 1 0 2 1; 0 1 1 3];

julia> bw = biwhiten(Y);

julia> rank(bw)
1

julia> shrink(bw, HardThreshold()) == bw.S[1:1]
true

julia> shrink(bw)[1] < shrink(bw, OperatorShrinker())[1] < bw.S[1]
true
source
Biwhitening.scores — Function
Z = scores(bw::Biwhitened, dim, shrinker::Shrinker=HardThreshold())

Coordinates of the rows (dim = 1) or columns (dim = 2) of the data on the rank(bw) components above the noise: the singular vectors bw.U or bw.V, each multiplied by the corresponding singular value from shrink(bw, shrinker). Z has one row for each row (or column) of the data and one column for each component. With the default shrinker, these are the principal component scores of the scaled matrix, in which the noise has unit variance in each coordinate. Rows and columns whose scale is zero have zero scores.

source
Biwhitening.score_noise — Function
v = score_noise(bw::Biwhitened, dim, shrinker::Shrinker=HardThreshold())

The noise variance of each column of scores(bw, dim, shrinker), averaged over the rows (dim = 1) or columns (dim = 2) of the data with a nonzero scale. v has one element for each of the rank(bw) components above the noise, and sum(v) is the expected squared length of the noise in one row of the scores.

An observed singular vector is the sum of the signal's singular vector, multiplied by the cosine c of the angle between them, and a vector of noise of squared length 1 - c^2 (see Shrinker). A column of the scores is that vector times the singular value η from shrink, so its noise variance is η^2 * (1 - c^2) / n, where n is the number of rows (or columns) with a nonzero scale. With the default shrinker, the result is close to one for a component far above noise_edge and larger for one near it. Like shrink, it assumes that the noise of the scaled matrix has variance one (see noise_variance).

Examples

julia> Y = [40 1 0 2; 38 0 1 1; 1 0 2 1; 0 1 1 3];

julia> bw = biwhiten(Y);

julia> v = score_noise(bw, 1);

julia> length(v) == rank(bw)
true

julia> w = shrink(bw) ./ shrink(bw, HardThreshold());    # weights of the Frobenius rule

julia> score_noise(bw, 1, FrobeniusShrinker()) ≈ w .^ 2 .* v
true
source
Biwhitening.denoise — Function
X = denoise(bw::Biwhitened, shrinker::Shrinker=FrobeniusShrinker())

An estimate of the expected value of the data, as a dense matrix of the same size as the data. It is assembled from the components of the scaled matrix above the noise, with singular values from shrink(bw, shrinker), and multiplied by the scales bw.a and bw.b to return it to the units of the data.

source

Module

Biwhitening.Biwhitening — Module

Separation of signal from noise in a data matrix whose noise variance differs from entry to entry. Biwhitening scales the rows and columns so that the noise has average variance one in each of them, compares the spectrum of the scaled matrix with the Marchenko–Pastur law, counts the components above the noise, and shrinks their singular values.

The entry points are biwhiten and fit_variance.

source