Changing the reference measure of a relative entropy

For measures γ ≪ π ≪ R, the Radon–Nikodym chain rule dγ/dR = dγ/dπ · dπ/dR turns into an additive rule for log-likelihood ratios, llr γ R = llr γ π + llr π R almost everywhere with respect to γ. Integrating it against γ relates the relative entropies of γ against the two references R and π, up to the term ∫ llr π R dγ. When this term equals its value ∫ llr π R dπ at γ = π, the relation is Csiszár's Pythagorean identity klDiv γ R = klDiv γ π + klDiv π R.

The typical use is to identify an I-projection: if π belongs to a family of measures and log (dπ/dR) integrates to the same value against every member γ ≪ π of the family, then the identity shows at once that π minimises klDiv · R over the family. In entropic optimal transport the family is the set of couplings of two fixed marginals and log (dπ/dR) is a sum φ(x) + ψ(y) of two potentials, whose integral depends on a coupling only through its marginals.

Main statements

• TauCeti.llr_withDensity_exp: the log-likelihood ratio of R.withDensity (exp ∘ f) against R is f.
• TauCeti.llr_add_llr: the chain rule llr γ π + llr π R = llr γ R, γ-almost everywhere, for γ absolutely continuous with respect to both π and R.
• TauCeti.klDiv_eq_klDiv_add_klDiv: the Pythagorean identity.

References

• I. Csiszár, I-divergence geometry of probability distributions and minimization problems, Ann. Probab. 3 (1975), 146–158.

theorem TauCeti.llr_withDensity_exp {α : Type u_1} [MeasurableSpace α] {R : MeasureTheory.Measure α} [MeasureTheory.SigmaFinite R] {f : α → ℝ} (hf : AEMeasurable f R) : MeasureTheory.llr (R.withDensity fun (x : α) => ENNReal.ofReal (Real.exp (f x))) R =ᵐ[R] f

The log-likelihood ratio of the measure with density exp ∘ f with respect to R is f, almost everywhere.

theorem TauCeti.llr_add_llr {α : Type u_1} [MeasurableSpace α] {γ π R : MeasureTheory.Measure α} [MeasureTheory.SigmaFinite γ] [MeasureTheory.SigmaFinite π] [MeasureTheory.SigmaFinite R] (hγπ : γ.AbsolutelyContinuous π) (hγR : γ.AbsolutelyContinuous R) : MeasureTheory.llr γ π + MeasureTheory.llr π R =ᵐ[γ] MeasureTheory.llr γ R

Chain rule for log-likelihood ratios. If γ ≪ π and γ ≪ R, then llr γ π + llr π R = llr γ R, γ-almost everywhere. No relation between π and R is needed.

theorem TauCeti.klDiv_eq_klDiv_add_klDiv {α : Type u_1} [MeasurableSpace α] {γ π R : MeasureTheory.Measure α} [MeasureTheory.IsFiniteMeasure γ] [MeasureTheory.IsFiniteMeasure π] [MeasureTheory.IsFiniteMeasure R] (hγπ : γ.AbsolutelyContinuous π) (hπR : π.AbsolutelyContinuous R) (hγ : MeasureTheory.Integrable (MeasureTheory.llr π R) γ) (hπ : MeasureTheory.Integrable (MeasureTheory.llr π R) π) (h : ∫ (x : α), MeasureTheory.llr π R x ∂γ = ∫ (x : α), MeasureTheory.llr π R x ∂π) : InformationTheory.klDiv γ R = InformationTheory.klDiv γ π + InformationTheory.klDiv π R

Csiszár's Pythagorean identity. Let γ ≪ π ≪ R be finite measures such that llr π R is integrable against both γ and π, with the same integral. Then klDiv γ R = klDiv γ π + klDiv π R.