==== Front Entropy (Basel) Entropy (Basel) entropy Entropy 1099-4300 MDPI 33287092 10.3390/e22111327 entropy-22-01327 Article Strongly Convex Divergences Melbourne James Department of Electrical and Computer Engineering, University of Minnesota-Twin Cities, Minneapolis, MN 55455, USA; melbo013@umn.edu 21 11 2020 11 2020 22 11 132702 9 2020 09 11 2020 © 2020 by the author.2020Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).We consider a sub-class of the f-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or κ-convex divergences. We derive new and old relationships, based on convexity arguments, between popular f-divergences. information measuresf-divergencehypothesis testingtotal variationskew-divergenceconvexityPinsker’s inequalityBayes riskJensen–Shannon divergence ==== Body 1. Introduction The concept of an f-divergence, introduced independently by Ali-Silvey [1], Morimoto [2], and Csisizár [3], unifies several important information measures between probability distributions, as integrals of a convex function f, composed with the Radon–Nikodym of the two probability distributions. (An additional assumption can be made that f is strictly convex at 1, to ensure that Df(μ||ν)>0 for μ≠ν. This obviously holds for any f″(1)>0, and can hold for some f-divergences without classical derivatives at 0, for instance the total variation is strictly convex at 1. An example of an f-divergence not strictly convex is provided by the so-called “hockey-stick” divergence, where f(x)=(x−γ)+, see [4,5,6].) For a convex function f:(0,∞)→R such that f(1)=0, and measures P and Q such that P≪Q, the f-divergence from P to Q is given by Df(P||Q):=∫fdPdQdQ. The canonical example of an f-divergence, realized by taking f(x)=xlogx, is the relative entropy (often called the KL-divergence), which we denote with the subscript f omitted. f-divergences inherit many properties enjoyed by this special case; non-negativity, joint convexity of arguments, and a data processing inequality. Other important examples include the total variation, the χ2-divergence, and the squared Hellinger distance. The reader is directed to Chapter 6 and 7 of [7] for more background. We are interested in how stronger convexity properties of f give improvements of classical f-divergence inequalities. More explicitly, we consider consequences of f being κ-convex, in the sense that the map x↦f(x)−κx2/2 is convex. This is in part inspired by the work of Sason [8], who demonstrated that divergences that are κ-convex satisfy “stronger than χ2” data-processing inequalities. Perhaps the most well known example of an f-divergence inequality is Pinsker’s inequality, which bounds the square of the total variation above by a constant multiple of the relative entropy. That is for probability measures P and Q, |P−Q|TV2≤cD(P||Q). The optimal constant is achieved for Bernoulli measures, and under our conventions for total variation, c=1/2loge. Many extensions and sharpenings of Pinsker’s inequality exist (for examples, see [9,10,11]). Building on the work of Guntuboyina [9] and Topsøe [11], we achieve a further sharpening of Pinsker’s inequality in Theorem 9. Aside from the total variation, most divergences of interest have stronger than affine convexity, at least when f is restricted to a sub-interval of the real line. This observation is especially relevant to the situation in which one wishes to study Df(P||Q) in the existence of a bounded Radon–Nikodym derivative dPdQ∈(a,b)⊊(0,∞). One naturally obtains such bounds for skew divergences. That is divergences of the form (P,Q)↦Df((1−t)P+tQ||(1−s)P+sQ) for t,s∈[0,1], as in this case, (1−t)P+tQ(1−s)P+sQ≤max1−t1−s,ts. Important examples of skew-divergences include the skew divergence [12] based on the relative entropy and the Vincze–Le Cam divergence [13,14], called the triangular discrimination in [11] and its generalization due to Györfi and Vajda [15] based on the χ2-divergence. The Jensen–Shannon divergence [16] and its recent generalization [17] give examples of f-divergences realized as linear combinations of skewed divergences. Let us outline the paper. In Section 2, we derive elementary results of κ-convex divergences and give a table of examples of κ-convex divergences. We demonstrate that κ-convex divergences can be lower bounded by the χ2-divergence, and that the joint convexity of the map (P,Q)↦Df(P||Q) can be sharpened under κ-convexity conditions on f. As a consequence, we obtain bounds between the mean square total variation distance of a set of distributions from its barycenter, and the average f-divergence from the set to the barycenter. In Section 3, we investigate general skewing of f-divergences. In particular, we introduce the skew-symmetrization of an f-divergence, which recovers the Jensen–Shannon divergence and the Vincze–Le Cam divergences as special cases. We also show that a scaling of the Vincze–Le Cam divergence is minimal among skew-symmetrizations of κ-convex divergences on (0,2). We then consider linear combinations of skew divergences and show that a generalized Vincze–Le Cam divergence (based on skewing the χ2-divergence) can be upper bounded by the generalized Jensen–Shannon divergence introduced recently by Nielsen [17] (based on skewing the relative entropy), reversing the classical convexity bounds D(P||Q)≤log(1+χ2(P||Q))≤logeχ2(P||Q). We also derive upper and lower total variation bounds for Nielsen’s generalized Jensen–Shannon divergence. In Section 4, we consider a family of densities {pi} weighted by λi, and a density q. We use the Bayes estimator T(x)=argmaxiλipi(x) to derive a convex decomposition of the barycenter p=∑iλipi and of q, each into two auxiliary densities. (Recall, a Bayes estimator is one that minimizes the expected value of a loss function. By the assumptions of our model, that P(θ=i)=λi, and P(X∈A|θ=i)=∫Api(x)dx, we have Eℓ(θ,θ^)=1−∫λθ^(x)pθ^(x)(x)dx for the loss function ℓ(i,j)=1−δi(j) and any estimator θ^. It follows that Eℓ(θ,θ^)≥Eℓ(θ,T) by λθ^(x)pθ^(x)(x)≤λT(x)pT(x)(x). Thus, T is a Bayes estimator associated to ℓ. ) We use this decomposition to sharpen, for κ-convex divergences, an elegant theorem of Guntuboyina [9] that generalizes Fano and Pinsker’s inequality to f-divergences. We then demonstrate explicitly, using an argument of Topsøe, how our sharpening of Guntuboyina’s inequality gives a new sharpening of Pinsker’s inequality in terms of the convex decomposition induced by the Bayes estimator. Notation Throughout, f denotes a convex function f:(0,∞)→R∪{∞}, such that f(1)=0. For a convex function defined on (0,∞), we define f(0):=limx→0f(x). We denote by f∗, the convex function f∗:(0,∞)→R∪{∞} defined by f∗(x)=xf(x−1). We consider Borel probability measures P and Q on a Polish space X and define the f-divergence from P to Q, via densities p for P and q for Q with respect to a common reference measure μ as (1) Df(p||q)=∫Xfpqqdμ=∫{pq>0}qfpqdμ+f(0)Q({p=0})+f∗(0)P({q=0}). We note that this representation is independent of μ, and such a reference measure always exists, take μ=P+Q for example. For t,s∈[0,1], define the binary f-divergence (2) Df(t||s):=sfts+(1−s)f1−t1−s with the conventions, f(0)=limt→0+f(t), 0f(0/0)=0, and 0f(a/0)=alimt→∞f(t)/t. For a random variable X and a set A, we denote the probability that X takes a value in A by P(X∈A), the expectation of the random variable by EX, and the variance by Var(X):=E|X−EX|2. For a probability measure μ satisfying μ(A)=P(X∈A) for all Borel A, we write X∼μ, and, when there exists a probability density function such that P(X∈A)=∫Af(x)dγ(x) for a reference measure γ, we write X∼f. For a probability measure μ on X, and an L2 function f:X→R, we denote Varμ(f):=Var(f(X)) for X∼μ. 2. Strongly Convex Divergences Definition 1. A R∪{∞}-valued function f on a convex set K⊆R is κ-convex when x,y∈K and t∈[0,1] implies (3) f((1−t)x+ty)≤(1−t)f(x)+tf(y)−κt(1−t)(x−y)2/2. For example, when f is twice differentiable, (3) is equivalent to f″(x)≥κ for x∈K. Note that the case κ=0 is just usual convexity. Proposition 1. For f:K→R∪{∞} and κ∈[0,∞), the following are equivalent: f is κ-convex. The function f−κ(t−a)2/2 is convex for any a∈R. The right handed derivative, defined as f+′(t):=limh↓0f(t+h)−f(t)h satisfies, f+′(t)≥f+′(s)+κ(t−s) for t≥s. Proof.  Observe that it is enough to prove the result when κ=0, where the proposition is reduced to the classical result for convex functions. □ Definition 2. An f-divergence Df is κ-convex on an interval K for κ≥0 when the function f is κ-convex on K. Table 1 lists some κ-convex f-divergences of interest to this article. Observe that we have taken the normalization convention on the total variation (the total variation for a signed measure μ on a space X can be defined through the Hahn-Jordan decomposition of the measure into non-negative measures μ+ and μ− such that μ=μ+−μ−, as ∥μ∥=μ+(X)+μ−(X) (see [18]); in our notation, |μ|TV=∥μ∥/2) which we denote by |P−Q|TV, such that |P−Q|TV=supA|P(A)−Q(A)|≤1. In addition, note that the α-divergence interpolates Pearson’s χ2-divergence when α=3, one half Neyman’s χ2-divergence when α=−3, the squared Hellinger divergence when α=0, and has limiting cases, the relative entropy when α=1 and the reverse relative entropy when α=−1. If f is κ-convex on [a,b], then recalling its dual divergence f∗(x):=xf(x−1) is κa3-convex on [1b,1a]. Recall that f∗ satisfies the equality Df∗(P||Q)=Df(Q||P). For brevity, we use χ2-divergence to refer to the Pearson χ2-divergence, and we articulate Neyman’s χ2 explicitly when necessary. The next lemma is a restatement of Jensen’s inequality. Lemma 1. If f is κ-convex on the range of X, Ef(X)≥f(E(X))+κ2Var(X). Proof.  Apply Jensen’s inequality to f(x)−κx2/2. □ For a convex function f such that f(1)=0 and c∈R, the function f˜(t)=f(t)+c(t−1) remains a convex function, and what is more satisfies Df(P||Q)=Df˜(P||Q) since ∫c(p/q−1)qdμ=0. Definition 3 (χ2-divergence). For f(t)=(t−1)2, we write χ2(P||Q):=Df(P||Q). We pursue a generalization of the following bound on the total variation by the χ2-divergence [19,20,21]. Theorem 1 ([19,20,21]). For measures P and Q, (4) |P−Q|TV2≤χ2(P||Q)2. We mention the work of Harremos and Vadja [20], in which it is shown, through a characterization of the extreme points of the joint range associated to a pair of f-divergences (valid in general), that the inequality characterizes the “joint range”, that is, the range of the function (P,Q)↦(|P−Q|TV,χ2(P||Q)). We use the following lemma, which shows that every strongly convex divergence can be lower bounded, up to its convexity constant κ>0, by the χ2-divergence, Lemma 2. For a κ-convex f, Df(P||Q)≥κ2χ2(P||Q). Proof.  Define a f˜(t)=f(t)−f+′(1)(t−1) and note that f˜ defines the same κ-convex divergence as f. Thus, we may assume without loss of generality that f+′ is uniquely zero when t=1. Since f is κ-convex ϕ:t↦f(t)−κ(t−1)2/2 is convex, and, by f+′(1)=0, ϕ+′(1)=0 as well. Thus, ϕ takes its minimum when t=1 and hence ϕ≥0 so that f(t)≥κ(t−1)2/2. Computing, Df(P||Q)=∫fdPdQdQ≥κ2∫dPdQ−12dQ=κ2χ2(P||Q).  □ Based on a Taylor series expansion of f about 1, Nielsen and Nock ([22], [Corollary 1]) gave the estimate (5) Df(P||Q)≈f″(1)2χ2(P||Q) for divergences with a non-zero second derivative and P close to Q. Lemma 2 complements this estimate with a lower bound, when f is κ-concave. In particular, if f″(1)=κ, it shows that the approximation in (5) is an underestimate. Theorem 2. For measures P and Q, and a κ convex divergence Df, (6) |P−Q|TV2≤Df(P||Q)κ. Proof.  By Lemma 2 and then Theorem 1, (7) Df(P||Q)κ≥χ2(P||Q)2≥|P−Q|TV.  □ The proof of Lemma 2 uses a pointwise inequality between convex functions to derive an inequality between their respective divergences. This simple technique was shown to have useful implications by Sason and Verdu in [6], where it appears as Theorem 1 and is used to give sharp comparisons in several f-divergence inequalities. Theorem 3 (Sason–Verdu [6]). For divergences defined by g and f with cf(t)≥g(t) for all t, then Dg(P||Q)≤cDf(P||Q). Moreover, if f′(1)=g′(1)=0, then supP≠QDg(P||Q)Df(P||Q)=supt≠1g(t)f(t). Corollary 1. For a smooth κ-convex divergence f, the inequality (8) Df(P||Q)≥κ2χ2(P||Q) is sharp multiplicatively in the sense that (9) infP≠QDf(P||Q)χ2(P||Q)=κ2. if f″(1)=κ. In information geometry, a standard f-divergence is defined as an f-divergence satisfying the normalization f(1)=f′(1)=0,f″(1)=1 (see [23]). Thus, Corollary 1 shows that 12χ2 provides a sharp lower bound on every standard f-divergence that is 1-convex. In particular, the lower bound in Lemma 2 complimenting the estimate (5) is shown to be sharp. Proof.  Without loss of generality, we assume that f′(1)=0. If f″(1)=κ+2ε for some ε>0, then taking g(t)=(t−1)2 and applying Theorem 3 and Lemma 2 (10) supP≠QDg(P||Q)Df(P||Q)=supt≠1g(t)f(t)≤2κ. Observe that, after two applications of L’Hospital, limε→0g(1+ε)f(1+ε)=limε→0g′(1+ε)f′(1+ε)=g″(1)f″(1)=2κ≤supt≠1g(t)f(t). Thus, (9) follows. □ Proposition 2. When Df is an f divergence such that f is κ-convex on [a,b] and that Pθ and Qθ are probability measures indexed by a set Θ such that a≤dPθdQθ(x)≤b, holds for all θ and P:=∫ΘPθdμ(θ) and Q:=∫ΘQθdμ(θ) for a probability measure μ on Θ, then (11) Df(P||Q)≤∫ΘDf(Pθ||Qθ)dμ(θ)−κ2∫Θ∫XdPθdQθ−dPdQ2dQdμ, In particular, when Qθ=Q for all θ (12) Df(P||Q)≤∫ΘDf(Pθ||Q)dμ(θ)−κ2∫Θ∫XdPθdQ−dPdQ2dQdμ(θ)≤∫ΘDf(Pθ||Q)dμ(θ)−κ∫Θ|Pθ−P|TV2dμ(θ) Proof.  Let dθ denote a reference measure dominating μ so that dμ=φ(θ)dθ then write νθ=ν(θ,x)=dQθdQ(x)φ(θ). (13) Df(P||Q)=∫XfdPdQdQ=∫Xf∫ΘdPθdQdμ(θ)dQ=∫Xf∫ΘdPθdQθν(θ,x)dθdQ By Jensen’s inequality, as in Lemma 1 f∫ΘdPθdQθνθdθ≤∫θfdPθdQθνθdθ−κ2∫ΘdPθdQθ−∫ΘdPθdQθνθdθ2νθdθ Integrating this inequality gives (14) Df(P||Q)≤∫X∫θfdPθdQθνθdθ−κ2∫ΘdPθdQθ−∫ΘdPθdQθνθdθ2νθdθdQ Note that ∫X∫ΘdPθdQθdQ−∫ΘdPθdQθ0νθ0dθ02νθdθdQ=∫Θ∫XdPθdQθ−dPdQ2dQdμ, and (15) ∫X∫ΘfdPθdQθν(θ,x)dθdQ=∫Θ∫XfdPθdQθν(θ,x)dQdθ=∫Θ∫XfdPθdQθdQθdμ(θ)=∫ΘD(Pθ||Qθ)dμ(θ) Inserting these equalities into (14) gives the result. To obtain the total variation bound, one needs only to apply Jensen’s inequality, (16) ∫XdPθdQ−dPdQ2dQ≥∫XdPθdQ−dPdQdQ2=|Pθ−P|TV2.  □ Observe that, taking Q=P=∫ΘPθdμ(θ) in Proposition 2, one obtains a lower bound for the average f-divergence from the set of distribution to their barycenter, by the mean square total variation of the set of distributions to the barycenter, (17) κ∫Θ|Pθ−P|TV2dμ(θ)≤∫ΘDf(Pθ||P)dμ(θ). An alternative proof of this can be obtained by applying |Pθ−P|TV2≤Df(Pθ||P)/κ from Theorem 2 pointwise. The next result shows that, for f strongly convex, Pinsker type inequalities can never be reversed, Proposition 3. Given f strongly convex and M>0, there exists P, Q measures such that (18) Df(P||Q)≥M|P−Q|TV. Proof.  By κ-convexity ϕ(t)=f(t)−κt2/2 is a convex function. Thus, ϕ(t)≥ϕ(1)+ϕ+′(1)(t−1)=(f+′(1)−κ)(t−1) and hence limt→∞f(t)t≥limt→∞κt/2+(f+′(1)−κ)1−1t=∞. Taking measures on the two points space P={1/2,1/2} and Q={1/2t,1−1/2t} gives Df(P||Q)≥12f(t)t which tends to infinity with t→∞, while |P−Q|TV≤1. □ In fact, building on the work of Basu-Shioya-Park [24] and Vadja [25], Sason and Verdu proved [6] that, for any f divergence, supP≠QDf(P||Q)|P−Q|TV=f(0)+f∗(0). Thus, an f-divergence can be bounded above by a constant multiple of a the total variation, if and only if f(0)+f∗(0)<∞. From this perspective, Proposition 3 is simply the obvious fact that strongly convex functions have super linear (at least quadratic) growth at infinity. 3. Skew Divergences If we denote Cvx(0,∞) to be quotient of the cone of convex functions f on (0,∞) such that f(1)=0 under the equivalence relation f1∼f2 when f1−f2=c(x−1) for c∈R, then the map f↦Df gives a linear isomorphism between Cvx(0,∞) and the space of all f-divergences. The mapping T:Cvx(0,∞)→Cvx(0,∞) defined by Tf=f∗, where we recall f∗(t)=tf(t−1), gives an involution of Cvx(0,∞). Indeed, DTf(P||Q)=Df(Q||P), so that DT(T(f))(P||Q)=Df(P||Q). Mathematically, skew divergences give an interpolation of this involution as (P,Q)↦Df((1−t)P+tQ||(1−s)P+sQ) gives Df(P||Q) by taking s=1 and t=0 or yields Df∗(P||Q) by taking s=0 and t=1. Moreover, as mentioned in the Introduction, skewing imposes boundedness of the Radon–Nikodym derivative dPdQ, which allows us to constrain the domain of f-divergences and leverage κ-convexity to obtain f-divergence inequalities in this section. The following appears as Theorem III.1 in the preprint [26]. It states that skewing an f-divergence preserves its status as such. This guarantees that the generalized skew divergences of this section are indeed f-divergences. A proof is given in the Appendix A for the convenience of the reader. Theorem 4 (Melbourne et al [26]). For t,s∈[0,1] and a divergence Df, then (19) Sf(P||Q):=Df((1−t)P+tQ||(1−s)P+sQ) is an f-divergence as well. Definition 4. For an f-divergence, its skew symmetrization, Δf(P||Q):=12DfP||P+Q2+12DfQ||P+Q2. Δf is determined by the convex function (20) x↦1+x2f2x1+x+f21+x. Observe that Δf(P||Q)=Δf(Q||P), and when f(0)<∞, Δf(P||Q)≤supx∈[0,2]f(x)<∞ for all P,Q since dPd(P+Q)/2, dQd(P+Q)/2≤2. When f(x)=xlogx, the relative entropy’s skew symmetrization is the Jensen–Shannon divergence. When f(x)=(x−1)2 up to a normalization constant the χ2-divergence’s skew symmetrization is the Vincze–Le Cam divergence which we state below for emphasis. The work of Topsøe [11] provides more background on this divergence, where it is referred to as the triangular discrimination. Definition 5. When f(t)=(t−1)2t+1, denote the Vincze–Le Cam divergence by Δ(P||Q):=Df(P||Q). If one denotes the skew symmetrization of the χ2-divergence by Δχ2, one can compute easily from (20) that Δχ2(P||Q)=Δ(P||Q)/2. We note that although skewing preserves 0-convexity, by the above example, it does not preserve κ-convexity in general. The skew symmetrization of the χ2-divergence a 2-convex divergence while f(t)=(t−1)2/(t+1) corresponding to the Vincze–Le Cam divergence satisfies f″(t)=8(t+1)3, which cannot be bounded away from zero on (0,∞). Corollary 2. For an f-divergence such that f is a κ-convex on (0,2), (21) Δf(P||Q)≥κ4Δ(P||Q)=κ2Δχ2(P||Q), with equality when the f(t)=(t−1)2 corresponding the the χ2-divergence, where Δf denotes the skew symmetrized divergence associated to f and Δ is the Vincze- Le Cam divergence. Proof.  Applying Proposition 2 0=DfP+Q2||Q+P2≤12DfP||Q+P2+12DfQ||Q+P2−κ8∫2PP+Q−2QP+Q2d(P+Q)/2=Δf(P||Q)−κ4Δ(P||Q).  □ When f(x)=xlogx, we have f″(x)≥loge2 on [0,2], which demonstrates that up to a constant loge8 the Jensen–Shannon divergence bounds the Vincze–Le Cam divergence (see [11] for improvement of the inequality in the case of the Jensen–Shannon divergence, called the “capacitory discrimination” in the reference, by a factor of 2). We now investigate more general, non-symmetric skewing in what follows. Proposition 4. For α,β∈[0,1], define (22) C(α):=1−αwhenα≤βαwhenα>β, and (23) Sα,β(P||Q):=D((1−α)P+αQ||(1−β)P+βQ). Then, (24) Sα,β(P||Q)≤C(α)D∞(α||β)|P−Q|TV, where D∞(α||β):=logmaxαβ,1−α1−β is the binary ∞-Rényi divergence [27]. We need the following lemma originally proved by Audenart in the quantum setting [28]. It is based on a differential relationship between the skew divergence [12] and the [15] (see [29,30]). Lemma 3 (Theorem III.1 [26]). For P and Q probability measures and t∈[0,1], (25) S0,t(P||Q)≤−logt|P−Q|TV. Proof of Theorem 4. If α≤β, then D∞(α||β)=log1−α1−β and C(α)=1−α. In addition, (26) (1−β)P+βQ=t(1−α)P+αQ+(1−t)Q with t=1−β1−α, thus (27) Sα,β(P||Q)=S0,t((1−α)P+αQ||Q)≤(−logt)|((1−α)P+αQ)−Q|TV=C(α)D∞(α||β)|P−Q|TV, where the inequality follows from Lemma 3. Following the same argument for α>β, so that C(α)=α, D∞(α||β)=logαβ, and (28) (1−β)P+βQ=t(1−α)P+αQ+(1−t)P for t=βα completes the proof. Indeed, (29) Sα,β(P||Q)=S0,t((1−α)P+αQ||P)≤−logt|((1−α)P+αQ)−P|TV=C(α)D∞(α||β)|P−Q|TV.  □ We recover the classical bound [11,16] of the Jensen–Shannon divergence by the total variation. Corollary 3. For probability measure P and Q, (30) JSD(P||Q)≤log2|P−Q|TV Proof.  Since JSD(P||Q)=12S0,12(P||Q)+12S1,12(P||Q). □ Proposition 4 gives a sharpening of Lemma 1 of Nielsen [17], who proved Sα,β(P||Q)≤D∞(α||β), and used the result to establish the boundedness of a generalization of the Jensen–Shannon Divergence. Definition 6 (Nielsen [17]). For p and q densities with respect to a reference measure μ, wi>0, such that ∑i=1nwi=1 and αi∈[0,1], define (31) JSα,w(p:q)=∑i=1nwiD((1−αi)p+αiq||(1−α¯)p+α¯q) where ∑i=1nwiαi=α¯. Note that, when n=2, α1=1, α2=0 and wi=12, JSα,w(p:q)=JSD(p||q), the usual Jensen–Shannon divergence. We now demonstrate that Nielsen’s generalized Jensen–Shannon Divergence can be bounded by the total variation distance just as the ordinary Jensen–Shannon Divergence. Theorem 5. For p and q densities with respect to a reference measure μ, wi>0, such that ∑i=1nwi=1 and αi∈(0,1), (32) logeVarw(α)|p−q|TV2≤JSα,w(p:q)≤AH(w)|p−q|TV where H(w):=−∑iwilogwi≥0 and A=maxi|αi−α¯i| with α¯i=∑j≠iwjαj1−wi. Note that, since α¯i is the w average of the αj terms with αi removed, α¯i∈[0,1] and thus A≤1. We need the following Theorem from Melbourne et al. [26] for the upper bound. Theorem 6 ([26] Theorem 1.1). For fi densities with respect to a common reference measure γ and λi>0 such that ∑i=1nλi=1, (33) hγ(∑iλifi)−∑iλihγ(fi)≤TH(λ), where hγ(fi):=−∫fi(x)logfi(x)dγ(x) and T=supi|fi−f˜i|TV with f˜i=∑j≠iλj1−λifj. Proof of Theorem 5. We apply Theorem 6 with fi=(1−αi)p+αiq, λi=wi, and noticing that in general (34) hγ(∑iλifi)−∑iλhγ(fi)=∑iλiD(fi||f), we have (35) JSα,w(p:q)=∑i=1nwiD((1−αi)p+αiq||(1−α¯)p+α¯q)≤TH(w). It remains to determine T=maxi|fi−f˜i|TV, (36) f˜i−fi=f−fi1−λi=((1−α¯)p+α¯q)−((1−αi)p+αiq)1−wi=(αi−α¯)(p−q)1−wi=(αi−α¯i)(p−q). Thus, T=maxi(αi−α¯i)|p−q|TV=A|p−q|TV, and the proof of the upper bound is complete. To prove the lower bound, we apply Pinsker’s inequality, 2loge|P−Q|TV2≤D(P||Q), (37) JSα,w(p:q)=∑i=1nwiD((1−αi)p+αiq||(1−α¯)p+α¯q)≥12∑i=1nwi2loge|((1−αi)p+αiq)−((1−α¯)p+α¯q)|TV2=loge∑i=1nwi(αi−α¯)2|p−q|TV2=logeVarw(α)|p−q|TV2.  □ Definition 7. Given an f-divergence, densities p and q with respect to common reference measure, α∈[0,1]n and w∈(0,1)n such that ∑iwi=1 define its generalized skew divergence (38) Dfα,w(p:q)=∑i=1nwiDf((1−αi)p+αiq||(1−α¯)p+α¯q). where α¯=∑iwiαi. Note that, by Theorem 4, Dfα,w is an f-divergence. The generalized skew divergence of the relative entropy is the generalized Jensen–Shannon divergence JSα,w. We denote the generalized skew divergence of the χ2-divergence from p to q by (39) χα,w2(p:q):=∑iwiχ2((1−αi)p+αiq||(1−α¯p+α¯q) Note that, when n=2 and α1=0, α2=1 and wi=12, we recover the skew symmetrized divergence in Definition 4 (40) Df(0,1),(1/2,1/2)(p:q)=Δf(p||q) The following theorem shows that the usual upper bound for the relative entropy by the χ2-divergence can be reversed up to a factor in the skewed case. Theorem 7. For p and q with a common dominating measure μ, χα,w2(p:q)≤N∞(α,w)JSα,w(p:q). Writing N∞(α,w)=maximax1−αi1−α¯,αiα¯. For α∈[0,1]n and w∈(0,1)n such that ∑iwi=1, we use the notation N∞(α,w):=maxieD∞(αi||α¯) where α¯∑iwiαi. Proof.  By definition, JSα,w(p:q)=∑i=1nwiD((1−αi)p+αiq||(1−α¯)p+α¯q). Taking Pi to be the measure associated to (1−αi)p+αiq and Q given by (1−α¯)p+α¯q, then (41) dPidQ=(1−αi)p+αiq(1−α¯)p+α¯q≤max1−αi1−α¯,αiα¯=eD∞(αi||α¯)≤N∞(α,w). Since f(x)=xlogx, the convex function associated to the usual KL divergence, satisfies f″(x)=1x, f is e−D∞(α)-convex on [0,supx,idPidQ(x)], applying Proposition 2, we obtain (42) D∑iwiPi||Q≤∑iwiD(Pi||Q)−∑iwi∫XdPidQ−dPdQ2dQ2N∞(α,w). Since Q=∑iwiPi, the left hand side of (42) is zero, while (43) ∑iwi∫XdPidQ−dPdQ2dQ=∑iwi∫XdPidP−12dP=∑iwiχ2(Pi||P)=χα,w2(p:q). Rearranging gives, (44) χα,w2(p:q)2N∞(α,w)≤JSα,w(p:q), which is our conclusion. □ 4. Total Variation Bounds and Bayes Risk In this section, we derive bounds on the Bayes risk associated to a family of probability measures with a prior distribution λ. Let us state definitions and recall basic relationships. Given probability densities {pi}i=1n on a space X with respect a reference measure μ and λi≥0 such that ∑i=1nλi=1, define the Bayes risk, (45) R:=Rλ(p):=1−∫Xmaxi{λipi(x)}dμ(x) If ℓ(x,y)=1−δx(y), and we define T:=(x)argmaxiλipi(x) then observe that this definition is consistent with, the usual definition of the Bayes risk associated to the loss function ℓ. Below, we consider θ to be a random variable on {1,2,…,n} such that P(θ=i)=λi, and x to be a variable with conditional distribution P(X∈A|θ=i)=∫Api(x)dμ(x). The following result shows that the Bayes risk gives the probability of the categorization error, under an optimal estimator. Proposition 5. The Bayes risk satisfies R=minθ^Eℓ(θ,θ^(X))=Eℓ(θ,T(X)) where the minimum is defined over θ^:X→{1,2,…,n}. Proof.  Observe that R=1−∫XλT(x)pT(x)(x)dμ(x)=Eℓ(θ,T(X)). Similarly, Eℓ(θ,θ^(X))=1−∫Xλθ^(x)pθ^(x)(x)dμ(x)≥1−∫XλT(x)pT(x)(x)dμ(x)=R, which gives our conclusion. □ It is known (see, for example, [9,31]) that the Bayes risk can also be tied directly to the total variation in the following special case, whose proof we include for completeness. Proposition 6. When n=2 and λ1=λ2=12, the Bayes risk associated to the densities p1 and p2 satisfies (46) 2R=1−|p1−p2|TV Proof.  Since pT=|p1−p2|+p1+p22, integrating gives ∫XpT(x)dμ(x)=|p1−p2|TV+1 from which the equality follows. □ Information theoretic bounds to control the Bayes and minimax risk have an extensive literature (see, for example, [9,32,33,34,35]). Fano’s inequality is the seminal result in this direction, and we direct the reader to a survey of such techniques in statistical estimation (see [36]). What follows can be understood as a sharpening of the work of Guntuboyina [9] under the assumption of a κ-convexity. The function T(x)=argmaxi{λipi(x)} induces the following convex decompositions of our densities. The density q can be realized as a convex combination of q1=λTq1−Q where Q=1−∫λTqdμ and q2=(1−λT)qQ, q=(1−Q)q1+Qq2. If we take p∑iλipi, then p can be decomposed as ρ1=λTpT1−R and ρ2=p−λTpTR so that p=(1−R)ρ1+Rρ2. Theorem 8. When f is κ-convex, on (a,b) with a=infi,xpi(x)q(x) and b=supi,xpi(x)q(x) ∑iλiDf(pi||q)≥Df(R||Q)+κW2 where W:=W(λi,pi,q):=(1−R)21−Qχ2(ρ1||q1)+R2Qχ2(ρ2||q2)+W0 for W0≥0. W0 can be expressed explicitly as W0=∫(1−λT)Varλi≠Tpiqdμ=∫∑i≠Tλi|pi−∑j≠Tλj1−λTpj|2qdμ, where for fixed x, we consider the variance Varλi≠Tpiq to be the variance of a random variable taking values pi(x)/q(x) with probability λi/(1−λT(x)) for i≠T(x). Note this term is a non-zero term only when n>2. Proof.  For a fixed x, we apply Lemma 1 (47) ∑iλifpiq=λTfpTq+(1−λT)∑i≠Tλi1−λTfpiq≥λTfpTq+(1−λT)fp−λTpTq(1−λT)+κ2Varλi≠Tpiq Integrating, (48) ∑iλiDf(pi||q)≥∫λTfpTqq+∫(1−λT)f−λTpT+∑iλipiq(1−λT)q+κ2W0, where (49) W0=∫∑i≠T(x)λi1−λT(x)|pi−∑j≠Tλj1−λTpj|2qdμ. Applying the κ-convexity of f, (50) ∫λTfpTqq=(1−Q)∫q1fpTq≥(1−Q)f∫λTpT1−Q+κ2Varq1pTq=(1−Q)f((1−R)/(1−Q))+Qκ2W1, with (51) W1:=Varq1pTq=1−R1−Q2Varq1λTpTλTq1−Q1−R=1−R1−Q2Varq1ρ1q1=1−R1−Q2χ2(ρ1||q1) Similarly, (52) ∫(1−λT)fp−λTpTq(1−λT)q=Q∫q2fp−λTpTq(1−λT)≥Qf∫q2p−λTpTq(1−λT)+Qκ2W2=QfR1−Q+Qκ2W2 where (53) W2:=Varq2p−λTpTq(1−λT)=RQ2Varq2p−λTpTq(1−λT)QR=RQ2Varq2p−λTpTq(1−λT)−RQ2=RQ2∫q2ρ2q2−12=RQ2χ2(ρ2||q2) Writing W=W0+W1+W2, we have our result. □ Corollary 4. When λi=1n, and f is κ-convex on (infi,xpi/q,supi,xpi/q) (54) 1n∑iDf(pi||q)≥Df(R||(n−1)/n)+κ2n2(1−R)2χ2(ρ1||q)+nRn−12χ2(ρ2||q)+W0 further when n=2, (55) Df(p1||q)+Df(p2||q)2≥Df1−|p1−p2|TV2||12+κ2(1+|p1−p2|TV)2χ2(ρ1||q)+(1−|p1−p2|TV)2χ2(ρ2||q). Proof.  Note that q1=q2=q, since λi=1n implies λT=1n as well. In addition, Q=1−∫λTqdμ=n−1n so that applying Theorem 8 gives (56) ∑i=1nDf(pi||q)≥nDf(R||(n−1)/n)+κnW(λi,pi,q)2. The term W can be simplified as well. In the notation of the proof of Theorem 8, (57) W1=n2(1−R)2χ2(ρ1,q)W2=nRn−12χ2(ρ2||q)W0=∫1n−1∑i≠T(pi−1n−1∑j≠Tpj)2qdμ. For the special case, one needs only to recall R=1−|p1−p2|TV2 while inserting 2 for n. □ Corollary 5. When pi≤q/t∗ for t∗>0, and f(x)=xlogx ∑iλiD(pi||q)≥D(R||Q)+t∗W(λi,pi,q)2 for D(pi||q) the relative entropy. In particular, ∑iλiD(pi||q)≥D(p||q)+D(R||P)+t∗W(λi,pi,p)2 where P=1−∫λTpdμ for p=∑iλipi and t∗=minλi. Proof.  For the relative entropy, f(x)=xlogx is 1M-convex on [0,M] since f″(x)=1/x. When pi≤q/t∗ holds for all i, then we can apply Theorem 8 with M=1t∗. For the second inequality, recall the compensation identity, ∑iλiD(pi||q)=∑iλiD(pi||p)+D(p||q), and apply the first inequality to ∑iD(pi||p) for the result.  □ This gives an upper bound on the Jensen–Shannon divergence, defined as JSD(μ||ν)=12D(μ||μ/2+ν/2)+12D(ν||μ/2+ν/2). Let us also note that through the compensation identity ∑iλiD(pi||q)=∑iλiD(pi||p)+D(p||q), ∑iλiD(pi||q)≥∑iλiD(pi||p) where p=∑iλipi. In the case that λi=1N (58) ∑iλiD(pi||q)≥∑iλiD(pi||p)≥Qf1−RQ+(1−Q)fR1−Q+t∗W2 Corollary 6. For two densities p1 and p2, the Jensen–Shannon divergence satisfies the following, (59) JSD(p1||p2)≥D1−|p1−p2|TV2||1/2+14(1+|p1−p2|TV)2χ2(ρ1||p)+(1−|p1−p2|TV)2χ2(ρ2||p) with ρ(i) defined above and p=p1/2+p2/2. Proof.  Since pi(p1+p2)/2≤2 and f(x)=xlogx satisfies f″(x)≥12 on (0,2). Taking q=p1+p22, in the n=2 example of Corollary 4 with κ=12 yields the result. □ Note that 2D((1+V)/2||1/2)=(1+V)log(1+V)+(1−V)log(1−V)≥V2loge, we see that a further bound, (60) JSD(p1||p2)≥loge2V2+(1+V)2χ2(ρ1||p)+(1−V)2χ2(ρ2||p)4, can be obtained for V=|p1−p2|TV. On Topsøe’s Sharpening of Pinsker’s Inequality For Pi,Q probability measures with densities pi and q with respect to a common reference measure, ∑i=1nti=1, with ti>0, denote P=∑itiPi, with density p=∑itipi, the compensation identity is (61) ∑i=1ntiD(Pi||Q)=D(P||Q)+∑i=1ntiD(Pi||P). Theorem 9. For P1 and P2, denote Mk=2−kP1+(1−2−k)P2, and define M1(k)=Mk1{P1>P2}+P21{P1≤P2}Mk{P1>P2}+P2{P1≤P2}M2(k)=Mk1{P1≤P2}+P21{P1>P2}Mk{P1≤P2}+P2{P1>P2}, then the following sharpening of Pinsker’s inequality can be derived, D(P1||P2)≥(2loge)|P1−P2|TV2+∑k=0∞2kχ2(M1(k),Mk+1)2+χ2(M2(k),Mk+1)2. Proof.  When n=2 and t1=t2=12, if we denote M=P1+P22, then (61) reads as (62) 12D(P1||Q)+12D(P2||Q)=D(M||Q)+JSD(P1||P2). Taking Q=P2, we arrive at (63) D(P1||P2)=2D(M||P2)+2JSD(P1||P2) Iterating and writing Mk=2−kP1+(1−2−k)P2, we have (64) D(P1||P2)=2nD(Mn||P2)+2∑k=0nJSD(Mn||P2) It can be shown (see [11]) that 2nD(Mn||P2)→0 with n→∞, giving the following series representation, (65) D(P1||P2)=2∑k=0∞2kJSD(Mk||P2). Note that the ρ-decomposition of Mk is exactly ρi=Mk(i), thus, by Corollary 6, (66) D(P1||P2)=2∑k=0∞2kJSD(Mk||P2)≥∑k=0∞2k|Mk−P2|TV2loge+χ2(M1(k),Mk+1)2+χ2(M2(k),Mk+1)2=(2loge)|P1−P2|TV2+∑k=0∞2kχ2(M1(k),Mk+1)2+χ2(M2(k),Mk+1)2. Thus, we arrive at the desired sharpening of Pinsker’s inequality. □ Observe that the k=0 term in the above series is equivalent to (67) 20χ2(M1(0),M0+1)2+χ2(M2(0),M0+1)2=χ2(ρ1,p)2+χ2(ρ2,p)2, where ρi is the convex decomposition of p=p1+p22 in terms of T(x)=argmax{p1(x),p2(x)}. 5. Conclusions In this article, we begin a systematic study of strongly convex divergences, and how the strength of convexity of a divergence generator f, quantified by the parameter κ, influences the behavior of the divergence Df. We prove that every strongly convex divergence dominates the square of the total variation, extending the classical bound provided by the χ2-divergence. We also study a general notion of a skew divergence, providing new bounds, in particular for the generalized skew divergence of Nielsen. Finally, we show how κ-convexity can be leveraged to yield improvements of Bayes risk f-divergence inequalities, and as a consequence achieve a sharpening of Pinsker’s inequality. Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Funding This research was funded by NSF grant CNS 1809194. Conflicts of Interest The authors declare no conflict of interest. Appendix A Theorem A1. The class of f-divergences is stable under skewing. That is, if f is convex, satisfying f(1)=0, then (A1) f^(x):=(tx+(1−t))frx+(1−r)tx+(1−t) is convex with f^(1)=0 as well. Proof.  If μ and ν have respective densities u and v with respect to a reference measure γ, then rμ+(1−r)ν and tμ+1−tν have densities ru+(1−r)v and tu+(1−t)v (A2) Sf,r,t(μ||ν)=∫fru+(1−r)vtu+(1−t)v(tu+(1−t)v)dγ (A3) =∫fruv+(1−r)tuv+(1−t)(tuv+(1−t))vdγ (A4) =∫f^uvvdγ. Since f^(1)=f(1)=0, we need only prove f^ convex. For this, recall that the conic transform g of a convex function f defined by g(x,y)=yf(x/y) for y>0 is convex, since (A5) y1+y22fx1+x22/y1+y22=y1+y22fy1y1+y2x1y1+y2y1+y2x2y2 (A6) ≤y12f(x1/y1)+y22f(x2/y2). Our result follows since f^ is the composition of the affine function A(x)=(rx+(1−r),tx+(1−t)) with the conic transform of f, (A7) f^(x)=g(A(x)).  □ entropy-22-01327-t001_Table 1Table 1 Examples of Strongly Convex Divergences. Divergence f κ Domain relative entropy (KL) tlogt 1M (0,M] total variation |t−1|2 0 (0,∞) Pearson’s χ2 (t−1)2 2 (0,∞) squared Hellinger 2(1−t) M−32/2 (0,M] reverse relative entropy −logt 1/M2 (0,M] Vincze- Le Cam (t−1)2t+1 8(M+1)3 (0,M] Jensen–Shannon (t+1)log2t+1+tlogt 1M(M+1) (0,M] Neyman’s χ2 1t−1 2/M3 (0,M] Sason’s s log(s+t)(s+t)2−log(s+1)(s+1)2 2log(s+M)+3 [M,∞), s>e−3/2 α-divergence 41−t1+α21−α2,α≠±1 Mα−32 [M,∞),α>3(0,M],α<3 ==== Refs References 1. Ali S.M. Silvey S.D. A general class of coefficients of divergence of one distribution from another J. Roy. Statist. Soc. Ser. B 1966 28 131 142 10.1111/j.2517-6161.1966.tb00626.x 2. Morimoto T. Markov processes and the H-theorem J. Phys. Soc. Jpn. 1963 18 328 331 10.1143/JPSJ.18.328 3. Csiszár I. Eine informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten Magyar Tud. Akad. Mat. Kutató Int. Közl. 1963 8 85 108 4. Csiszár I. Information-type measures of difference of probability distributions and indirect observation Stud. Sci. Math. Hung. 1967 2 229 318 5. Polyanskiy Y. Poor H.V. Verdú S. Channel coding rate in the finite blocklength regime IEEE Trans. Inf. Theory 2010 56 2307 2359 10.1109/TIT.2010.2043769 6. Sason I. Verdú S. f -divergence inequalities IEEE Trans. Inf. Theory 2016 62 5973 6006 10.1109/TIT.2016.2603151 7. Polyanskiy Y. Wu Y. Lecture Notes on Information Theory Available online: http://people.lids.mit.edu/yp/homepage/data/itlectures_v5.pdf (accessed on 13 November 2019) 8. Sason I. On data-processing and majorization inequalities for f-divergences with applications Entropy 2019 21 1022 10.3390/e21101022 9. Guntuboyina A. Lower bounds for the minimax risk using f -divergences, and applications IEEE Trans. Inf. Theory 2011 57 2386 2399 10.1109/TIT.2011.2110791 10. Reid M. Williamson R. Generalised Pinsker inequalities arXiv 2009 0906.1244 11. Topsøe F. Some inequalities for information divergence and related measures of discrimination IEEE Trans. Inf. Theory 2000 46 1602 1609 10.1109/18.850703 12. Lee L. Measures of distributional similarity Proceedings of the 37th Annual Meeting of the Association For Computational Linguistics on Computational Linguistics Association for Computational Linguistics Stroudsburg, PA, USA 1999 25 32 13. Le Cam L. Asymptotic Methods in Statistical Decision Theory Springer Series in Statistics Springer New York, NY, USA 1986 14. Vincze I. On the concept and measure of information contained in an observation Contrib. Probab. 1981 207 214 10.1016/B978-0-12-274460-0.50023-0 15. Györfi L. Vajda I. A class of modified Pearson and Neyman statistics Stat. Decis. 2001 19 239 251 10.1524/strm.2001.19.3.239 16. Lin J. Divergence measures based on the Shannon entropy IEEE Trans. Inf. Theory 1991 37 145 151 10.1109/18.61115 17. Nielsen F. On a generalization of the Jensen–Shannon divergence and the Jensen–Shannon centroid Entropy 2020 22 221 10.3390/e22020221 18. Folland G. Real Analysis: Modern Techniques and Their Applications John Wiley & Sons Hoboken, NJ, USA 1999 19. Gibbs A.L. Su F.E. On choosing and bounding probability metrics Int. Stat. Rev. 2002 70 419 435 10.1111/j.1751-5823.2002.tb00178.x 20. Harremoës P. Vajda I. On pairs of f-divergences and their joint range IEEE Trans. Inf. Theory 2011 57 3230 3235 10.1109/TIT.2011.2137353 21. Reiss R. Approximate Distributions of Order Statistics: With Applications to Nonparametric Statistics Springer Berlin/Heidelberg, Germany 2012 22. Nielsen F. Nock R. On the chi square and higher-order chi distances for approximating f-divergences IEEE Signal Process. Lett. 2013 21 10 13 10.1109/LSP.2013.2288355 23. Amari S. Information Geometry and Its Applications Springer Berlin/Heidelberg, Germany 2016 194 24. Basu A. Shioya H. Park C. Statistical Inference: The Minimum Distance Approach CRC Press Boca Raton, FL, USA 2011 25. Vajda I. On the f-divergence and singularity of probability measures Period. Math. Hung. 1972 2 223 234 10.1007/BF02018663 26. Melbourne J. Talukdar S. Bhaban S. Madiman M. Salapaka M.V. The differential entropy of mixtures: new bounds and applications arXiv 2020 1805.11257 27. Erven T.V. Harremos P. Rényi divergence and Kullback-Leibler divergence IEEE Trans. Inf. Theory 2014 60 3797 3820 10.1109/TIT.2014.2320500 28. Audenaert K.M.R. Quantum skew divergence J. Math. Phys. 2014 55 112202 10.1063/1.4901039 29. Melbourne J. Madiman M. Salapaka M.V. Relationships between certain f-divergences Proceedings of the 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton) Monticello, IL, USA, 24–27 September 2019 1068 1073 30. Nishiyama T. Sason I. On relations between the relative entropy and χ 2-divergence, generalizations and applications Entropy 2020 22 563 10.3390/e22050563 31. Nielsen F. Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means Pattern Recognit. Lett. 2014 42 25 34 10.1016/j.patrec.2014.01.002 32. Birgé L. A new lower bound for multiple hypothesis testing IEEE Trans. Inf. Theory 2005 51 1611 1615 10.1109/TIT.2005.844101 33. Chen X. Guntuboyina A. Zhang Y. On Bayes risk lower bounds J. Mach. Learn. Res. 2016 17 7687 7744 34. Xu A. Raginsky M. Information-theoretic lower bounds on Bayes risk in decentralized estimation IEEE Trans. Inf. Theory 2016 63 1580 1600 10.1109/TIT.2016.2646342 35. Yang Y. Barron A. Information-theoretic determination of minimax rates of convergence Ann. Statist. 1999 27 1564 1599 36. Scarlett J. Cevher V. An introductory guide to Fano’s inequality with applications in statistical estimation arXiv 2019 1901.00555