Skip to content

Instantly share code, notes, and snippets.

@nikete
Last active January 14, 2025 23:56
Show Gist options
  • Save nikete/9aec37d67ba1d12157b8d0a38e29a933 to your computer and use it in GitHub Desktop.
Save nikete/9aec37d67ba1d12157b8d0a38e29a933 to your computer and use it in GitHub Desktop.
o1 pro on

Proof

Let $\Omega$ be finite, and let $P,Q$ be probability distributions on $\Omega$ with $|P - Q|_1 \leq \frac{1}{2}$. Denote $|P-Q|1$ by $\alpha$. Let $H(P) = -\sum{x\in \Omega}p(x)\log p(x).$ We prove $$ |H(P) - H(Q)| \leq \alpha \log!\left(\frac{|\Omega|}{\alpha}\right). $$

Let $\alpha = |P - Q|1$. Let $S = {x \in \Omega : p(x) \geq \frac{\alpha}{|\Omega|}}$. Then $\sum{x\in S^c}p(x) \leq |S^c|\cdot \frac{\alpha}{|\Omega|} \leq \alpha$. Hence, for all $x \in S$, $\log!\left(\frac{1}{p(x)}\right) \leq \log!\left(\frac{|\Omega|}{\alpha}\right)$. Assume w.l.o.g. that $H(P)\geq H(Q)$. Then $$ H(P)-H(Q) = \sum_{x\in \Omega}\left[p(x)\log\frac{1}{p(x)} - q(x)\log\frac{1}{q(x)}\right]. $$ Split the sum over $S$ and $S^c$: $$ \sum_{x\in \Omega}p(x)\log\frac{1}{p(x)} = \sum_{x\in S}p(x)\log\frac{1}{p(x)} + \sum_{x\in S^c}p(x)\log\frac{1}{p(x)}. $$ By line 4, for $x \in S$, $\log\frac{1}{p(x)} \leq \log\frac{|\Omega|}{\alpha}$. For $x \in S^c$, $p(x)\leq \frac{\alpha}{|\Omega|}$. Thus $$ \sum_{x\in S^c} p(x)\log\frac{1}{p(x)} \leq \left(\frac{\alpha}{|\Omega|}\cdot|S^c|\right)\log\frac{|\Omega|}{\alpha} \leq \alpha\log\frac{|\Omega|}{\alpha}. $$ Let $\Delta_x = p(x)-q(x)$. Then $\sum_{x\in \Omega}\Delta_x=0$ and $\sum_{x\in \Omega}|\Delta_x|=\alpha$. Rewrite $$ p(x)\log\frac{1}{p(x)} - q(x)\log\frac{1}{q(x)} = \Delta_x\log\frac{1}{p(x)} + q(x)\left[\log\frac{1}{p(x)} - \log\frac{1}{q(x)}\right]. $$ For $x$ with $\Delta_x>0$, we have $p(x)\geq q(x)$. Then $\log\frac{1}{p(x)} - \log\frac{1}{q(x)} = \log!\left(\frac{q(x)}{p(x)}\right)\leq 0$. Thus, on those $x$ with $\Delta_x>0$, the second term in line 9 is $\leq 0$. Hence $$ p(x)\log\frac{1}{p(x)} - q(x)\log\frac{1}{q(x)} \leq \Delta_x\log\frac{1}{p(x)} \leq \Delta_x\log\frac{|\Omega|}{\alpha}. $$ Summing over all $x$ with $\Delta_x>0$ yields $$ \sum_{\substack{x:,\Delta_x>0}}\left[p(x)\log\frac{1}{p(x)} - q(x)\log\frac{1}{q(x)}\right] \leq \left(\sum_{\Delta_x>0}\Delta_x\right)\log\frac{|\Omega|}{\alpha}. $$ Since $\sum_{\Delta_x>0}\Delta_x \leq \alpha$, the sum in line 12 is $\leq \alpha\log!\left(\frac{|\Omega|}{\alpha}\right)$. A symmetric argument applies to those $x$ with $\Delta_x<0$. Combining the bounds for $x\in S\cup S^c$ and for the cases $\Delta_x>0$ or $\Delta_x<0$, we get $$ H(P)-H(Q) \leq \alpha\log!\left(\frac{|\Omega|}{\alpha}\right). $$ Reversing the roles of $P$ and $Q$ shows $\left|H(P)-H(Q)\right|\leq \alpha\log!\left(\frac{|\Omega|}{\alpha}\right)$. $\square$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment