README.md

Self note: normalize P/U/N into a “quality” score (fair across sample sizes)

Given counts per item:

• P = positive, U = neutral, N = negative
• T = P + U + N

0) Decide what neutral means

Default assumption: neutral = “meh / no push”, so it dilutes by being included in T.

Vote mapping (utility):

• positive -> +1
• neutral -> 0
• negative -> -1

True latent mixture: θ = (θP, θU, θN) True sentiment score: S(θ) = θP - θN (range [-1,+1])

Naive score (don’t use for ranking):

• S_naive = (P - N) / T

Problem: small T is noisy.

---

1) Add shrinkage (Dirichlet prior)

Use Dirichlet prior on θ:

• θ ~ Dirichlet(αP, αU, αN)

Pick prior strength k (typical 10).

Option A (easy): symmetric prior

• αP = αU = αN = k/3

Option B (better): global baseline prior

• gP = ΣP/ΣT, gU = ΣU/ΣT, gN = ΣN/ΣT
• α = k * (gP, gU, gN)

Posterior parameters:

• aP = P + αP
• aU = U + αU
• aN = N + αN
• A = aP + aU + aN (= T + k)

Smoothed mean sentiment:

• μ = E[S] = (aP - aN) / A

---

2) Quality = conservative lower bound (ranking metric)

Goal: “pretty sure it’s good”, not “maybe good”.

Analytic variance for S = θP - θN under Dirichlet(a):

• Var(S) = (A*(aP + aN) - (aP - aN)^2) / (A^2 * (A + 1))
• σ = sqrt(Var(S))

One-sided 5% lower bound (default):

• Q = μ - 1.645 * σ

Notes:

• Q in [-1,+1]
• Small T => larger σ => Q drops (good)
• Big T => σ shrinks => Q ~ μ

---

3) What to display vs what to sort by

Sort by:

• Q (lower bound)

Display:

• μ (smoothed mean)
• T (evidence)
• optionally decisiveness: D = 1 - U/T

---

4) Variants if neutral semantics differ

Neutral is “no signal” (don’t dilute):

• compute using only P and N:
  • T' = P + N
  • apply same prior logic to (P,N) as a Beta posterior
  • score is θP - θN = 2θP - 1

Neutral is slightly positive:

• use weights w = (1, 0.2, -1)
• then S(θ) = wP θP + wU θU + wN θN
• same Dirichlet posterior, same “lower bound” idea

quality.py

import math
import numpy as np

def quality_score(P, U, N, k=10.0, weights=(1.0, 0.0, -1.0), global_rates=None, q=0.05):
    wP, wU, wN = weights

    if global_rates is None:
        alphaP = alphaU = alphaN = k / 3.0
    else:
        gP, gU, gN = global_rates
        alphaP, alphaU, alphaN = k * gP, k * gU, k * gN

    aP, aU, aN = P + alphaP, U + alphaU, N + alphaN
    A = aP + aU + aN

    # posterior mean of score
    mu = (wP * aP + wU * aU + wN * aN) / A

    # posterior variance of score (Dirichlet)
    sum_w2a = (wP*wP)*aP + (wU*wU)*aU + (wN*wN)*aN
    sum_wa  = (wP*aP + wU*aU + wN*aN)
    var = (A * sum_w2a - sum_wa * sum_wa) / (A*A*(A+1.0))
    sd = math.sqrt(max(var, 0.0))

    # one-sided normal quantile for q=0.05
    z = 1.645 if abs(q - 0.05) < 1e-12 else float(np.abs(np.quantile(np.random.normal(size=200000), q)))
    lb = mu - z * sd

    return mu, lb

def quality_score_mc(P, U, N, k=10.0, weights=(1.0, 0.0, -1.0), global_rates=None, q=0.05, n_samples=5000, rng=None):
    rng = np.random.default_rng() if rng is None else rng
    w = np.array(weights, dtype=float)

    if global_rates is None:
        alpha = np.array([k/3.0, k/3.0, k/3.0], dtype=float)
    else:
        alpha = k * np.array(global_rates, dtype=float)

    a = np.array([P, U, N], dtype=float) + alpha

    # Dirichlet sampling via Gamma
    x = rng.gamma(shape=a, scale=1.0, size=(n_samples, 3))
    theta = x / x.sum(axis=1, keepdims=True)
    s = theta @ w
    return float(s.mean()), float(np.quantile(s, q))