In x264, the --aq-mode
option controls macroblock-level adaptive quantization. In mode 0 (disabled), all blocks use the frame’s base QP. In mode 1 (variance AQ) and mode 2 (auto-variance AQ), x264 measures each block’s activity (roughly its AC variance) to adjust its QP: complex blocks get higher QP (fewer bits) while flat/dark blocks get lower QP (more bits). The goal is to maintain overall bitrate while improving perceptual quality (less banding in flat areas). x264’s code was tuned so that AQ modes use roughly the same total bits as no-AQ (see comment in code). Below we detail each mode’s formula, code logic, and effects.
AQ mode 0 turns off adaptive quantization. In this case x264 simply sets all MB-level offsets to zero. In code, if i_aq_mode==X264_AQ_NONE
or strength=0, x264 does:
memset(frame->f_qp_offset, 0, mb_count*sizeof(float));