Implications of Classifier-Free Guidance in Auto-regressive Models

[fkcptlst]

Hi, thank you for the insightful work!

I have some concerns regarding the classifier-free guidance (CFG) in auto-regressive models.

CFG in this work is implemented as follows:

VAR/models/var.py

Lines 191 to 192 in 1ae5177

	t = cfg * ratio
	logits_BlV = (1+t) * logits_BlV[:B] - t * logits_BlV[B:]

However, it's important to note that CFG in auto-regressive models differs fundamentally from that in diffusion models (as outlined in Section 4 of this blog). In essence, the guidance in diffusion models is not theoretically applicable to auto-regressive models.

I am curious if this difference yields any notable empirical results. Have you conducted any quantitative or qualitative studies on the impact of CFG on this auto-regressive model? I would greatly appreciate any insights or empirical findings you could share on this subject.

[keyu-tian]

@fkcptlst in the Ablation Study section of the paper we tested the influence of CFG. We simply follow Google MUSE's CFG introduced in their paper. https://sander.ai/2022/05/26/guidance.html seems a thorough analysis on CFG. We'll check that later and maybe try some more implementations. Thank you for providing this!