Which statement best describes the forward and reverse processes in a typical diffusion model (e.g., Denoising Diffusion Probabilistic Models)?

• A single-pass network takes random noise and produces an image in one forward pass, with no iterative steps.
• During the forward process, noise is iteratively removed from real data until it becomes pure noise, and in reverse the model adds noise step by step to create new images.
• In the forward process, a small amount of noise is added to real data at each step until it becomes nearly pure noise; the reverse process is then learned to denoise step by step. <- correct
• The diffusion model relies on adversarial training where a discriminator oversees both noising and denoising.

解法

DDPM 定义了一条固定的马尔可夫链，前向过程 q 逐步加高斯噪声；模型学习逆向马尔可夫链，每步预测被加入的噪声。

Which characterizes the Kullback-Leibler divergence D(P || Q)?

• D(P || Q) is symmetric, D(P || Q) = D(Q || P).
• D(P || Q) satisfies the triangle inequality (true metric).
• D(P || Q) is always non-negative and equals zero iff P and Q are identical almost everywhere. <- correct
• D(P || Q) can be negative if P has nonzero probability where Q is zero.

解法

由 Gibbs 不等式，KL ≥ 0，等号当且仅当 P = Q 几乎处处成立；KL 不对称，也不是度量。

Which activation function saturates for both large negative and large positive inputs?

• ReLU
• Tanh <- correct
• Swish (SiLU)
• Leaky ReLU

解法

Tanh 把输入压到 (-1, 1) 且两端饱和；ReLU / LeakyReLU / Swish 在正方向都无上界。

How does Rotary Positional Embedding (RoPE) encode positional information?

• Add a sinusoidal position vector to each token embedding before self-attention.
• Apply a position-dependent rotation to Q and K vectors, encoding relative positions through phase differences. <- correct
• Learn absolute positional embeddings and add them to token embeddings.
• Modify attention scores by directly computing a learned bias based on token distance.

How many ReLU (or GELU) layers are used in a classical transformer layer?

• 1 <- correct
• 2
• 3
• 4

Why is cross-entropy (with softmax outputs) such a popular loss for multi-class classification?

• It maximizes entropy in the predictions, spreading probability evenly across classes.
• It exactly matches the 0-1 loss at test time.
• It allows gradient-based training and strongly penalizes assigning low probability to the correct class, aligning with the likelihood principle. <- correct
• It only requires logits and cannot be used with probabilities.

Please implement a softmax function with numpy. You are not allowed to call any existing functions such as scipy.special.softmax.

Note: don't write a for loop. Hint: a^b / a^c = a^(b-k) / a^(c-k).

Given a single layer of self-attention, manually figure out the parameters Q, K, V (i.e. W_q, W_k, W_v) required to compute the sum of two scalar inputs.

Inputs:
 v.shape == (1, 2, 1) # batch x sequence x hidden, values e.g. [1, 10]
Expected:
 result[0, 0, 0] == 1 + 10 == 11