Deep Learning - MCQ Practice Questions

The sigmoid activation function squashes inputs into the range $(0,1)$. Its derivative is at most $0.25$, so during backpropagation through many layers, gradients are repeatedly multiplied by values less than $1$, causing them to shrink exponentially — this is the vanishing gradient problem. ReLU and Leaky ReLU were introduced partly to mitigate this issue.

Stride specifies the number of pixels by which the convolutional filter shifts across the input at each step. A stride of $1$ moves the filter one pixel at a time, while a stride of $2$ moves it two pixels, effectively reducing the spatial dimensions of the output feature map.

Batch Normalization normalizes the activations (pre- or post-activation) of a layer by computing the mean ${\mu }_B$ and variance ${\sigma }_B^2$ over the current mini-batch. The normalized value is $\hat{x}=\frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$, which is then scaled and shifted by learnable parameters $\gamma$ and $\beta$.

Dropout randomly sets a fraction $p$ of neuron activations to zero during each training forward pass. This prevents co-adaptation of neurons, forcing the network to learn more robust and distributed representations. At inference time, all neurons are active, and weights are scaled by $(1-p)$ to compensate.

Adam (Adaptive Moment Estimation) maintains exponentially decaying moving averages of past gradients $m_t$ (first moment) and past squared gradients $v_t$ (second moment). The update rule is ${\theta }_{t+1}={\theta }_t-\frac{\eta }{\sqrt{{\hat{v}}_t}+ϵ}{\hat{m}}_t$, combining the benefits of momentum and RMSProp.

The forget gate in an LSTM uses a sigmoid activation to output values between $0$ and $1$ for each element of the cell state $C_{t-1}$. A value near $0$ means 'forget this', while a value near $1$ means 'keep this'. It is computed as $f_t=\sigma (W_f\cdot [h_{t-1},x_t]+b_f)$.

Applying the formula: $\lfloor \frac{32+2(0)-5}{1}+1\rfloor =\lfloor \frac{27}{1}+1\rfloor =28$. So the output feature map is $28\times 28$.

The core innovation of the Transformer architecture is the self-attention mechanism, which allows every token in a sequence to attend to every other token simultaneously. This parallelism overcomes the sequential dependency of RNNs/LSTMs, enabling much faster training and better capturing of long-range dependencies.

When the target dataset is small and similar to the source dataset, the pre-trained convolutional base already contains relevant feature representations. Freezing these weights prevents overfitting on the small dataset, while training only the final classifier head adapts the model to the new task with minimal risk of overfitting.

In the cross-entropy formula $L=-{\sum }_{c=1}^C{y}_c\log ({\hat{p}}_c)$, only the term corresponding to the true class contributes since $y_c=0$ for all other classes. Here the true class is class 2 with $y_2=1$ and ${\hat{p}}_2=0.7$, so $L=-(1\cdot \log (0.7))=-\log (0.7)\approx 0.357$.

In a GAN, the discriminator acts as a binary classifier. Its goal is to correctly identify whether a given sample is real (from the true data distribution) or fake (generated by the generator). The generator tries to fool the discriminator, while the discriminator tries to resist being fooled — this adversarial dynamic drives training.

The ReLU (Rectified Linear Unit) activation function outputs zero for all negative inputs. If a neuron's weights cause it to always receive negative pre-activation values, the gradient through it becomes zero permanently, and the neuron never updates — this is called the 'dying ReLU' problem. Leaky ReLU was introduced to address this.

In BPTT, gradients are multiplied repeatedly through many time steps. If the gradient magnitudes are less than 1, repeated multiplication causes them to shrink exponentially (vanishing gradients), making it hard to learn long-range dependencies. If greater than 1, they grow exponentially (exploding gradients). Techniques like gradient clipping, LSTMs, and GRUs were developed to mitigate these issues.

The receptive field of a neuron is the region of the original input (e.g., pixels in an image) that can affect the value of that neuron. In deeper layers, neurons have larger receptive fields because each layer aggregates information from a broader spatial region of the previous layer. Designing networks with large effective receptive fields is important for capturing global context.

In CNNs, a single convolutional filter slides (convolves) across the entire spatial extent of the input, using the same set of weights at every location. This property is called weight sharing. It drastically reduces the number of learnable parameters compared to a fully connected layer and exploits the translation invariance property of images.

Hyperparameters are configuration settings that govern the training process and model structure — they are set before training and are not learned from data. Examples include learning rate, number of layers, number of neurons per layer, batch size, and dropout rate. In contrast, weights and biases are parameters learned during training via gradient descent.

Max pooling selects the maximum value within each pooling window. This operation retains the most prominent (strongest) activation in a local region, making the network more robust to small translations and distortions. It also introduces a degree of spatial invariance. Max pooling is the most widely used pooling type in standard CNN architectures.

Residual connections add the input $x$ of a block directly to its output $\mathcal{F}(x)$, so the block learns the residual $\mathcal{F}(x)=H(x)-x$ rather than the full mapping $H(x)$. If the optimal mapping is close to the identity, it is easier to push $\mathcal{F}(x)$ toward zero than to learn the identity directly. This also helps gradients flow through the network unimpeded, enabling training of very deep networks (100+ layers).

When $d_k$ (the dimension of the keys) is large, the dot products $Q{K}^T$ can become very large in magnitude. Large input values to the softmax function lead to extremely small gradients (the softmax output becomes nearly one-hot), making training difficult. Dividing by $\sqrt{d_k}$ keeps the dot products in a more reasonable range, ensuring the softmax produces well-distributed attention weights and gradients flow properly.

In a standard autoencoder, the encoder maps an input to a single fixed latent vector, which makes the latent space potentially discontinuous and unsuitable for generation. A VAE instead encodes each input as a distribution (parameterized by a mean $\mu$ and variance ${\sigma }^2$). During training, latent vectors are sampled from this distribution using the reparameterization trick $z=\mu +\sigma \cdot ϵ$, where $ϵ\sim \mathcal{N}(0,I)$. This makes the latent space continuous and structured, enabling meaningful generation of new samples.