Machine Learning - MCQ Practice Questions

Bias refers to the error due to overly simplistic assumptions in the learning algorithm. A high-bias model underfits the data. Variance refers to the sensitivity of the model to fluctuations in training data. A high-variance model overfits the data by learning noise. The tradeoff involves finding a balance to minimize total error.

The sigmoid function $\sigma (z)=\frac{1}{1+e^{-z}}$ maps any real-valued input to the range $(0,1)$. At the boundaries, $\sigma (z)\to 0$ as $z\to -\infty$ and $\sigma (z)\to 1$ as $z\to +\infty$. It never actually reaches 0 or 1, so the strict range is $(0,1)$, which is best described by the closed interval $[0,1]$ among the given options.

Lasso (Least Absolute Shrinkage and Selection Operator) regression uses L1 regularization, adding a penalty term $\lambda {\sum }_j|w_j|$ to the loss function. This can shrink some coefficients exactly to zero, effectively performing feature selection. Ridge regression uses L2 regularization with a penalty of $\lambda {\sum }_j{w}_j^2$.

The formula $H=-{\sum }_{i=1}^c{p}_i{\log }_2(p_i)$ defines Shannon Entropy, which measures the impurity or disorder at a node. Information Gain is computed as the reduction in entropy after a split. Gini Impurity uses a different formula: $G=1-{\sum }_{i=1}^c{p}_i^2$. Entropy is the correct answer here.

Bagging (Bootstrap Aggregating) trains multiple models in parallel, each on a different random bootstrap sample of the training data, and combines predictions by averaging or voting. Boosting trains models sequentially, where each new model focuses on correcting the errors made by previous models by adjusting sample weights. Random Forest is a popular bagging method; AdaBoost and Gradient Boosting are boosting methods.

The F1-score is the harmonic mean of Precision and Recall:

It is preferred over simple average when there is class imbalance, as it penalizes extreme differences between precision and recall. Option A is the arithmetic mean, not the harmonic mean.

The k-means algorithm minimizes the Within-Cluster Sum of Squares (WCSS), also called inertia:

where ${\mu }_k$ is the centroid of cluster $C_k$. The algorithm iteratively assigns points to the nearest centroid and recomputes centroids to minimize this objective.

The kernel trick is a key feature of SVMs. It allows the algorithm to compute dot products in a high-dimensional (or even infinite-dimensional) feature space implicitly using a kernel function $K(x_i,x_j)=\varphi (x_i)\cdot \varphi (x_j)$, without explicitly computing $\varphi$. This enables SVMs to handle non-linear data. SVMs maximize the margin (distance between classes), not within-class variance. Support vectors are the points closest to the decision boundary, not the farthest.

Leave-One-Out Cross-Validation (LOOCV) is ideal for small datasets. In LOOCV, each data point is used once as a test set while all remaining $n-1$ points form the training set. This is repeated $n$ times. It makes maximum use of available data and provides an almost unbiased estimate of model performance, though it can be computationally expensive for large datasets.

In gradient descent, the parameter update rule is $w:=w-\alpha {\nabla }_{w}J(w)$. If the learning rate $\alpha$ is too large, the step size overshoots the minimum. This causes the loss function to oscillate around or diverge away from the minimum rather than converging. A too-small learning rate causes very slow convergence. An optimal learning rate balances convergence speed and stability.