AWS Certified Machine Learning – Specialty — Practice Set 1

Question 1

A machine learning engineer is building a training pipeline for a fraud detection model. The training data is stored in an Amazon S3 bucket as 50,000 Parquet files totaling 8 TB. The files are partitioned by year, month, and day. The Amazon SageMaker training job currently reads each file individually using the default File mode, causing the training data download phase to take 3 hours before the actual training begins. The engineer needs to reduce the data loading time to under 30 minutes without rewriting the training code. Which solution MOST effectively addresses this requirement?

Accepted Answer

Amazon FSx for Lustre integrated with Amazon S3 provides a high-performance parallel file system that can deliver hundreds of GB/s of throughput, making it ideal for large-scale ML training datasets. When the SageMaker training job mounts the FSx for Lustre file system, data is accessed at filesystem speeds without a pre-download phase — training starts immediately and data is lazily loaded. This is the recommended approach for training on multi-terabyte datasets where the download phase is a bottleneck. Option B Pipe mode streams data and reduces pre-download time, but requires the training code to handle the FIFO pipe interface (which violates 'without rewriting the training code'). Additionally, Pipe mode works best with RecordIO format, not arbitrary Parquet files read via file system paths. Option A consolidating into a single file reduces GET overhead but does not change the fundamental 8 TB download requirement. Option D S3 Transfer Acceleration improves cross-region upload/download speeds but offers only marginal improvement for large-volume data and does not eliminate the sequential download phase.

Question 2

A company has clickstream data arriving in real time from a web application at 500,000 events per second. The data must be available for ML feature engineering within 60 seconds of ingestion and must be queryable by data scientists using standard SQL for exploratory analysis. The raw events contain JSON payloads of variable schema. The company needs a pipeline that can handle schema evolution without requiring manual intervention when new event fields are added. Which architecture BEST meets all requirements?

Accepted Answer

Kinesis Data Streams absorbs the 500,000 events/second ingestion rate. Kinesis Data Firehose with Lambda transformation converts raw JSON to Parquet and delivers to S3 within 60–120 seconds (configurable buffer window). AWS Glue crawlers running on a schedule automatically detect new fields in the Parquet files and update the Glue Data Catalog schema — providing automatic schema evolution without manual intervention. Amazon Athena provides standard SQL querying on the S3 data via the Glue catalog. This is the fully managed, serverless pipeline with no infrastructure to maintain. Option B Amazon Kinesis Data Analytics with DynamoDB does not easily support the schema evolution requirement and PartiQL on DynamoDB has significant limitations compared to standard SQL for analytical queries. Option C Amazon SQS with Lambda writing to Aurora cannot sustain 500,000 events/second within Lambda concurrency limits (max 3,000 concurrent Lambdas by default) and Aurora write throughput limits. Option D Kafka on EKS and Redshift is a valid enterprise architecture but involves significant operational overhead compared to the fully managed Kinesis/Firehose/S3/Athena stack.

Question 3

A machine learning engineer needs to create a feature store for a recommendation system. Features are computed from user behavior data stored in Amazon S3 (updated daily) and from real-time clickstream events. The recommendation model requires both historical aggregated features (computed offline) and real-time features (last 5 minutes of activity) to be served together at inference time with sub-10ms latency. The feature serving layer must handle 50,000 requests per second. Which architecture MOST effectively meets all requirements?

Accepted Answer

Amazon SageMaker Feature Store is the purpose-built AWS service for ML feature management. It provides a dual-store architecture: an offline store (S3-backed) for training data retrieval and an online store (DynamoDB-backed) for low-latency inference serving. GetRecord calls on the online store return in single-digit milliseconds and the online store scales to handle high throughput. Both batch-computed and real-time features can be written to the same feature group, and the online store always serves the latest value per record. This eliminates the need for a custom feature serving layer. Option B requires building and maintaining a custom microservice and uses S3 Select for batch features, which has higher latency than DynamoDB and would exceed the 10ms SLA. Option C using DynamoDB directly requires building a custom schema for feature groups and lacks the Feature Store's training data management, point-in-time correctness, and lineage tracking. Option D using Kinesis Data Streams for inference feature serving is architecturally incorrect — reading from Kinesis at inference time does not provide the per-user latest feature value pattern needed for recommendation systems.

Question 4

A company has 5 years of historical sales data stored across multiple data sources: an Amazon RDS MySQL database, several Amazon S3 CSV files, and an on-premises Oracle database. A data scientist needs to consolidate all this data, apply transformations (join tables, normalize values, remove duplicates), and load the result into Amazon S3 in Parquet format for ML training. The team has limited ETL engineering resources and needs a solution that minimizes coding effort. Which service BEST meets these requirements?

Accepted Answer

AWS Glue Studio provides a visual, no-code/low-code ETL development environment that minimizes coding effort. It natively connects to Amazon S3, Amazon RDS (via JDBC), and on-premises databases (via JDBC over Direct Connect or VPN through the Glue virtual private endpoints). The visual transform nodes handle joins, deduplication, and normalization without writing code, and the output can be configured to write directly to S3 in Parquet format. This is the LEAST coding-intensive option. Option A Lambda has a 15-minute execution timeout and memory limits that make it unsuitable for processing multi-year historical data volumes, and requires writing all transformation logic in Python. Option C EMR with PySpark is powerful but requires writing Spark code and managing a cluster, adding both coding effort and operational overhead. Option D Redshift cannot directly query the on-premises Oracle database without complex federated query setup and does not address the on-premises source requirement as cleanly as Glue.

Question 5

A machine learning engineer is building a data pipeline that processes medical imaging data (DICOM files) stored in Amazon S3. Each DICOM file is 50–200 MB, and the pipeline must process 100,000 files per day. The processing steps include format conversion, image normalization, and feature extraction using a custom Python script that takes 2–5 minutes per file. The pipeline must be scalable, fault-tolerant, and resume processing from where it left off if interrupted. Which architecture BEST meets these requirements?

Accepted Answer

The SQS + ECS Fargate pattern is BEST for this workload: SQS provides durable, fault-tolerant message queuing with built-in visibility timeout for in-progress work (so interrupted tasks are automatically requeued), dead-letter queues for permanent failures, and the ability to scale ECS tasks based on queue depth. ECS Fargate auto-scaling can spin up hundreds of parallel tasks. Processing 2–5 minutes per file with 100,000 files/day requires high parallelism, which this architecture provides without limits. Option A SageMaker Processing Jobs are appropriate but do not natively 'resume from where left off' — if the job fails, all 100,000 files must be reprocessed or the engineer must manually manage which files were completed. Option C Lambda is limited to 15-minute execution timeout; a 5-minute file processing time is within limits, but 100 reserved concurrency allows only 100 parallel executions and Lambda does not provide a 'resume from where left off' mechanism if the function deployment is interrupted. Option D Step Functions Map state with 100,000 concurrent iterations approaches Step Functions limits (100,000 concurrent executions) and Lambda payload size limits for large file keys.

Question 6

A data scientist is preparing a dataset for a binary classification model. The target variable has 95% negative examples and 5% positive examples. After training an initial logistic regression model, the model achieves 95% accuracy but the recall for the positive class is only 12%, meaning the model almost never correctly identifies positive examples. What is the MOST likely cause of this problem and the BEST initial remediation?

Accepted Answer

The 95% accuracy with 12% positive-class recall is a textbook symptom of class imbalance where the model learns to predict the majority class for all examples. A model that predicts all-negative on a 95/5 split achieves exactly 95% accuracy while completely missing the minority class. The correct remediation is to address the imbalance using class weighting (which adjusts the loss function to penalize misclassification of the minority class more heavily) or SMOTE oversampling, and to switch to a class-imbalance-aware metric like F1, AUC-PR, or recall. Option A L2 regularization addresses overfitting, not class imbalance — regularization would actually reduce model capacity and worsen recall on the minority class. Option C switching to XGBoost changes the model architecture but does not address the root cause; XGBoost on an imbalanced dataset will similarly fail without scale_pos_weight adjustment. Option D PCA dimensionality reduction does not address class imbalance.

Question 7

A machine learning engineer is analyzing a tabular dataset with 500 features for a regression task. Initial model training with all 500 features produces high training accuracy but poor test accuracy (large train-test gap). Feature importance analysis shows that 450 features have near-zero importance scores. The engineer needs to select the MOST relevant features to improve generalization. Which combination of techniques MOST effectively identifies the relevant feature subset? (Choose TWO.)

Accepted Answer

Recursive Feature Elimination (RFE) is a robust wrapper-based feature selection method that uses the model's own importance scores to iteratively prune features, validating each step with cross-validation — ensuring the selected subset genuinely improves test performance. Lasso (L1) regularization is a filter/embedded method that directly penalizes model complexity by driving irrelevant feature coefficients to zero during training, acting as automatic feature selection. Together, these two approaches address the problem from both a wrapper (RFE) and embedded (Lasso) perspective, providing complementary feature selection signals. Option B PCA creates new composite features (principal components) rather than selecting original features — interpretability is lost and it does not identify which original features are relevant. Option D Pearson correlation only captures linear relationships between individual features and the target, missing interaction effects; many important features may have low individual correlation but high predictive power in combination. Option E SHAP values are excellent for post-hoc feature importance analysis but SageMaker Clarify requires a deployed model and adds infrastructure overhead for what is essentially an EDA task.

Question 8

A data scientist is preparing training data for a customer churn prediction model. The dataset contains a 'last_purchase_date' column with 8% missing values, an 'account_age_days' column with 0.3% missing values, and a 'lifetime_value' column with 22% missing values. The churn label is 'is_churned'. The data scientist suspects that the missingness in 'lifetime_value' is correlated with churned customers (i.e., churned customers are less likely to have a recorded lifetime value). Which imputation strategy is MOST appropriate for each column?

Accepted Answer

This scenario describes three different missing data patterns requiring tailored strategies. For 'account_age_days' (0.3% missing, likely Missing Completely at Random), mean imputation introduces negligible bias. For 'last_purchase_date' (8% missing), date fields are typically right-skewed and median is more robust than mean. For 'lifetime_value' (22% missing, suspected Missing Not at Random — MNAR — because missingness correlates with the target variable), a missingness indicator feature is critical: it encodes the predictive signal that not having a lifetime value recorded is itself informative about churn. Simple imputation alone would destroy this signal. Option A mean imputation for all columns destroys the MNAR signal in 'lifetime_value' by imputing a plausible-looking value, causing the model to lose a predictive feature. Option C dropping rows with missing values under MNAR conditions introduces systematic bias — the remaining dataset would be underrepresented in churned customers, distorting the model. Option D KNN is computationally expensive and still cannot capture the MNAR signal without an indicator feature.

Question 9

A company has a dataset of 2 million e-commerce product listings for training a product categorization model. Each listing has a short text description (average 50 words) and a category label (500 distinct categories). The data scientist notices that the 'description' field contains significant noise: HTML tags, Unicode control characters, product codes in mixed formats (e.g., 'SKU-12345', 'SKU12345', '#12345'), and brand names in various capitalizations. The data scientist needs to normalize text to improve model performance. Which preprocessing steps should be applied to the 'description' field before tokenization? (Choose TWO.)

Accepted Answer

HTML tag removal and lowercasing (Option A) are fundamental text cleaning steps: HTML tags are structural noise that adds no semantic value to product descriptions, and case normalization significantly reduces vocabulary size by merging case variants of the same word — critical for a 2M-record dataset where vocabulary size directly impacts model size and training time. Product code normalization (Option C) is especially important here because each unique SKU ('SKU-12345', 'SKU-67890') would otherwise appear as a unique token that occurs only once, inflating vocabulary with rare tokens that provide no generalizable signal. Replacing them with '[PRODUCT_CODE]' teaches the model that these patterns are all equivalent references to a product identifier. Option B stemming is a blunt tool that can distort meaning ('universe' → 'univers') and is generally less effective than lemmatization for classification tasks; modern tokenizers handle morphological variants better. Option D stop word removal can hurt performance on short texts (50 words) where every word carries informational value, and is less critical for deep learning models. Option E BPE tokenizers do not inherently handle HTML tags cleanly and would tokenize HTML attributes as content tokens, adding noise.

Question 10

A data scientist is performing EDA on a dataset for a time-series forecasting model. The dataset contains hourly electricity consumption readings for 1,000 meters over 3 years. The data scientist needs to visualize and quantify seasonal patterns at multiple time scales (daily, weekly, and annual cycles) and identify any trend components. Which analytical approach is MOST appropriate for this decomposition?

Accepted Answer

STL (Seasonal-Trend decomposition using Loess) is the standard technique for decomposing time series into trend, seasonal, and residual components. It is robust to outliers and handles multiple seasonality periods by being applied iteratively at each time scale (e.g., daily period of 24, weekly period of 168, annual period of 8,760 hours). The decomposition provides not just visualization but also quantitative component estimates that can be used directly as features for forecasting models. Option A the Pearson correlation matrix reveals autocorrelation but does not decompose the signal into interpretable trend and seasonal components. Option C PCA on a time series treats time steps as independent dimensions and does not preserve temporal ordering or extract seasonal components with interpretable seasonality periods. Option D aggregating means by hour/day/month provides a useful visualization but does not separate the overlapping contributions of multiple seasonal cycles or isolate the trend component — all three seasonal cycles are confounded in the aggregated means.

Question 11

A machine learning engineer is preparing features for a fraud detection model. The dataset contains a 'transaction_amount' feature with a distribution that is heavily right-skewed (mean: $250, median: $45, 99th percentile: $8,000, maximum: $450,000). There are also 0.1% of values that are negative (data entry errors). The model is a gradient boosting classifier. Which preprocessing steps are MOST appropriate for the 'transaction_amount' feature?

Accepted Answer

Gradient boosting tree-based models (XGBoost, LightGBM, CatBoost) are indeed scale-invariant and distribution-agnostic because they split features based on rank order, not absolute values. A log transform or normalization does not improve tree-based model performance — the splits and resulting predictions are identical for monotonic transformations. The correct action for the 0.1% negative values (known data entry errors) is imputation to 0 or the median, since negative transaction amounts are logically invalid and represent noise. Option A log1p transformation is valuable for linear models and neural networks but provides no benefit for gradient boosting, adding unnecessary preprocessing complexity. Option B min-max normalization is also unnecessary for tree-based models and is actually harmful if the max value ($450,000) changes between train and test sets, causing out-of-range scaled values. Option D z-score standardization similarly provides no benefit for gradient boosting and the claim that high z-scores 'make outliers easily detectable' misrepresents how tree-based models use features — the model can identify high-value transactions via tree splits without any normalization.

Question 12

A machine learning engineer is training an image classification model on Amazon SageMaker using the built-in Image Classification algorithm (ResNet-50) on a dataset of 500,000 images across 200 classes. The training job on a single ml.p3.2xlarge instance (1 GPU) is estimated to take 72 hours. The team needs to reduce training time to under 8 hours while maintaining model accuracy. Which SageMaker training configuration MOST effectively achieves this?

Accepted Answer

SageMaker distributed data parallelism across multiple instances is the correct approach for reducing training time at this scale. With 9 ml.p3.2xlarge instances (each with 1 GPU), the distributed training library distributes mini-batches across all 9 GPUs simultaneously, reducing epoch time by ~9x: 72 / 9 = 8 hours. The linear learning rate scaling rule (multiply base LR by the number of GPUs) compensates for the effectively larger global batch size to maintain convergence speed and accuracy. Option A switching to ml.p3.16xlarge (8 GPUs) achieves ~8x speedup to about 9 hours — close but not guaranteed to be under 8 hours, and single-machine multi-GPU training has diminishing returns due to GPU interconnect overhead. Option C Spot training reduces cost but not wall-clock time — interruptions can actually extend total time. Option D mixed-precision training provides a 2–4x speedup (reducing 72 hours to 18–36 hours), which does not meet the 8-hour target.

Question 13

A machine learning engineer is training an XGBoost model on Amazon SageMaker for a regression task. The training loss decreases steadily but the validation loss starts increasing after epoch 50, while training loss continues to decrease until epoch 200. The team has already applied L1 and L2 regularization (alpha=0.1, lambda=1.0) with no improvement. Which hyperparameter changes should the engineer try NEXT to address this overfitting? (Choose TWO.)

Accepted Answer

Both max_depth reduction and min_child_weight increase are direct tree complexity controls in XGBoost that address overfitting at the structural level. 'max_depth' (currently 10) controls how deep each individual tree can grow — deep trees can memorize training patterns; reducing to 3–6 is the standard XGBoost recommendation. 'min_child_weight' sets the minimum sum of instance weights in a leaf; increasing it prevents the model from creating splits that capture only a few training examples (a direct overfitting cause), especially when the divergence starts at epoch 50 (indicating early complex splits). Option B increasing n_estimators makes overfitting worse — the train-validation gap is already widening, and more rounds will push training loss lower while validation loss continues to rise. Option C subsample regularization is valid but subsampling alone is less directly targeted at the observed max_depth-driven overfitting than structural tree constraints. Option D reducing learning rate with the same n_estimators just means the model takes longer to reach the same overfit state; to use a lower learning rate effectively, n_estimators would need to be increased substantially, which risks more overfitting. Options A and E are the most direct structural fixes.

Question 14

A machine learning engineer needs to select the best machine learning algorithm for a business problem where a new customer support ticket must be automatically routed to one of 50 specialist teams based on the ticket's text content. The dataset has 500,000 labeled tickets. The model must return the team assignment within 100ms at inference time. Which algorithm family is MOST appropriate for this problem?

Accepted Answer

Amazon SageMaker's BlazingText in supervised text classification mode implements the FastText algorithm, which is specifically designed for multi-class text classification with large vocabularies. It trains an order of magnitude faster than deep learning models, handles 500,000 training examples efficiently, and inference is sub-millisecond — far within the 100ms SLA. For a 50-class routing task on ticket text, BlazingText achieves competitive accuracy with much lower latency and cost than transformer models. Option A K-Means is unsupervised clustering — it cannot use the labeled ticket data for learning team-specific routing patterns; cluster centroids do not correspond to known specialist teams without additional mapping. Option C fine-tuned BERT large is likely the highest accuracy option but BERT large inference takes 200–500ms on CPU (exceeding the 100ms SLA) unless GPU inference is used, which significantly increases cost. Option D Linear Learner with TF-IDF is a valid baseline but FastText (BlazingText) consistently outperforms bag-of-words linear models on text classification because it considers subword n-gram features.

Question 15

A machine learning engineer is using Amazon SageMaker Automatic Model Tuning (hyperparameter optimization) to tune an XGBoost model. The tuning job runs 100 trials sequentially and each trial takes 15 minutes, resulting in a 25-hour total tuning time. The engineer needs to reduce the total tuning time to under 4 hours without significantly degrading the quality of the optimal hyperparameters found. Which configuration changes MOST effectively achieve this? (Choose TWO.)

Accepted Answer

Increasing MaxParallelJobs to 30 (Option B) reduces wall-clock time from 25 hours to approximately 1 hour (ceil(100/30) × 15 min) — meeting the 4-hour target. While high parallelism reduces Bayesian optimization efficiency, the overall reduction in time with 100 trials still allows the optimizer to explore the space effectively. Enabling early stopping (Option D) reduces average trial duration by terminating poor-performing trials early (e.g., stopping at epoch 20 of 100 if the metric is clearly not converging), reducing each trial's average wall time from 15 to potentially 5–7 minutes. Combined with parallelism, this further compresses total time. Option A reduces trials to 20 but still takes 5 hours sequentially (exceeding the 4-hour target), and 20 trials may be insufficient for a complex XGBoost hyperparameter space. Option C Random search cannot guarantee that all 100 trials run simultaneously — SageMaker still limits parallel jobs, and Random search generally requires more trials than Bayesian to achieve equivalent quality. Option E is a legitimate optimization strategy but requires running two tuning jobs and adds time for the pilot analysis phase, making it less direct than parallelism and early stopping.

Question 16

A machine learning engineer needs to build a model that can detect anomalies in time-series network traffic data from 10,000 IoT devices. Each device produces 1 reading per second. There are no labeled anomaly examples available — the team does not know which historical readings are anomalies. The model must identify both point anomalies (single spike values) and contextual anomalies (values normal in isolation but abnormal given recent history). Which Amazon SageMaker built-in algorithm is MOST appropriate?

Accepted Answer

Amazon SageMaker's built-in Random Cut Forest (RCF) algorithm is specifically designed for unsupervised anomaly detection in time-series data. The 'shingle size' hyperparameter creates sliding windows of recent readings, enabling RCF to detect contextual anomalies (where a single value looks normal in isolation but is anomalous given the preceding readings). RCF is designed to handle streaming data at IoT scale and outputs a continuous anomaly score for each reading. No labeled data is required. Option A IP Insights is designed for detecting anomalous IP address/entity relationships in network access logs, not for numeric sensor reading anomaly detection. Option C K-Means can be used for anomaly detection via distance-to-centroid but requires selecting K, assumes spherical clusters, and does not natively handle temporal context for contextual anomalies. Option D DeepAR is an excellent forecasting algorithm, and using prediction intervals for anomaly detection is a valid approach, but it requires sufficient historical data per device to learn individual device patterns, requires inference on each new point, and is significantly more complex and expensive to operate at 10,000 devices × 1 reading/second compared to RCF.

Question 17

A data scientist is building a recommendation system for a media streaming platform with 5 million users and 500,000 content items. The platform has 3 years of user-content interaction history (play, pause, skip, like events). The team wants to use collaborative filtering to generate personalized recommendations. Which Amazon SageMaker built-in algorithm MOST directly supports this use case?

Accepted Answer

Amazon SageMaker's Factorization Machines (FM) algorithm is the canonical choice for recommendation systems based on collaborative filtering. FMs are specifically designed to work with sparse, high-cardinality user-item interaction matrices (5M users × 500K items fits this profile). They learn latent factor embeddings for users and items, capture second-order feature interactions, and predict interaction scores that can be ranked for recommendations. FMs are natively supported in SageMaker with optimized sparse data handling. Option B BlazingText Word2Vec is a sentence/word embedding method; adapting it for recommendation requires significant preprocessing to convert interaction histories to sequences, and the resulting item2vec approach, while valid, is not a native SageMaker recommendation algorithm. Option C PCA is a dimensionality reduction technique, not a recommendation algorithm; using PCA cosine similarity for recommendations is a baseline approach that does not learn interaction patterns. Option D KNN recommendation is memory-based collaborative filtering that requires computing similarities across all 5M users at inference time, which is computationally infeasible at this scale.

Question 18

A machine learning engineer is training a deep learning model for medical image segmentation on Amazon SageMaker. The model requires 40 GB of GPU memory during training but all available SageMaker instance types with a single GPU have at most 16 GB of GPU memory (ml.p3.2xlarge). The engineer needs to train the model with the full memory requirement. Which training strategy MOST effectively addresses the GPU memory constraint without sacrificing model architecture?

Accepted Answer

SageMaker Model Parallelism (SMP) is the correct solution when the model itself is too large to fit on a single GPU. SMP partitions the model's layers across multiple GPUs on multiple instances, allowing a 40 GB model to be distributed across, for example, three ml.p3.2xlarge instances with 16 GB each. Pipeline parallelism with micro-batching maintains GPU utilization during model-parallel training. This is the purpose-built AWS solution for this constraint. Option B data parallelism replicates the FULL model on each GPU — since the model is 40 GB and each GPU has only 16 GB, the model cannot fit on any single GPU with data parallelism, making this approach impossible. Option C reducing batch size to 1 reduces activation memory (which scales with batch size) but does not reduce model weight memory (which is fixed). If the model weights alone exceed 16 GB, batch size 1 still cannot fit the model. Option D gradient checkpointing reduces activation memory, not model weight memory. For a 40 GB memory requirement that includes large model weights, gradient checkpointing alone is unlikely to bring it below 16 GB.

Question 19

A machine learning engineer trained a binary classification model using Amazon SageMaker that predicts whether a loan application should be approved. The business stakeholder requires that the model's false negative rate (FNR) — approving applications that should have been rejected — not exceed 5%, as these represent financial losses. The current model at the default 0.5 threshold has FNR = 12% and FPR = 8%. Which action MOST directly reduces the false negative rate to meet the 5% business requirement?

Accepted Answer

The false negative rate (FNR = FN / (FN + TP)) is the proportion of actual positives (applications that should be rejected) that the model incorrectly classifies as negative (approved). Lowering the classification threshold from 0.5 to a lower value (e.g., 0.3) causes the model to predict 'reject' for more borderline cases, increasing true positives and reducing false negatives, thereby decreasing FNR. The ROC curve allows the engineer to find the exact threshold where FNR ≤ 5% while quantifying the trade-off with FPR. Threshold adjustment is the fastest and most direct way to meet a specific FNR requirement without retraining. Option A retraining with class weights is valid but requires a new training run; threshold adjustment achieves the same FNR reduction on the existing model without retraining. Option C increasing the threshold to 0.7 makes the model less likely to predict 'reject', which would increase (not decrease) false negatives, worsening FNR. Option D optimizing for F1 balances precision and recall, but the business requirement is specifically FNR ≤ 5%, not balanced F1; F1 optimization does not directly constrain FNR to a target value.

Question 20

A machine learning engineer is evaluating two models for a customer churn prediction task. Model A has AUC-ROC = 0.91 and AUC-PR = 0.42. Model B has AUC-ROC = 0.88 and AUC-PR = 0.67. The dataset has 2% positive (churned) examples and 98% negative. Which model should the engineer choose and why?

Accepted Answer

On severely class-imbalanced datasets (2% positive), AUC-ROC is a misleading metric because it incorporates the True Negative Rate (Specificity) in its calculation, and with 98% negatives, even a weak model can achieve high specificity (correctly classifying negatives), inflating the AUC-ROC. AUC-PR, by contrast, focuses entirely on precision (among predicted positives, how many are actually positive) and recall (among actual positives, how many were predicted positive) — both of which are directly relevant to detecting the minority churned class. Model B's AUC-PR of 0.67 vs Model A's 0.42 indicates that Model B substantially outperforms Model A in identifying actual churned customers, which is the business objective. Option A is incorrect because AUC-ROC can be inflated by high true negative rates on imbalanced data. Option C is incorrect — false positive and false negative costs in churn prediction are clearly asymmetric (missing a churner is more costly than falsely flagging a non-churner) and AUC-PR is preferred regardless. Option D is incorrect — AUC-ROC and AUC-PR are complementary metrics; AUC-ROC is appropriate for balanced datasets, and AUC-PR is preferred for imbalanced datasets.

Question 21

A machine learning engineer has trained a fraud detection model on Amazon SageMaker and needs to deploy it for real-time inference. The model must respond to inference requests within 50ms at the 99th percentile. The expected inference volume is 1,000 requests per second during business hours and drops to fewer than 10 requests per second overnight. The company wants to minimize infrastructure costs. Which deployment configuration MOST cost-effectively meets the latency and throughput requirements?

Accepted Answer

An Amazon SageMaker real-time endpoint with auto scaling is the correct choice for a latency-sensitive (50ms p99) workload with variable traffic patterns. Auto scaling adjusts instance count based on InvocationsPerInstance, scaling up during the 1,000 RPS peak and scaling down to 1 instance overnight (reducing cost by ~97% during low-traffic hours). Real-time endpoints with ml.c5 or ml.c6i instances easily achieve sub-50ms latency for typical tabular fraud detection models. Option B SageMaker Serverless Inference has cold start latency of 1–5 seconds for the first request after a cold period, which would violate the 50ms p99 latency requirement during overnight periods when the endpoint might be cold. Option C Lambda has a maximum execution memory of 10 GB and loading large ML model artifacts into Lambda is complex; more importantly, Lambda has cold starts and is generally not recommended for latency-sensitive ML inference at 1,000 RPS due to initialization overhead. Option D Batch Transform is designed for offline bulk inference, not real-time per-request inference — it introduces minutes of latency, completely violating the 50ms requirement.

Question 22

A company deployed a credit scoring model to an Amazon SageMaker endpoint three months ago. Model performance metrics from production indicate that the model's Gini coefficient has degraded from 0.72 at deployment to 0.54 over the past month. The data science team suspects data drift — the distribution of input features in production differs from the training data. Which sequence of actions MOST effectively diagnoses and addresses the drift?

Accepted Answer

Amazon SageMaker Model Monitor is the purpose-built service for detecting data drift in production ML endpoints. It captures a sample of inference requests, computes feature statistics, and compares them against baseline statistics computed from the training data — detecting shifts in mean, standard deviation, distribution, and feature correlations. The resulting violation reports identify exactly which features have drifted and by how much, enabling targeted investigation (e.g., is the drift in the applicant income feature because of economic changes, or a data pipeline bug?). Once the root cause is confirmed, retraining with current-distribution data via SageMaker Pipelines automates the remediation. Option B rolling back the model does not address the root cause (the world has changed, not the model), and manual statistical comparison is ad hoc and slow. Option C SageMaker Clarify bias monitoring detects demographic bias in predictions, not general data distribution drift — these are related but distinct concepts, and Clarify alone cannot diagnose feature-level distribution drift. Option D blindly adding 3 months of recent data without first diagnosing whether the drift is due to legitimate distribution change or a data pipeline bug could reinforce errors rather than improve the model.

Question 23

A company needs to deploy two versions of a recommendation model simultaneously: a new challenger model and the existing champion model. The company wants to route 10% of production traffic to the challenger model to evaluate its performance under real conditions while the champion model serves 90% of traffic. Both models must share the same endpoint URL so that the client application does not need to change. Which Amazon SageMaker feature enables this configuration?

Accepted Answer

Amazon SageMaker Production Variants is the native SageMaker feature for A/B testing and canary deployments at the endpoint level. By configuring multiple variants on a single endpoint with different InitialVariantWeight values (9 for champion, 1 for challenger), SageMaker automatically routes traffic in the specified ratio from the same endpoint URL. The client application sends all requests to the same endpoint and SageMaker handles traffic splitting transparently. CloudWatch metrics are automatically separated by variant name, enabling independent performance comparison. Option A using API Gateway + Lambda achieves the same traffic splitting but adds two additional managed services to maintain, increases latency with the Lambda hop, and requires custom routing logic — this is unnecessary given SageMaker's native variant weights. Option C multi-model endpoints share a single container/model server and load models on demand based on the TargetModel parameter; this requires client-side routing logic (violating 'the client application does not need to change') and is designed for hosting many models efficiently, not A/B testing. Option D inference pipelines chain models in sequence for feature transformation, not for parallel A/B testing.

Question 24

A machine learning engineer needs to run a batch inference job on 10 million customer records stored in Amazon S3 to generate churn probability scores. The inference job is expected to take 4 hours and does not require real-time responses. The model artifact is stored in Amazon S3. The team wants to avoid managing any persistent inference infrastructure. Which Amazon SageMaker feature is MOST appropriate for this use case?

Accepted Answer

Amazon SageMaker Batch Transform is the purpose-built feature for offline batch inference on large datasets. It reads input data from S3, distributes the inference workload across multiple instances (configurable via InstanceCount), writes predictions to an S3 output path, and automatically terminates all instances upon completion — no persistent infrastructure to manage. This is the canonical solution for bulk ML scoring without real-time latency requirements. Option A using a real-time endpoint for 10 million records requires maintaining a persistent endpoint, managing client-side parallelism and retry logic, and is architecturally inappropriate for batch workloads. Option C Asynchronous Inference is designed for single large payload requests (e.g., one long document or large image) that take too long for synchronous response, not for bulk batch scoring of millions of records. Option D a Processing Job with a custom script is viable but requires writing model loading and inference code manually rather than using the pre-built Batch Transform framework that handles serialization, parallelism, and result aggregation natively.

Question 25

A company needs to build a fully automated ML pipeline that retrains and redeploys a product recommendation model whenever new training data lands in Amazon S3. The pipeline must include data validation (reject batches with more than 5% missing values), model training, automatic model evaluation (only deploy if new model AUC > current production model AUC), and endpoint update with zero downtime. The pipeline must be reproducible and auditable, with all run artifacts versioned and traceable. Which architecture MOST effectively implements all requirements?

Accepted Answer

Amazon SageMaker Pipelines is the purpose-built, native MLOps orchestration service for exactly this pattern. It provides: (1) a DAG-based pipeline definition for all ML workflow steps, (2) built-in Condition steps that implement the 'only deploy if new model AUC > current AUC' logic natively, (3) SageMaker Model Registry for model versioning with approval workflows, (4) automatic experiment tracking via SageMaker Experiments for full reproducibility and auditability, and (5) native integration with EventBridge for event-driven triggers. All pipeline runs are stored with full artifact lineage. Zero-downtime endpoint updates are achieved by using UpdateEndpoint (rolling update) in the Lambda step. Option B Step Functions + Lambda achieves the same orchestration but requires building all ML-specific logic (training job polling, experiment tracking, model comparison) from scratch, with much higher code complexity and no native ML artifact tracking. Option C CodePipeline is designed for CI/CD of application code, not ML pipelines — it lacks native SageMaker step types, experiment tracking, and model registry integration. Option D Airflow (MWAA) is a general-purpose workflow engine that can orchestrate SageMaker, but it adds operational overhead for a use case that SageMaker Pipelines solves natively and with more ML-specific features.

MLS-C01 — AWS Certified Machine Learning – Specialty