Microsoft Certified: Machine Learning Operations Engineer Associate — Practice Set 1

Question 1

A machine learning team wants to provision an Azure Machine Learning workspace and all associated resources (Storage Account, Key Vault, Application Insights, Container Registry) in a repeatable, version-controlled way. Which approach aligns with Infrastructure as Code (IaC) best practices?

Accepted Answer

IaC with Bicep templates defines Azure resources declaratively and deploys them consistently via Azure CLI, which can be version-controlled in Git and executed automatically in a GitHub Actions pipeline. Manual portal clicks (A) are not repeatable or version-controlled. Cost Management (C) manages budgets but does not provision resources. Individual manual workspaces (D) lead to configuration drift and no audit trail.

Question 2

A data science team needs to share pre-processed datasets and registered model artefacts across multiple Azure Machine Learning workspaces in different business units. What is the recommended Azure Machine Learning feature for this?

Accepted Answer

Azure Machine Learning registries are purpose-built for cross-workspace sharing of assets such as datasets, environments, components, and registered models, with versioning and lineage tracking. Manual AzCopy (A) has no versioning or lineage. A public blob container (C) exposes sensitive artefacts without access control. Retraining from scratch in each workspace (D) is wasteful and produces inconsistent model versions.

Question 3

An enterprise requires that the Azure Machine Learning workspace only be accessible from within the corporate virtual network and must not be reachable from the public internet. Which configuration should the MLOps engineer apply?

Accepted Answer

Disabling public network access and configuring a private endpoint routes all traffic through the Azure private network (VNet), ensuring the workspace is unreachable from the public internet while still accessible from within the corporate network. IP firewall rules (A) still leave the public endpoint exposed. An NSG on compute only (C) does not protect the workspace management plane. A resource lock (D) prevents modifications but not network access.

Question 4

A GitHub Actions workflow needs to submit training jobs to an Azure Machine Learning workspace without storing any credentials in the repository. Which mechanism should the engineer configure?

Accepted Answer

Azure Machine Learning's GitHub integration supports OIDC-based federated identity credentials, which allow GitHub Actions to exchange a short-lived GitHub token for an Azure access token — no secrets stored in the repo. Storing the primary key as a secret (A) is better than hardcoding (C) but still places a long-lived credential in GitHub. Hardcoding the connection string (C) is a severe security risk. Granting public read access (D) is completely insecure.

Question 5

A data scientist wants to track parameters, metrics, and artefacts from multiple training runs in Azure Machine Learning so that experiments can be compared. Which tool should they use to instrument their training scripts?

Accepted Answer

MLflow is the open-standard experiment tracking library natively supported by Azure Machine Learning as a tracking backend. Data scientists call mlflow.log_param(), mlflow.log_metric(), and mlflow.log_artifact() in their training scripts and can compare runs in the Azure Machine Learning Studio UI. Application Insights (A) is for application telemetry, not ML experiment tracking. DevOps Boards (C) track work items, not experiment metrics. Manual JSON files (D) have no structured comparison or visualisation.

Question 6

After training, a data scientist wants to register a model in Azure Machine Learning using MLflow so that it can be versioned and deployed later. Which MLflow function should they call at the end of the training script?

Accepted Answer

mlflow.<flavour>.log_model() (e.g., mlflow.sklearn.log_model(), mlflow.pytorch.log_model()) logs the model artefact and, when registered_model_name is specified, automatically registers it in the Azure Machine Learning model registry. start_run() (A) opens an experiment run context but does not register models. log_metric() (C) records scalar values, not model artefacts. end_run() (D) closes the run context but does not register artefacts on its own.

Question 7

Contoso Bank deploys a fraud detection model to a real-time managed online endpoint. The team wants to validate a new model version before routing all production traffic to it, minimising risk if the new model underperforms. Which deployment strategy should they use?

Accepted Answer

Progressive (blue/green or canary) rollout in Azure Machine Learning managed online endpoints allows traffic to be split between the existing deployment and the new one. The percentage routed to the new version is incrementally increased as confidence grows, with the ability to roll back instantly if metrics degrade. Deleting and redeploying (A) causes downtime and has no rollback. A batch endpoint (C) introduces latency and cannot handle real-time fraud detection. Local testing (D) is not a production deployment strategy.

Question 8

A deployed credit-scoring model begins producing unexpected predictions three months after release. The MLOps team suspects that the statistical distribution of incoming feature data has shifted compared to the training data. What should they monitor to confirm this hypothesis?

Accepted Answer

Data drift monitoring compares the statistical distribution of incoming production data to the baseline training data, detecting feature shifts that degrade model performance. CPU utilisation (A) measures compute load, not data quality. HTTP 200 counts (C) confirm successful requests but not prediction quality. Blob read latency (D) is an infrastructure metric unrelated to data distribution.

Question 9

An MLOps engineer is evaluating a newly registered model against Responsible AI principles before approving it for production deployment. Which TWO evaluation steps should be performed using Azure Machine Learning's Responsible AI dashboard? Choose 2.

Accepted Answer

Azure Machine Learning's Responsible AI dashboard includes error analysis (A), which reveals where the model fails disproportionately across cohorts, and model explainability / feature importance (B), which identifies the most influential features for model transparency and fairness assessment. HTTP response time (C) is an operational metric, not a Responsible AI evaluation. Archiving previous versions (D) is model lifecycle management. Deploying without evaluation (E) is contrary to responsible deployment practice.

Question 10

A data science team wants to train a very large deep learning model (billions of parameters) that does not fit in the memory of a single GPU. Which Azure Machine Learning capability should they use?

Accepted Answer

Azure Machine Learning supports distributed training across multi-GPU clusters using frameworks such as PyTorch Distributed, Horovod, or DeepSpeed, enabling model and data parallelism for very large models. CPU-only clusters (A) are too slow for deep learning at this scale. Local laptop training (C) is infeasible for billion-parameter models. AutoML (D) explores architectures but does not split a model across GPUs to fit larger models.

Question 11

An ML platform team wants to automate hyperparameter tuning for a training job in Azure Machine Learning to find the optimal learning rate and batch size combination without manually submitting dozens of runs. Which feature should they use?

Accepted Answer

Azure Machine Learning sweep jobs define a search space for hyperparameters (e.g., random, grid, or Bayesian sampling), submit parallel child runs, and apply early termination policies (e.g., Bandit, Median Stopping) to cancel poorly-performing runs — automating and accelerating hyperparameter search. Manual one-by-one submission (A) is tedious and serial. DevOps variable groups (C) store configuration but cannot orchestrate a multi-run hyperparameter search. A fixed learning rate (D) is rarely optimal and ignores task-specific tuning needs.

Question 12

A production model's performance monitoring dashboard triggers an alert showing that the model's precision has dropped below the accepted threshold over the past week. The team needs to automatically initiate retraining. Which Azure Machine Learning capability should be configured?

Accepted Answer

Azure Machine Learning monitoring supports configuring alert-triggered actions — when a metric (e.g., precision) crosses a threshold, an Azure Event Grid event or Logic App can automatically submit a retraining pipeline, ensuring the model is updated promptly. Fixed-schedule retraining (B) wastes compute when the model is performing well. Deleting the endpoint (C) stops all service, causing downtime. Adjusting the confidence threshold (D) manipulates the reported metric without improving the actual model.

Question 13

A GenAIOps engineer is setting up a Microsoft Foundry project for a generative AI application. Multiple teams need to access the same Azure OpenAI resource but with different permission levels (some read-only, others deployers). What should be configured to enforce this access control?

Accepted Answer

Azure RBAC with managed identities enables fine-grained, auditable access control — different roles (e.g., Cognitive Services User vs. Cognitive Services Contributor) can be assigned to different teams without sharing any credentials. Sharing a single API key (A) provides no granularity or auditability. A separate subscription per team (C) is excessive and creates governance fragmentation. A resource group lock (D) prevents modifications but does not differentiate user permissions.

Question 14

A company wants to deploy a foundation model via Microsoft Foundry for a customer-facing application that will serve thousands of requests per minute with consistent low latency SLAs. Which deployment option is most appropriate?

Accepted Answer

Provisioned throughput units (PTUs) reserve dedicated compute capacity for the deployed model, delivering predictable latency and throughput without competing with other tenants — essential for SLA-bound, high-volume workloads. Serverless API (A) is cost-efficient for variable or low-volume traffic but may throttle under sustained high load. Local laptop inference (C) cannot support thousands of concurrent users. Azure Batch (D) is for high-throughput offline batch workloads, not real-time customer-facing requests.

Question 15

A GenAIOps engineer needs to ensure that the prompts used by a production generative AI application can be versioned, reviewed by peers, and rolled back if a change causes regression in response quality. Which approach should they implement?

Accepted Answer

Storing prompt files in Git enables full version history, branching, code review via pull requests, and rollback to previous versions — the same software engineering practices applied to application code. Hardcoding prompts in binaries (A) requires a full app redeploy for every prompt tweak. Queue Storage (C) is for transient message passing, not versioned configuration. Key Vault (D) stores secrets and certificates; it is not designed for prompt versioning with diff and rollback.

Question 16

A team is comparing two prompt variants for a customer support chatbot to determine which generates more accurate and helpful responses. What does Microsoft Foundry provide to support this comparison?

Accepted Answer

Microsoft Foundry's prompt flow supports creating prompt variants and running comparative evaluations, enabling teams to view quality metrics (groundedness, relevance, coherence) side-by-side across variants before selecting the best one for production. DevOps Boards (A) track project tasks, not prompt quality metrics. A blob diff tool (C) shows text differences but not quality evaluation. The pricing calculator (D) estimates costs but not response quality.

Question 17

An MLOps team is deploying a GenAIOps infrastructure on Azure that must restrict all network traffic between Foundry resources to the private network. Which TWO steps are required? Choose 2.

Accepted Answer

Private endpoints (A) route all inter-service traffic through the Azure private network, preventing exposure to the public internet. Bicep templates deployed via Azure CLI (B) provide repeatable IaC provisioning that encodes the private networking configuration. Enabling public access (C) directly contradicts the network isolation requirement. Storing templates as PDFs (D) is not executable IaC. Sharing personal keys via email (E) violates credential management best practices.

Question 18

A team is selecting a foundation model to use for semantic search over internal engineering documentation. The documents are highly technical and contain specialised jargon. The team wants the model to produce embeddings that capture domain-specific meaning. What should they consider?

Accepted Answer

General-purpose embedding models may underperform on highly specialised technical vocabularies. Fine-tuning an embedding model on domain-specific data improves the quality of embeddings for that vocabulary, leading to better semantic search accuracy. Relying on a general model without tuning (A) is a valid starting point but suboptimal for highly specialised corpora. DALL-E (C) is an image generation model with no embedding capability. Model size alone (D) does not guarantee better domain-specific semantic accuracy.

Question 19

A GenAIOps engineer wants to measure whether a RAG-based chatbot's answers are grounded in the retrieved source documents — i.e., the model is not generating information beyond what appears in the retrieved context. Which AI quality metric should be evaluated?

Accepted Answer

Groundedness measures whether a model's response is supported by the retrieved source context, detecting hallucinations where the model introduces facts not present in the grounding documents. Coherence (A) measures logical flow and readability of the response. Fluency (C) measures grammatical and linguistic quality. Latency (D) is an operational metric, not a content quality metric.

Question 20

Fabrikam's legal team discovers that the company's public-facing AI assistant occasionally generates responses that include discriminatory language. The engineering team needs to detect and block such content before it reaches users. Which evaluation and safety configuration should they implement?

Accepted Answer

Microsoft Foundry's risk and safety evaluations detect harmful content categories (hate, self-harm, violence, sexual content) in both prompts and responses, and Azure AI Content Safety filters block content above configurable severity thresholds before it is returned to users. Temperature of 0.0 (A) makes outputs more deterministic but does not remove harmful content. Human monitoring (C) is impractical at scale and introduces unacceptable latency. Character limits (D) do not target harmful content specifically.

Question 21

A GenAIOps team wants to track the number of tokens consumed per request, overall throughput (requests per second), and p95 response latency for a deployed Azure OpenAI model. Where should they configure this observability?

Accepted Answer

Microsoft Foundry provides built-in continuous monitoring capabilities that expose operational metrics (latency, throughput, token consumption) in near real-time, with detailed tracing for debugging individual requests. Manual spreadsheet exports (A) are delayed and error-prone. Laptop load tests (C) measure client-side latency and cannot observe production at scale. Monthly billing pages (D) show aggregate costs, not granular real-time operational metrics.

Question 22

A GenAIOps engineer is setting up automated evaluation workflows for a generative AI application. Which TWO components should be included to make the evaluation comprehensive? Choose 2.

Accepted Answer

A comprehensive automated evaluation requires a representative test dataset with expected answers (A) — without this, metrics cannot be computed. Automated workflows using built-in and custom metrics (B) allow consistent, scalable, repeatable evaluation without human review for every run. Waiting for user complaints (C) is reactive and harmful to users. Manual-only review (D) is unscalable. Disabling logging (E) removes the data needed for evaluation.

Question 23

A RAG pipeline returns irrelevant documents from Azure AI Search, causing the language model to generate unhelpful answers. The team wants to improve retrieval precision. Which two optimisations should they evaluate first?

Accepted Answer

The two most impactful RAG retrieval tuning levers are the similarity threshold (filtering chunks below a relevance score) and chunk size (ensuring each retrieved segment contains enough coherent context). These directly improve precision and relevance of what the model receives. Increasing temperature (B) affects generation randomness, not retrieval quality. Removing the retrieval step (C) eliminates the grounding that makes RAG valuable. Switching to Blob Storage (D) removes the search index and makes retrieval impossible.

Question 24

A legal technology company builds a document search system using Azure AI Search. Users report that keyword matches on legal terms produce many irrelevant results when the exact words are not present but the legal concept is the same. What retrieval optimisation should the engineer implement to address this?

Accepted Answer

Hybrid search merges semantic vector search (which captures meaning and synonyms) with keyword BM25 search (which catches exact term matches), and Azure AI Search's Reciprocal Rank Fusion (RRF) combines their scores. This addresses the scenario where exact keywords are absent but the legal concept is the same. Returning more results (A) increases recall but does not improve the relevance ranking. Alphabetical sorting (C) is unrelated to semantic relevance. Smaller chunks (D) may lose context needed to understand legal concepts.

Question 25

A company has fine-tuned an Azure OpenAI model to generate responses in a specific customer service tone. After deployment, they notice the fine-tuned model's performance has degraded compared to the pre-production evaluation. Which monitoring and optimisation action is most appropriate?

Accepted Answer

A structured post-deployment lifecycle for fine-tuned models includes monitoring production performance metrics, diagnosing gaps (often addressed with synthetic or augmented training data to cover under-represented scenarios), validating improvements offline, and redeploying the improved model through the same GenAIOps pipeline. Reverting immediately (A) abandons the fine-tuning investment without diagnosing the root cause. Duplicating training examples (C) causes overfitting without adding new information. Increasing max_tokens (D) affects response length but does not fix a fine-tuning quality issue.

AI-300 — Machine Learning Operations Engineer Associate