Microsoft Certified: Azure Databricks Data Engineer Associate — Practice Set 1

Question 1

You need to run an Azure Databricks job that processes a large batch data transformation overnight. The job only runs for 2 hours and then terminates. You want to minimize cost by avoiding idle compute. Which compute type is best suited for this scenario?

Accepted Answer

Job compute (job clusters) in Azure Databricks are created when a job runs and terminated automatically when the job completes. They are optimized for batch workloads and minimize cost because they do not remain idle. All-purpose clusters are for interactive development and remain running (incurring cost) until manually terminated or auto-terminated. SQL warehouses are for SQL analytics queries. Shared clusters are for collaborative interactive work.

Question 2

You are configuring an Azure Databricks cluster for a production ETL job. You need the cluster to automatically add worker nodes when the workload increases and remove them when not needed, while keeping a minimum of 2 workers at all times. Which cluster configuration should you enable?

Accepted Answer

Autoscaling in Azure Databricks allows the cluster to automatically scale the number of worker nodes between a configured minimum and maximum based on current workload demand. Setting minimum workers to 2 ensures there are always at least 2 workers available. Autoscaling reduces cost during low-demand periods and prevents job failures during spikes. Auto-termination shuts down idle clusters but does not scale. Cluster pooling reduces startup time. Fixed size does not adapt to load changes.

Question 3

Your data engineering team uses Azure Databricks with Unity Catalog. You need to create a logical namespace under a catalog to organize tables and views for the marketing department's data. Which Unity Catalog object should you create?

Accepted Answer

In Unity Catalog's three-level namespace (catalog.schema.table), a schema (also called a database) is the second level that groups related tables, views, functions, and volumes within a catalog. You create a schema under a catalog to organize objects for a specific team or domain (e.g., marketing). Volumes store unstructured files within Unity Catalog. External locations and storage credentials configure access to cloud storage.

Question 4

Your organization wants to expose a large external dataset stored in Azure Data Lake Storage Gen2 that is managed by another organization's Azure Databricks workspace. You need to access this data in your workspace without copying it. Which Azure Databricks Unity Catalog feature enables read access to an external catalog in another Databricks workspace?

Accepted Answer

A foreign catalog in Unity Catalog is implemented using a Delta Sharing connection to a remote Databricks workspace. Delta Sharing is an open protocol for sharing live data across organizations. The sharing provider creates a share in their workspace; the recipient creates a foreign catalog that references the shared data. This enables cross-workspace and cross-organization data access without data duplication. External tables access data directly but require storage credentials. Mount points are legacy DBFS-based access.

Question 5

You need to grant a group of data analysts SELECT access on all tables in a specific schema in Unity Catalog. Which Unity Catalog statement correctly grants this privilege?

Accepted Answer

In Unity Catalog, the correct SQL syntax to grant SELECT on all current and future tables in a schema is: GRANT SELECT ON ALL TABLES IN SCHEMA catalog.schema TO <principal>. The principal can be a user, group, or service principal. Unity Catalog uses a standard GRANT/REVOKE SQL privilege model with three-level namespace. The SCHEMA-level grant in option A grants on the schema object itself, not the tables within it.

Question 6

You are implementing data governance for a Unity Catalog table that contains PII (personally identifiable information). Which two Unity Catalog features should you configure to restrict access to sensitive columns and rows? Choose 2.

Accepted Answer

Column masks (A) in Unity Catalog allow you to define SQL expressions that return a masked or redacted value for sensitive columns based on the querying user's group memberships or attributes (e.g., return NULL for non-authorized users). Row filters (B) define filter predicates that restrict which rows a user can see, enabling use cases like restricting analysts to only see data from their own region. OPTIMIZE, VACUUM, and Z-ordering are performance and maintenance operations, not security controls.

Question 7

Your organization needs to track who accessed what data, when, and from which notebook in Azure Databricks. This information is required for a security audit. Which Unity Catalog feature should you configure to capture this information?

Accepted Answer

Unity Catalog audit logging captures detailed audit events for data access operations (SELECT, INSERT, DELETE, GRANT, REVOKE) including the user identity, timestamp, object accessed, and the notebook or job that initiated the access. Audit logs are delivered to Azure Monitor or Azure Storage. Delta transaction log (DESCRIBE HISTORY) tracks data changes to a table but not user access events. Cluster and Spark event logs track compute-level events, not data access.

Question 8

You are designing a data sharing strategy for your Azure Databricks workspace. Your organization needs to share live Delta tables with a partner organization's Databricks workspace without copying the data. The shared data must reflect updates in real time. Which Unity Catalog feature should you implement?

Accepted Answer

Delta Sharing is an open protocol built into Unity Catalog for securely sharing live Delta tables with external organizations. You create a share object containing the tables to share and a recipient object representing the partner organization. The recipient accesses the data via the Delta Sharing protocol without copying it, and updates are reflected in real time. DBFS mount points and ADF copy activities create data copies. External tables require direct storage access credentials.

Question 9

You are ingesting data from an Azure Data Lake Storage Gen2 folder that receives new CSV files every hour. You need a scalable ingestion method that automatically detects new files and processes them incrementally. Which Azure Databricks ingestion feature should you use?

Accepted Answer

Auto Loader (accessed via cloudFiles source in Spark Structured Streaming / Lakeflow Spark Declarative Pipelines) incrementally ingests new files from cloud storage by tracking which files have already been processed using a checkpoint mechanism. It automatically discovers new files added to the source path and scales efficiently to millions of files without listing the entire directory each time. COPY INTO also does incremental ingestion but is less scalable for large file volumes. ADF copy and manual triggers require orchestration and do not track file state automatically.

Question 10

You are designing a data lakehouse in Azure Databricks with Unity Catalog. Your source data contains historical records and you need to track changes over time (Type 2 slowly changing dimension). When a customer's address changes, you need to keep the old record with an end date and insert a new active record. Which SCD type are you implementing?

Accepted Answer

SCD Type 2 (Slowly Changing Dimension Type 2) preserves the full history of changes by adding a new row for each change with start and end date columns. The old record is marked as inactive (end date set), and the new record is marked as current (end date NULL or far future). This approach allows accurate historical analysis. SCD Type 1 overwrites the old value losing history. SCD Type 3 only stores current and one previous value. SCD Type 0 never changes.

Question 11

You are loading data into a Unity Catalog Delta table. You need to update existing rows that match on a key column and insert new rows that do not exist. Which SQL DML operation achieves this in a single statement?

Accepted Answer

MERGE INTO (also called upsert) is a DML statement that combines INSERT and UPDATE operations in a single atomic statement. It matches rows between the source and target based on a join condition, updates matching rows, and inserts non-matching rows. MERGE also supports DELETE for matched rows. INSERT INTO adds new rows without updating existing ones. CTAS creates a new table from a query result. INSERT OVERWRITE replaces all rows in a partition.

Question 12

You are optimizing a large Delta table in Azure Databricks that contains billions of rows and is frequently queried by date range filters. The table has many small files due to frequent streaming writes. Which two operations should you run to consolidate small files and improve query performance?

Accepted Answer

OPTIMIZE consolidates small Delta table files into larger files (typically 1 GB target size), reducing the number of files that need to be read for queries. ZORDER BY co-locates related data in the same files based on the specified column (e.g., date), enabling data skipping to skip entire files that don't contain matching date ranges. VACUUM removes old file versions but doesn't consolidate files. CACHE TABLE caches data in Spark memory for a session. Dropping and recreating is destructive.

Question 13

You are implementing data quality checks in a Lakeflow Spark Declarative Pipeline. You need to enforce that the order_id column is never NULL and that the order_amount is always greater than 0. Which two approaches can you use in Lakeflow Spark Declarative Pipelines? Choose 2.

Accepted Answer

In Lakeflow Spark Declarative Pipelines, (A) pipeline expectations using CONSTRAINT ... EXPECT syntax allow you to define data quality rules on streaming or batch tables. You can configure the action on failure: warn (log and continue), drop (drop failing rows), or fail (stop the pipeline). (B) NOT NULL constraints in the table DDL also enforce nullability at the table level in Unity Catalog Delta tables. Azure Policy applies to Azure resources, not data values. Row filters control access. Auto-termination is compute management.

Question 14

You are a data engineer ingesting streaming data from Azure Event Hubs into a Delta table in Azure Databricks using Spark Structured Streaming. You need to ensure that duplicate messages (which Azure Event Hubs may deliver during failover) are not written to the Delta table. Which approach ensures exactly-once semantics for this streaming write?

Accepted Answer

Using foreachBatch with Delta MERGE achieves exactly-once semantics for streaming from Azure Event Hubs. In each micro-batch, you use MERGE INTO to match on a unique message ID: update if exists, insert if new. This deduplicates at the Delta table level regardless of Event Hubs redelivery. Spark Structured Streaming with Delta provides exactly-once guarantees only when the sink has idempotent writes. batch read doesn't handle streaming. Event Hubs consumer groups provide at-least-once delivery. Writing to Blob and then COPY INTO is a two-step batch pattern.

Question 15

You are designing a partitioning strategy for a large Delta table that stores IoT sensor data. The table has 5 billion rows and is queried by both sensor_id (high cardinality, 1 million unique values) and date (lower cardinality, 3 years of data). Which partitioning approach minimizes query latency while avoiding small file problems?

Accepted Answer

Partitioning by date (a low-to-medium cardinality column) creates a manageable number of partitions while enabling partition pruning for date-range queries. Liquid clustering on sensor_id then co-locates data for the same sensor within files, enabling data skipping for sensor-specific queries. Partitioning by high-cardinality sensor_id would create 1 million partitions with tiny files, causing the small file problem. Partitioning by both columns creates excessive partition proliferation. No partitioning requires full scans.

Question 16

You need to ingest data from a source system that tracks row-level inserts, updates, and deletes as change events (change data capture). Which Lakeflow Spark Declarative Pipelines feature is designed to process CDC feeds and apply changes to a target Delta table?

Accepted Answer

APPLY CHANGES INTO is a Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) feature specifically designed for processing change data capture (CDC) feeds. It reads a stream of CDC events (INSERT, UPDATE, DELETE) and applies them to a target Delta table, maintaining the current state. It handles out-of-order events using sequence numbers. Auto Loader ingests new files incrementally. COPY INTO is for bulk file ingestion. Plain Structured Streaming does not natively handle CDC event semantics.

Question 17

You are profiling a dataset in Azure Databricks to understand its distribution before applying transformations. You run a notebook and want to generate summary statistics including count, mean, standard deviation, min, and max for all numeric columns. Which PySpark function generates this summary?

Accepted Answer

df.describe() in PySpark computes summary statistics (count, mean, standard deviation, min, max) for all numeric and string columns in a DataFrame. For a more complete statistical profile, df.summary() can be used which also includes percentiles. df.show() displays row data. df.schema returns the column names and data types. df.count() returns the total number of rows only.

Question 18

You are implementing a CI/CD pipeline for Azure Databricks notebooks and jobs. You want to version-control all notebooks, job configurations, and cluster configurations in a Git repository. Which Azure Databricks feature enables bundling all these assets for deployment?

Accepted Answer

Databricks Asset Bundles (DAB) is the Databricks-native IaC framework for defining, packaging, and deploying Databricks resources (jobs, notebooks, pipelines, cluster configurations, permissions) as code using YAML configuration files. Bundles can be deployed using the Databricks CLI or REST API and integrate naturally with CI/CD pipelines. DBC files are legacy notebook export formats. Cluster policy templates only define cluster constraints. Unity Catalog export is for metadata.

Question 19

You have a Lakeflow Job in Azure Databricks with three tasks: Task A runs first, Task B depends on Task A succeeding, and Task C depends on Task B succeeding. Task B fails during a run. Which Lakeflow Jobs capability allows you to rerun only the failed task and its downstream tasks without re-running Task A?

Accepted Answer

Lakeflow Jobs (formerly Databricks Workflows) supports repair run, which allows you to rerun only the failed and skipped tasks in a previous job run without re-executing already successful tasks. This saves time and compute cost when only a subset of tasks need to be re-executed after fixing an issue. Restarting from the beginning re-runs all tasks including the already-successful Task A. Cloning and editing the job are manual workarounds that don't leverage the built-in repair capability.

Question 20

You need to apply version control best practices to your Azure Databricks project. Your team uses Git for source control. Which two Git practices should you implement in your Databricks development workflow? Choose 2.

Accepted Answer

Best practices include (A) using feature branches for isolated development and pull requests for code review before merging to main — this prevents unstable code from reaching production and enables collaboration. (C) Databricks Git folder integration allows developers to commit and sync notebooks directly from the Databricks workspace to the connected Git repository. Committing to a single branch (B), deleting branches after each run (D), and hardcoding secrets (E) are anti-patterns.

Question 21

You are monitoring an Azure Databricks Spark job that is running significantly slower than expected. You need to identify whether the slowdown is caused by data skew, shuffle issues, or I/O bottlenecks. Which tool should you use to investigate the Spark execution plan and per-stage metrics?

Accepted Answer

The Spark UI provides detailed execution information including the DAG (Directed Acyclic Graph) visualization of stages and tasks, per-stage metrics (shuffle read/write bytes, task duration, GC time), and per-task metrics that reveal data skew (large variance in task durations or data processed). You can identify shuffle bottlenecks, skew, and spill using the Stage Details page. Azure Monitor shows cluster-level metrics. Unity Catalog audit logs track data access. Cluster event logs show cluster lifecycle events.

Question 22

You are designing a production Lakeflow Job in Azure Databricks. One task runs a Python notebook that calls an external REST API. The external API occasionally returns HTTP 429 (Too Many Requests) errors. You need the task to automatically retry with a delay when this error occurs. Which Lakeflow Jobs configuration should you set?

Accepted Answer

Lakeflow Jobs (Databricks Workflows) supports configuring a retry policy per task, including the maximum number of retries and the delay between retries (in minutes). This is ideal for transient errors like HTTP 429 rate limiting from external APIs. The job framework automatically retries the task after the specified delay. Cluster auto-restart restarts the Spark cluster, not the task logic. A try/except block that suppresses errors would mark the task as successful when it should retry. A long timeout just waits without retrying.

Question 23

You need to send Azure Databricks Spark driver logs and cluster metrics to Azure Monitor Log Analytics so your operations team can create centralized alerts and dashboards. Which Azure Databricks feature enables log streaming to Azure Monitor?

Accepted Answer

Azure Databricks supports log delivery to Azure Monitor Log Analytics through the Azure Monitor agent for Databricks. You configure the Log Analytics workspace ID and key in the cluster's Spark configuration or via the workspace diagnostic settings, enabling automatic streaming of cluster logs, metrics, and events to Log Analytics. This allows centralized monitoring, alerting, and dashboarding. Init scripts can copy logs to DBFS but don't stream to Azure Monitor. Manual downloads are not scalable. Installing agents on driver nodes is not the supported approach.

Question 24

Your data engineering team has developed and tested changes to a Databricks Asset Bundle in the development environment. You need to deploy the bundle to the production environment using the Databricks CLI as part of a CI/CD pipeline. Which Databricks CLI command deploys the bundle to the production target?

Accepted Answer

The databricks bundle deploy --target <target_name> command deploys all resources defined in the Databricks Asset Bundle to the specified target environment (e.g., production). It creates or updates jobs, pipelines, cluster policies, and other resources in the target workspace. 'databricks bundle run' executes a specific resource after deployment. 'databricks bundle validate' checks the bundle configuration for errors without deploying. 'databricks bundle sync' synchronizes local files to the workspace.

Question 25

You are designing a testing strategy for an Azure Databricks data pipeline before deploying to production. You need to verify that individual notebook functions (such as a data cleansing function) return the correct output for given inputs. Which type of test should you write?

Accepted Answer

Unit tests verify individual functions or components in isolation — for example, testing that a Python function that cleanses addresses correctly handles null values, extra whitespace, and invalid characters. In Azure Databricks, unit tests are typically written using pytest and can be run in notebooks or as part of a CI/CD pipeline. Unit tests are the fastest and cheapest to run. End-to-end tests verify the full pipeline. Integration tests verify interactions between components. UAT involves business stakeholders validating business requirements.

DP-750 — Azure Databricks Data Engineer Associate