AWS Certified Data Engineer – Associate — Practice Set 1

Question 1

A company receives millions of IoT sensor readings per minute from factory equipment. The data must be delivered to Amazon S3 for long-term storage and batch analytics. The solution must require the LEAST operational overhead. Which AWS service should a data engineer use to ingest and deliver the data?

Accepted Answer

Amazon Kinesis Data Firehose is a fully managed service that automatically scales to match throughput and delivers streaming data directly to Amazon S3 with no consumer code to write or infrastructure to manage. Option A requires managing EC2 instances and writing consumer logic. Option C requires orchestrating SQS polling, Lambda concurrency, and batching logic. Option D requires managing an MSK cluster and EC2 workers for Kafka Connect, which is the highest operational overhead of all options.

Question 2

A data engineer needs to build a pipeline that reads change data capture (CDC) events from an Amazon RDS for PostgreSQL database and replicates them in near-real-time to an Amazon Redshift data warehouse for analytics. The solution must minimize impact on the source database. Which approach BEST meets these requirements?

Accepted Answer

AWS DMS with CDC reads the database transaction logs rather than querying tables, which minimises load on the source database. It streams changes continuously to the target without periodic full scans. Option B requires polling the source database every five minutes and performing expensive full or incremental queries, increasing load on the source. Option C restores snapshots hourly, which is not near-real-time and adds significant cost and latency. Option D uses EventBridge Pipes, which captures database-level events from supported services but does not natively support RDS PostgreSQL CDC log streaming to Redshift.

Question 3

A company processes clickstream data from its e-commerce website. A data engineer must transform the raw JSON events into Parquet format, enrich them by joining with a customer dimension table stored in Amazon S3, and load the results into Amazon Redshift every hour. The pipeline must be serverless with no cluster management. Which solution MOST efficiently meets these requirements?

Accepted Answer

AWS Glue is a fully serverless ETL service that natively converts JSON to Parquet, supports joins with reference data in Amazon S3 through the Data Catalog, and can trigger the Amazon Redshift COPY command after the job completes — all without managing any servers. Option B requires provisioning and terminating an EMR cluster each hour, adding operational overhead and cold-start latency. Option C uses a streaming-optimised service (Flink) for a batch hourly job and does not directly load to Redshift. Option D uses Athena for transformation (which is not designed for large-scale ETL writes) and AWS DataSync, which is a file-transfer service not suited for database loading.

Question 4

An organization ingests log files from hundreds of on-premises servers into Amazon S3 throughout the day. A data engineer must ensure that whenever a new log file lands in a specific S3 prefix, an AWS Glue job is automatically triggered to parse and catalogue the data. Which solution meets this requirement with the LEAST custom code?

Accepted Answer

AWS Glue event-based triggers natively integrate with Amazon EventBridge and can start a Glue job automatically when an S3 object-created event matches a specified prefix — requiring no custom Lambda code or Step Functions orchestration. Option A works but requires writing and maintaining a Lambda function. Option B also works but adds Step Functions complexity for a simple trigger use case. Option C uses polling every minute, which wastes compute, adds latency, and is not event-driven.

Question 5

A financial services company streams trade events into Amazon Kinesis Data Streams at a peak rate of 50,000 records per second. A data engineer notices that the stream consumers are experiencing ProvisionedThroughputExceededException errors during market-open periods. The stream currently has 10 shards. The application uses the Kinesis Client Library (KCL). Which combination of actions should the data engineer take to resolve the throttling with MINIMAL cost increase? (Choose TWO.)

Accepted Answer

Enhanced fan-out gives each registered consumer its own dedicated 2 MB/s read throughput pipe, eliminating contention between multiple consumers on the same shard without requiring more shards. Implementing exponential backoff with jitter is a standard resiliency pattern for handling transient throttling. Together these address the root cause (shared throughput between consumers) and gracefully handle burst spikes. Option B would increase the shard count 5x, significantly increasing cost, but does not solve the problem if multiple consumers share read bandwidth. Option D is not a drop-in replacement for Kinesis Data Streams when record-level processing is needed. Option E has no effect on read throughput throttling.

Question 6

A data engineer is building a data pipeline that reads records from an Amazon DynamoDB table and needs to apply complex multi-step transformations before loading results into Amazon S3. The pipeline must support visual authoring and reuse of transformation logic across multiple pipelines. Which AWS service BEST fits this requirement?

Accepted Answer

AWS Glue Studio provides a visual drag-and-drop interface for building ETL pipelines, supports reading from DynamoDB as a source, and allows engineers to author reusable custom transforms that can be shared across jobs. Option B requires writing all transformation logic in PySpark without visual authoring. Option C uses Lambda chained via SQS, which lacks visual authoring and becomes complex for multi-step transformations. Option D uses Athena SQL, which does not provide visual authoring of transformations or direct DynamoDB reading.

Question 7

A data engineer must ingest data from a third-party REST API that returns paginated JSON responses and load the results into Amazon S3 daily. The engineer wants to use a managed connector without writing custom integration code. Which AWS service MOST directly supports this use case?

Accepted Answer

Amazon AppFlow is a fully managed integration service that provides pre-built connectors for dozens of SaaS sources and supports custom connectors for REST APIs, delivering data to Amazon S3 on a schedule without custom code. Option A requires writing and maintaining Python code to handle pagination, authentication, and retries. Option C uses AWS Data Pipeline, a legacy service that does not have managed REST API connectors. Option D configures Firehose as a receiver (HTTP endpoint destination), not as an HTTP API caller — it cannot pull from a third-party REST API.

Question 8

A company has a large AWS Glue ETL job that transforms a 5 TB dataset stored in Amazon S3 and takes 4 hours to complete daily. The data engineer notices that the job spends 70% of its time on a single shuffle-heavy join between two large tables. The engineer wants to reduce the job runtime by at least 50% without changing the business logic. Which approach should the data engineer take?

Accepted Answer

Pre-partitioning both source datasets in Amazon S3 by the join key eliminates the shuffle phase entirely because co-located data for matching keys resides in the same partitions. Partition pruning further reduces the data scanned. This directly addresses the root cause — the shuffle-heavy join — by co-locating the data before the job even begins. Option A (job bookmarks) helps with incremental loads but does not reduce the shuffle on a full scan. Option B describes valid Spark optimisations but pre-partitioning in S3 is more fundamental and avoids in-memory shuffles completely. Option C adds more DPUs, which parallelises the shuffle but does not eliminate it, and increases cost without addressing the root cause.

Question 9

A data engineer is building a pipeline to process streaming retail transaction data. The business requires that duplicate transactions — identified by a unique transaction ID — must be detected and removed before the data is written to Amazon S3. The pipeline must operate in near-real-time. Which solution BEST meets this requirement?

Accepted Answer

Amazon Kinesis Data Firehose supports Lambda transformation functions that are invoked before data is written to the destination. The Lambda function can look up the transaction ID in Amazon DynamoDB (which provides single-digit millisecond lookups) and drop duplicate records in near-real-time. Option B runs deduplication daily, not in near-real-time. Option C uses SQS FIFO deduplication, which deduplicates within a 5-minute window and is designed for queue-based messaging rather than streaming analytics pipelines. Option D partitions data by transaction ID but does not remove duplicates — it only co-locates them.

Question 10

A company stores 200 TB of historical sales data in Amazon S3. Analysts run ad-hoc SQL queries against the data using Amazon Athena. Query performance is slow because the data is stored as uncompressed CSV files in a single S3 prefix. A data engineer must improve query performance without moving the data to a different storage service. Which action will MOST improve Athena query performance?

Accepted Answer

Converting to Parquet (a columnar format) enables Athena to scan only the columns needed by a query rather than reading entire rows, reducing the data scanned by up to 99% for analytical queries. Partitioning by date allows Athena's partition pruning to skip irrelevant prefixes entirely. Together these changes typically produce 10–100x query speedups. Option A speeds up data transfer rates to/from S3 but does not reduce the volume of data Athena must scan. Option C caches results for identical queries but does not improve first-run performance or queries with different filters. Option D adjusts the timeout and reuses cached results but has no effect on slow first-run scans.

Question 11

A data engineer is designing a data lake on Amazon S3 to support both streaming writes from Amazon Kinesis Data Firehose and time-travel queries from Amazon Athena. The engineer needs ACID transaction support and the ability to roll back to previous table versions. Which table format should the data engineer use?

Accepted Answer

Apache Iceberg provides full ACID transactions, snapshot isolation, and time-travel queries on Amazon S3. AWS Glue Data Catalog natively manages Iceberg table metadata, and Amazon Athena supports Iceberg DML (INSERT, UPDATE, DELETE, MERGE). This combination directly satisfies all stated requirements. Option A uses plain Parquet with Hive partitioning, which does not provide ACID transactions or time-travel. Option C uses Lake Formation governed tables, which adds fine-grained access control but does not inherently provide time-travel or ACID on S3. Option D uses S3 object versioning, which tracks file-level versions but does not provide SQL-level time-travel queries across a logical table.

Question 12

A company runs an Amazon Redshift cluster that stores 10 TB of data. A data engineer notices that some frequently accessed tables take a long time to query because Amazon Redshift is performing full table scans. The engineer suspects that the wrong distribution style and sort keys are set. Which Amazon Redshift design choice MOST reduces the amount of data scanned for analytical queries that filter by date range and join on customer ID?

Accepted Answer

Setting the distribution key to customer_id co-locates rows with the same customer on the same compute node, eliminating data movement during joins on customer_id. The compound sort key on (order_date, customer_id) allows zone map pruning to skip data blocks outside the queried date range. Together these eliminate redistributed joins and minimise blocks scanned. Option A uses EVEN distribution, which distributes rows randomly and maximises data movement during customer_id joins. Option C uses ALL distribution (replicates the entire table to every node), which is only suitable for small dimension tables and wastes storage for 10 TB tables; interleaved sort keys also have higher vacuum costs. Option D AUTO distribution is a reasonable default but the engineer has specific query patterns that warrant deliberate key selection.

Question 13

A retail company maintains a product catalogue in Amazon DynamoDB. The table has a partition key of product_id and a sort key of category. The company needs to support two additional access patterns: (1) retrieve all products with a price below a threshold, and (2) retrieve all products by brand name. Adding these query patterns must not impact existing read performance. Which solution MOST efficiently supports these new access patterns?

Accepted Answer

Global secondary indexes allow DynamoDB to support additional access patterns beyond the base table's primary key without affecting the performance of existing queries. A GSI with price as the partition key supports threshold-based price queries using a begins_with or comparison condition on the sort key if prices are stored as strings, or via a query with a filter. A GSI with brand as the partition key allows efficient retrieval of all products for a brand. GSIs are maintained asynchronously by DynamoDB and do not impact base table read performance. Option B introduces significant operational complexity with Lambda, OpenSearch, and stream management. Option C performs full table scans, which are slow and expensive at scale. Option D provides only a point-in-time export and introduces latency between the export and queryable state.

Question 14

A company stores raw data in Amazon S3 and uses the AWS Glue Data Catalog as its central metadata repository. The data engineering team wants to enforce column-level access controls so that analysts in the finance department can query salary columns in an HR table, while other analysts cannot. Which service should the data engineer use to implement this access control?

Accepted Answer

AWS Lake Formation provides fine-grained access control at the database, table, column, and row level within the AWS Glue Data Catalog. By granting column-level SELECT permissions only to the finance IAM role or group, Lake Formation enforces these controls for all query engines that integrate with the Data Catalog (Athena, Redshift Spectrum, AWS Glue). Option A uses S3 bucket policies that operate at the object (file) level and cannot restrict access to individual columns within a Parquet or CSV file. Option C IAM conditions on GetTable control metadata access but do not enforce column-level filtering at query time. Option D Amazon Macie is a data discovery and classification service, not an access control enforcement service.

Question 15

A data engineering team manages an Amazon S3 data lake that grows by 500 GB of raw log data daily. After 30 days the data is rarely accessed, and after 90 days it is virtually never accessed but must be retained for 7 years for regulatory compliance. The team wants to minimise storage costs. Which Amazon S3 lifecycle configuration BEST meets these requirements?

Accepted Answer

Transitioning to S3 Standard-IA after 30 days reduces the per-GB storage price for infrequently accessed data while maintaining millisecond retrieval. Moving to S3 Glacier Flexible Retrieval after 90 days provides the lowest-cost long-term storage for virtually unused data that must be retained. The expiration action at 7 years automatically deletes objects after the regulatory retention period ends, preventing indefinite accumulation. Option B uses S3 Intelligent-Tiering, which adds a per-object monitoring fee that is cost-effective for unpredictable access patterns but less optimal than explicit lifecycle rules when the access pattern is well-known. Option C replicating to another Region does not reduce storage cost and adds replication and cross-Region transfer costs. Option D Requester Pays shifts retrieval costs but does not change the storage cost for the team.

Question 16

A data engineer has an AWS Glue ETL job that runs every night. The job occasionally fails because a source file is malformed. The engineer wants to be notified immediately via email whenever the job fails. Which combination of AWS services should the data engineer use?

Accepted Answer

AWS Glue emits Job State Change events (including FAILED) to Amazon EventBridge. An EventBridge rule can match these events in near-real-time and route them to an Amazon SNS topic. A simple email subscription on the SNS topic delivers the failure notification. This requires no custom code and is the standard AWS pattern for Glue job failure alerting. Option A uses CloudTrail, which captures API-level events for auditing, not job execution state changes, and CloudTrail Insights detects anomalous API call volumes, not job failures. Option C CloudWatch Logs metric filters can work but require log pattern matching, which is more fragile and indirect than the native EventBridge event. Option D misuses job bookmarks (which track processed data, not failure state) and requires unnecessary custom code.

Question 17

A data engineer manages a complex multi-step data pipeline that consists of an AWS Glue crawler, two AWS Glue ETL jobs run in sequence, and a final Amazon Redshift COPY command. The engineer needs to orchestrate these steps so that each step only runs after the previous one succeeds, with automatic retry logic on failure. Which service should the data engineer use?

Accepted Answer

AWS Step Functions is purpose-built for orchestrating multi-step workflows across AWS services. It supports sequential execution (each state runs only after the previous succeeds), built-in retry configurations (with exponential backoff and max attempts) on each state, and native integrations with AWS Glue and Amazon Redshift Data API — all without custom code. Option A uses Glue Workflows, which orchestrates Glue-native resources well but requires a Lambda workaround for the Redshift COPY step and has less flexible retry logic. Option C Amazon MWAA is also capable but introduces an additional managed cluster and is better suited for complex DAG-based workflows with many teams. Option D uses a time-based scheduler, which cannot detect whether the previous step succeeded or failed.

Question 18

A company runs AWS Glue ETL jobs that process incremental data from Amazon S3 each day. Over time, the engineers notice that the jobs occasionally reprocess data that was already processed in a previous run, causing duplicate records in the target Amazon Redshift table. Which AWS Glue feature should the data engineer enable to prevent this?

Accepted Answer

AWS Glue job bookmarks maintain state about which data has already been processed in previous runs. When enabled, subsequent runs of the same job start from where the previous run left off, skipping already-processed files or records. This is the native Glue mechanism for incremental processing and directly prevents duplicate reprocessing. Option A detects data quality issues but does not prevent the job from reading already-processed files. Option C limits concurrency to prevent parallel runs but does not prevent a sequential re-run from reprocessing old data. Option D S3 versioning tracks file revisions but does not help the Glue job know which files it has already processed.

Question 19

A data engineer is troubleshooting an AWS Glue ETL job that processes 1 TB of Parquet files from Amazon S3. The job completes successfully but takes 6 hours when it should take approximately 1 hour. The Glue job uses Apache Spark and is allocated 20 DPUs. Amazon CloudWatch metrics show that all executors are running but CPU utilisation is below 10% throughout the job. What is the MOST likely cause and the appropriate fix?

Accepted Answer

Low CPU utilisation across all executors while the job is slow strongly indicates that the Spark job is running with too few partitions, leaving most executor cores idle waiting for a small number of tasks to complete. This commonly occurs with large Parquet files that produce few partitions. Using repartition() to increase the number of partitions distributes the work across all available cores, reducing runtime proportionally. Option A cross-Region S3 access adds latency but would manifest as high network wait time, not uniformly low CPU. Option C reducing DPUs reduces parallelism, which would make a CPU-bottlenecked job slower, not faster. Option D is incorrect; AWS Glue does publish Spark metrics to CloudWatch and also provides the Spark History Server UI.

Question 20

A data engineer must ensure that an Amazon Redshift cluster automatically creates a snapshot before any schema migration script is executed, so that the cluster can be restored if the migration causes data corruption. The migration scripts are executed from an AWS Lambda function. Which approach BEST meets this requirement?

Accepted Answer

Calling the Amazon Redshift create-cluster-snapshot API directly from the Lambda function immediately before executing the migration script guarantees a consistent restore point that is tightly coupled to the migration event. The function can poll the snapshot status and only proceed once the snapshot reaches the 'available' state. Option A automated snapshots run on a schedule (every 8 hours by default), not on-demand before each migration, so there may be up to 8 hours of data drift between the last snapshot and the migration. Option C requires the engineering team to time their migrations to a fixed window, which is operationally fragile. Option D cross-Region snapshot copy provides disaster recovery but does not provide a guaranteed pre-migration restore point in the primary Region.

Question 21

A data engineer must ensure that all data written to an Amazon S3 data lake bucket is encrypted at rest using keys managed by the company's security team. The security team requires full control over key rotation and the ability to audit key usage. Which encryption option BEST meets these requirements?

Accepted Answer

SSE-KMS with a customer-managed key (CMK) gives the security team full control over key policies, key rotation schedules, and access auditing through AWS CloudTrail (which logs every KMS API call). The security team can restrict which IAM principals can use the key. Option A SSE-S3 uses AWS-managed keys, which the security team cannot control, rotate on custom schedules, or audit at the key-usage level. Option C SSE-C requires the client to manage and pass the key with every request, which is operationally complex and the key is not stored or managed in KMS. Option D client-side encryption moves all key management complexity to the application layer and provides no AWS-native audit trail.

Question 22

A company's data lake on Amazon S3 contains personally identifiable information (PII) such as social security numbers and email addresses mixed within large JSON files. A data engineer must automatically identify and classify files containing PII across the entire bucket. Which AWS service should the data engineer use?

Accepted Answer

Amazon Macie is a data security service that uses machine learning to automatically discover and classify sensitive data in Amazon S3. It detects PII types including social security numbers, email addresses, credit card numbers, and more, and produces findings with the specific files and field patterns where PII was found. Option A AWS Glue DataBrew can apply PII transformations (masking, substitution) but requires the engineer to already know which columns contain PII and to run a recipe job; it does not auto-classify files. Option C AWS Security Hub aggregates security findings but does not scan S3 object content for PII patterns. Option D Amazon Comprehend can extract entities from text but requires custom pipelines per file and does not natively classify S3 buckets at scale.

Question 23

A healthcare company's data engineers need to query patient records stored in Amazon S3 using Amazon Athena. The security policy requires that (1) queries can only access records where the patient's state matches the analyst's assigned region, enforced at query time, and (2) specific columns containing diagnosis codes must be hidden from analysts who do not hold the HIPAA-certified role. Which combination of AWS services and features MOST securely meets these requirements without duplicating data?

Accepted Answer

AWS Lake Formation provides both row-level security (data filters based on partition values or column values matched to IAM session tags) and column-level permissions in a single governance layer. Row-level filters can restrict patients by state matched to the analyst's IAM tag, while column-level permissions deny SELECT on diagnosis code columns to roles that are not HIPAA-certified. This is enforced at query time by Athena via the Data Catalog without duplicating any data. Option A uses S3 prefix policies, which are file-level and cannot enforce column restrictions; IAM permission boundaries control what IAM permissions can be granted, not data column visibility. Option C Athena workgroups control query routing and cost but do not enforce row or column data restrictions. Option D requires maintaining a separate masked copy per user group, multiplying storage costs and creating data drift risk.

Question 24

A company requires that all data transferred between its on-premises data centre and AWS Glue ETL jobs is encrypted in transit using a dedicated private connection. The data must not traverse the public internet. Which solution BEST meets this requirement?

Accepted Answer

AWS Direct Connect establishes a dedicated private network connection between the on-premises data centre and AWS that never traverses the public internet. Configuring the AWS Glue job to run in a VPC with a private subnet routes all traffic through the Direct Connect virtual interface. This satisfies both the private connection and encryption-in-transit requirements when combined with in-transit encryption on the connection. Option A routes over the public internet despite TLS encryption, violating the private connection requirement. Option C uses Site-to-Site VPN, which tunnels over the public internet (even if encrypted with IPsec), which may not meet compliance requirements for a dedicated private connection. Option D configures Glue's internal Spark shuffle encryption and an S3 VPC endpoint, which are good practices but do not address the on-premises-to-AWS private connectivity requirement.

Question 25

A data engineering team wants to track all read and write API calls made against their AWS Glue Data Catalog, including who accessed which table and when, for compliance auditing. Which AWS service provides this audit trail with MINIMAL configuration?

Accepted Answer

AWS CloudTrail is the AWS service for recording API-level activity across all AWS services including AWS Glue. When enabled (which is the default for management events in new accounts), CloudTrail logs every GetTable, CreateTable, UpdateTable, and GetPartitions API call with the caller's identity, timestamp, and source IP — with no additional configuration beyond creating a trail. Option A CloudWatch detailed monitoring captures service-level metrics (job execution counts, DPU hours) but does not log individual Data Catalog API calls or caller identities. Option C AWS Config records resource configuration changes (e.g., table schema changes) but does not log data access API calls like GetTable. Option D Data Catalog encryption and resource policies protect data but do not generate an audit log of read API calls.

DEA-C01 — AWS Certified Data Engineer – Associate