Cisco Data Center AI Infrastructure Specialist — Practice Set 1

Question 1

A data center architect is planning a new AI/ML cluster. The primary workload is large-scale distributed model training that runs for hours and transfers large tensors between GPUs. Which characteristic BEST describes this workload type?

Accepted Answer

Distributed model training is a compute-intensive, batch workload. GPUs perform billions of floating-point operations per second during forward and backward passes, and the job runs as a long-duration batch rather than responding to individual real-time requests. Memory-intensive inference workloads are a separate class characterized by large model loading and low-latency response requirements.

Question 2

An architect is specifying rack power density for a new AI POD that will house NVIDIA H100 GPU servers. Each 4U server draws approximately 10 kW. What power density range should the architect plan for in a fully loaded AI rack?

Accepted Answer

Modern GPU-accelerated AI racks routinely reach 40–80 kW per rack, compared to 5–15 kW for traditional compute racks. A single NVIDIA H100 SXM5 server can draw 10 kW; a fully loaded rack with multiple such servers quickly reaches this range. This elevated power density drives the requirement for direct liquid cooling (DLC) solutions in AI POD designs.

Question 3

Which Cisco UCS server model is specifically designed to host NVIDIA H100 or A100 GPUs via a GPU mezzanine tray in the Cisco AI POD reference architecture?

Accepted Answer

The Cisco UCS X210c M7 compute node, used within the UCS X-Series modular system, supports NVIDIA H100 and A100 GPUs via a dedicated GPU mezzanine (mezz) tray. It connects to the Cisco UCS Fabric Interconnect 6536 via the X-Fabric module, providing 100 GbE server-side connectivity. The C240 and B200 are general-purpose rack and blade servers not optimized for this GPU mezz architecture.

Question 4

A network architect is comparing fat-tree and dragonfly topologies for a 512-GPU AI training cluster. Which statement BEST describes the advantage of a dragonfly topology for GPU-to-GPU all-reduce communication?

Accepted Answer

Dragonfly topology reduces the network diameter (maximum hop count) compared to fat-tree at very large GPU counts, decreasing all-reduce latency for collective communications. It also reduces the total cable count relative to a fully non-blocking fat-tree of the same scale. Fat-tree at 512+ GPUs requires large numbers of spine switches and cables; dragonfly groups switches into pods with direct inter-group links, lowering both hop count and cabling complexity. Both topologies support RoCEv2, and both benefit from ECMP.

Question 5

A cooling engineer is evaluating options for a 60 kW/rack AI pod. Which cooling approach is MOST appropriate for this power density?

Accepted Answer

At 40–80 kW per rack, traditional air cooling is insufficient — standard CRAC units and even in-row coolers are designed for densities up to 20–30 kW. Direct liquid cooling (DLC) using rear-door heat exchangers or cold plates mounted directly on GPU heat spreaders is required. DLC transfers heat directly to facility chilled water, efficiently handling 60+ kW rack densities without relying on large volumes of chilled air.

Question 6

What is the primary interconnect advantage of NVLink over PCIe for multi-GPU AI training within a single server node?

Accepted Answer

NVLink 4.0 on NVIDIA H100 provides up to 900 GB/s of bidirectional GPU-to-GPU bandwidth within an NVLink domain (e.g., 8 GPUs connected via NVSwitch), versus approximately 63 GB/s bidirectional for PCIe Gen 5 x16. This order-of-magnitude difference is critical for all-reduce collective operations during distributed training, where gradient tensors must be exchanged rapidly between GPUs. NVLink is an intra-node interconnect only — it does not span rack boundaries.

Question 7

The Cisco AI POD reference architecture combines which three core technology layers?

Accepted Answer

The Cisco AI POD is a validated reference architecture combining Cisco UCS X-Series compute (X210c M7 with GPU mezz), Cisco Nexus 9000 series switches for a spine-leaf AI fabric, and validated all-flash storage partners — primarily NetApp AFF or Pure Storage FlashBlade — to provide the low-latency, high-throughput storage required for AI dataset loading and checkpoint operations. Cisco Validated Designs (CVDs) document the specific component versions and configuration guidance.

Question 8

An architect is designing a 16-rack AI training cluster using Cisco Nexus 9336C-FX2 switches (36 ports of 100G or 400G). Each GPU rack has 8 servers each with one 400G GPU NIC. The design must be non-blocking at the leaf-spine layer. Which spine configuration satisfies non-blocking uplinks for 16 leaf switches each with 8 × 400G server ports?

Accepted Answer

Each leaf switch has 8 × 400G server downlinks = 3.2 Tbps downlink bandwidth. For non-blocking, the total uplink bandwidth must equal 3.2 Tbps. With 8 spine switches providing 2 × 400G uplinks per leaf per spine, that gives 8 × 2 × 400G = 6.4 Tbps uplink, which is non-blocking (oversubscription ratio 1:1 or better). Option B provides only 2 × 400G total uplink (800G) against 3.2 Tbps downlink — a 4:1 oversubscription. Option A with 4 spines and 2 uplinks per leaf per spine yields 4 × 2 × 400G = 3.2 Tbps, which is exactly 1:1 non-blocking and is also valid, but option C provides headroom. In practice the 8-spine design is the typical AI fabric recommendation for non-blocking with growth headroom.

Question 9

Which two Ethernet mechanisms are REQUIRED on a Cisco Nexus 9000 switch to create a lossless fabric for RoCEv2 AI traffic? (Choose 2)

Accepted Answer

RoCEv2 requires a lossless Ethernet fabric because RDMA transport has no retransmission mechanism — dropped packets cause fatal connection errors. Priority Flow Control (PFC, IEEE 802.1Qbb) provides hop-by-hop pause frames on a specific traffic class (typically priority 3) to prevent packet drops. Explicit Congestion Notification (ECN, RFC 3168) marks packets before drops occur, allowing the DCQCN algorithm on the RDMA NIC to reduce injection rate proactively. Both are required together for proper lossless operation.

Question 10

On a Cisco Nexus 9000 switch, a network engineer is configuring a QoS policy to enable PFC on priority 3 for RoCEv2 lossless traffic. Which configuration snippet correctly applies PFC pause on priority 3?

Accepted Answer

On Nexus 9000 NX-OS, lossless RoCEv2 requires a queuing policy-map applied at the system level that configures a dedicated no-drop queue (queue 3 by default for RoCEv2/RDMA) with a PFC pause buffer, pause threshold, and resume threshold. The policy-map targets class c-out-8q-q3 within the queuing type policy. The 'flowcontrol' command is link-level flow control (PAUSE frames without priority awareness), not PFC. Option A is per-interface PFC mode, which is a prerequisite but not sufficient alone — the queuing policy sets the buffer parameters.

Question 11

What is GPUDirect RDMA, and what component does it eliminate from the AI data path?

Accepted Answer

GPUDirect RDMA is an NVIDIA technology (supported on CUDA-capable GPUs with an RDMA-capable NIC such as NVIDIA ConnectX) that allows a network adapter or another GPU to perform DMA transfers directly into or out of GPU GDDR/HBM memory, completely bypassing the CPU and host system memory (DRAM). This eliminates the GPU→CPU memory copy and the CPU→NIC DMA, reducing latency and CPU overhead for inter-node GPU-to-GPU communication in distributed training.

Question 12

A network engineer is deploying Cisco Nexus Dashboard Insights (NDI) on an AI fabric. Which telemetry technology does NDI use to collect per-flow, per-hop latency and congestion data from Nexus 9000 switches without requiring separate monitoring probes?

Accepted Answer

Cisco Nexus Dashboard Insights uses In-Band Network Telemetry (INT) where the Nexus 9000 data plane stamps telemetry metadata (timestamp, queue depth, egress port utilization) directly into packet headers as they traverse the fabric. This data is streamed via gRPC to Nexus Dashboard Insights for per-flow, per-hop analysis without dedicated external probes. SNMP polling is too slow for real-time AI fabric congestion visibility, and NetFlow/RSPAN lack the per-hop latency granularity required.

Question 13

An architect is comparing InfiniBand HDR (200 Gbps) and RoCEv2 (100GbE) for the AI training fabric. Which statement MOST accurately reflects the trade-off between these two technologies?

Accepted Answer

InfiniBand provides extremely low latency (sub-1 µs) and high bandwidth with a purpose-built, reliable transport protocol (IB Verbs), but requires dedicated IB switches (e.g., NVIDIA Quantum) and HCAs — a separate fabric from Ethernet. RoCEv2 runs RDMA semantics over standard UDP/IPv4 or IPv6 on Ethernet, allowing reuse of existing Ethernet switching infrastructure (Nexus 9000), lower cost, and familiar management tooling, at the cost of requiring lossless Ethernet configuration and slightly higher latency than native IB at scale.

Question 14

During a distributed training job across 64 GPUs on a RoCEv2 fabric, an engineer observes periodic training throughput degradation. Nexus Dashboard Insights shows micro-burst congestion on spine uplinks but no PFC pause frames. Which ECMP enhancement should the engineer enable on the Nexus 9000 spines to reduce hash polarization and improve load distribution for AI all-reduce traffic?

Accepted Answer

Hash polarization occurs when standard 5-tuple ECMP sends multiple flows of an all-reduce collective down the same spine links, causing micro-bursts on some links while others are underutilized. Flowlet switching (adaptive load balancing) on NX-OS detects gaps between bursts (flowlets) within a TCP/RDMA flow and re-hashes each flowlet independently across available ECMP paths. An inactivity interval of 50–100 µs is appropriate for RDMA traffic burst patterns. Per-destination hashing worsens polarization. Disabling ECMP eliminates redundancy. OSPF maximum-paths affects routing, not per-flow load balancing at the switch forwarding layer.

Question 15

A spine link in a 400G AI fabric is showing repeated PFC pause storms that cascade across the fabric and stall training jobs. An engineer runs 'show queuing interface Ethernet1/1' and sees the RoCEv2 queue (queue 3) at 95% utilization with pause frames sent every 2 ms. Which corrective action MOST effectively addresses PFC pause storms without dropping RoCEv2 packets?

Accepted Answer

PFC pause storms occur when congestion is so persistent that pause frames propagate upstream, stalling unrelated flows. The correct remediation is to tune DCQCN (Data Center Quantized Congestion Notification): lower the ECN marking threshold so senders reduce their injection rate well before queue fill reaches the PFC pause threshold. This creates a tiered congestion response — ECN rate reduction first, PFC pause only as last resort. The PFC watchdog detects deadlock conditions. Disabling PFC would cause packet drops that crash RDMA connections. Moving to DSCP 0 removes lossless treatment entirely.

Question 16

An NVIDIA DGX H100 system contains 8 H100 SXM5 GPUs interconnected by NVLink 4.0 via NVSwitch. What is the total GPU-to-GPU NVLink bandwidth available within a single DGX H100 node?

Accepted Answer

Each H100 SXM5 GPU supports NVLink 4.0 with 900 GB/s bidirectional bandwidth to the NVSwitch fabric. With 8 GPUs each providing 900 GB/s, the aggregate NVLink bandwidth across the full DGX H100 node is 7.2 TB/s. This is orders of magnitude higher than PCIe Gen 5 x16 (~63 GB/s bidirectional), which is why NVLink-based DGX nodes dramatically outperform PCIe-connected multi-GPU systems for all-reduce collective operations.

Question 17

A storage architect needs to provide high-throughput parallel storage for a 64-GPU AI training cluster that will read 500 GB training datasets repeatedly from shared storage. Which storage protocol is BEST suited for low-latency, high-bandwidth access from all 64 GPU servers simultaneously?

Accepted Answer

NVMe over Fabrics (NVMe-oF) using RoCEv2 as the transport provides microsecond-level storage access latency and allows all 64 GPU servers to access NVMe storage namespaces simultaneously over the same lossless Ethernet AI fabric. This eliminates the SAS/SATA bottleneck of traditional storage and the limited scalability of single-head NFS. Running NVMe-oF over the existing RoCEv2 AI fabric avoids deploying a separate storage network, and Pure FlashBlade and NetApp AFF support NVMe-oF as primary protocols in AI POD designs.

Question 18

An ML engineer is choosing between model training and inference deployment and asks the infrastructure team about GPU memory requirements. Which statement CORRECTLY distinguishes GPU VRAM requirements for training versus inference?

Accepted Answer

Training a large model requires GPU VRAM to hold: (1) model weights, (2) optimizer states (Adam optimizer stores 1st and 2nd moment estimates — 2× the weight size), (3) gradients (equal to weight size), and (4) forward-pass activations for backpropagation. This can require 4–8× the memory of inference, which only needs to load model weights (and KV-cache for transformer autoregression). This is why large model training requires multi-GPU model parallelism while inference can often fit on a single or fewer GPUs.

Question 19

A data scientist is training a 70-billion parameter model that does not fit in the VRAM of a single GPU node. Which parallelism strategy distributes model layers across multiple GPU nodes so each node holds only a subset of the model?

Accepted Answer

Pipeline parallelism (a model parallelism strategy) partitions the model's layers into stages, each assigned to a different GPU node. Micro-batches flow through the pipeline stages sequentially. This allows training models too large to fit on any single GPU. Data parallelism replicates the full model on each worker — it does not solve the single-node memory limitation. Gradient checkpointing reduces activation memory but not weight/optimizer-state memory. Mixed precision reduces per-parameter memory by 2× but cannot address a 70B parameter model on typical GPU VRAM alone.

Question 20

An architect is selecting a parallel file system for a 512-GPU AI cluster that requires >500 GB/s aggregate throughput for dataset loading and >100 GB/s for model checkpoint writes. Which file system is specifically designed for HPC/AI parallel I/O with a metadata server (MDS) and object storage server (OSS) architecture?

Accepted Answer

Lustre is the dominant parallel file system in HPC and large-scale AI clusters (including most Top500 supercomputers). Its architecture separates the Metadata Server (MDS/MDT — handles namespace, directory operations, file creation) from Object Storage Servers (OSS/OST — store file data). Clients stripe files across multiple OSTs in parallel, achieving aggregate bandwidths exceeding 1 TB/s in large deployments. BeeGFS is also a valid parallel file system answer but uses a different distributed metadata architecture. GPFS/Spectrum Scale is correct but uses a shared-disk model with NSDs. HDFS is designed for MapReduce/batch analytics, not low-latency AI training I/O.

Question 21

An infrastructure administrator wants to use Cisco Intersight to manage a UCS X-Series AI compute cluster. Which of the following tasks can Intersight perform WITHOUT requiring a separate on-premises management tool?

Accepted Answer

Cisco Intersight is a SaaS-based infrastructure management platform that connects to UCS domains via Intersight Managed Mode (IMM). It provides centralized server inventory, real-time alerts, firmware lifecycle management, and workflow automation (via Intersight Orchestrator) — all without requiring a separate on-premises UCS Manager installation. Physical hardware installation, NVLink configuration (handled by NVIDIA drivers/NCCL), and Nexus BGP routing are outside its scope.

Question 22

An ML platform engineer wants to isolate GPU resources so multiple inference workloads can share a single NVIDIA A100 80GB GPU with guaranteed memory and compute partitions. Which NVIDIA technology enables this GPU partitioning?

Accepted Answer

Multi-Instance GPU (MIG) is an NVIDIA hardware partitioning feature available on A100, H100, and later GPUs. It allows a single physical GPU to be divided into up to 7 GPU instances (for A100 80GB), each with a dedicated slice of GPU compute (SMs), L2 cache, and HBM memory bandwidth. This provides full hardware isolation (unlike time-sliced vGPU) for running multiple independent inference workloads on a single GPU. The NVIDIA GPU Operator for Kubernetes automates MIG configuration in AI clusters managed by Kubernetes/Intersight.

Question 23

A DevOps engineer wants to automate GPU health validation before accepting a new AI compute node into production. Which Intersight Orchestrator workflow step would BEST verify that all GPUs in a UCS X210c M7 node are functioning correctly?

Accepted Answer

GPU health validation requires running compute stress tests (gpu-burn) or the NVIDIA DCGM (Data Center GPU Manager) diagnostic suite (dcgmi diag -r 3 for a thorough test including memory, PCIe, and compute checks) against each GPU. Intersight Orchestrator can be configured with a workflow that SSH-connects to a new node post-boot, triggers these diagnostics, collects results, and gates the node's admission into the Kubernetes GPU pool. POST errors catch hardware initialization failures but not subtle GPU compute errors. SNMP and NDI flow tables do not test GPU compute correctness.

Question 24

An operations team uses Cisco Nexus Dashboard for AI fabric visibility. They observe that GPU rack 7 leaf switch shows periodic 20% drops in fabric utilization during training jobs. Nexus Dashboard anomaly detection flags it as an 'elephant flow imbalance'. Which dashboard feature and subsequent action should the engineer use to investigate and resolve this?

Accepted Answer

Nexus Dashboard Insights (NDI) provides per-flow, per-hop telemetry via INT that allows engineers to identify which specific flows are concentrated in a single ECMP hash bucket (hash polarization). Once identified, the engineer uses Nexus Dashboard Fabric Controller (NDFC) to push a configuration change enabling flowlet-based adaptive load balancing on the affected leaf and spine switches. Rebooting clears the forwarding table but does not fix the underlying ECMP hash polarization. MTU changes affect fragmentation, not ECMP distribution. Manual port channel re-pinning is a manual workaround that does not scale and does not fix the root cause.

Question 25

A capacity planning engineer wants to use telemetry data from Cisco Intersight and Nexus Dashboard to predict when the AI cluster will require additional GPU compute. Which combination of metrics BEST drives GPU capacity planning for a growing AI training workload?

Accepted Answer

Effective GPU capacity planning requires GPU-specific metrics: sustained GPU compute utilization (indicating GPUs are constantly at capacity), GPU memory utilization (indicating training jobs are memory-bound and cannot increase batch size), Kubernetes GPU pod pending queue depth (indicating demand exceeds supply), and fabric bandwidth saturation trends from NDI (indicating the network fabric may constrain scaling before GPU count is exhausted). CPU utilization, temperature, and SSH sessions are indirect proxies that do not accurately reflect GPU-specific capacity constraints.

300-640 DCAI — Cisco Data Center AI Infrastructure Specialist