Skip to content

GPU Resource Management

GPU Resource Management in Kubernetes: From Extended Resource to DRA

This blog is part of our DRA series, continuing from our earlier posts: Introduction to DRA, Enabling DRA with Kind, and MIG with DRA . This post focuses on pre-DRA vs post-DRA GPU management on Rafay upstream Kubernetes clusters.

Overview

With the rise of AI, ML, and HPC workloads, GPU resource management has become a cornerstone of Kubernetes scheduling. Over time, Kubernetes has evolved from static, count-based GPU allocation using extended resources (nvidia.com/gpu) to the more flexible DRA framework now a stable feature in Kubernetes v1.34.

This guide walks through the evolution from pre DRA GPU management to DRA-based allocation and sharing, complete with examples.

Pre-DRA: GPU Management Using Extended Resources

Before DRA, Kubernetes workloads used the NVIDIA Device Plugin to expose GPUs as extended resources. These resources could then be requested by pods just like CPU or memory.

GPU Operator Components

To enable GPU scheduling, the NVIDIA GPU Operator packaged all required components:

  • Host components:

    • NVIDIA GPU driver

  • Kubernetes components:

    • NVIDIA device plugin
    • MIG Manager
    • DCGM Exporter
    • GPU Feature Discovery (GFD)

Each of these components was deployed as a DaemonSet on GPU nodes, ensuring the scheduler could detect and allocate GPU resources properly.

Requesting GPUs (Pre-DRA)

Here's an example of how users would request GPU access before DRA:

apiVersion: v1
kind: Pod
metadata:
  name: pod-gpu-classic
spec:
  containers:
    - name: app-container
      image: nvidia/cuda
      resources:
        limits:
          nvidia.com/gpu: 2

This tells Kubernetes to assign two GPUs to the container. The scheduler and device plugin work together to: - Locate a node with at least two available GPUs - Schedule the pod there - Inject the GPU devices into the container

If a specific GPU type was needed (e.g., A100-40GB), node labels and selectors could be used to ensure the pod landed on the right hardware.

Post-DRA: Dynamic Resource Allocation

Kubernetes v1.34 introduces DRA, which graduated to GA - a new, flexible, and vendor extendable approach for requesting resources such as GPUs.

Why DRA?

DRA addresses key limitations of the old model: - Enables fine grained GPU sharing providing complext constraints when requesting a GPU. - Allows custom APIs and parameters from vendors - Supports better isolation and resource reusability - Makes cross-pod sharing possible through claims

Requesting GPUs with DRA

Instead of requesting GPUs via a simple count (nvidia.com/gpu: 2), DRA introduces three main objects: 1. DeviceClass –Defines a category of devices that can be claimed and how to select specific device attributes in claims. 2. ResourceClaimTemplate – defines how a claim should be created 3. ResourceClaim – represents an actual allocated resource

Here's the DRA equivalent for requesting two GPUs:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: gpu
  name: multiple-gpus
spec:
  spec:
    devices:
      requests:
      - name: gpu-1
        exactly:
          deviceClassName: gpu.example.com
      - name: gpu-2
        exactly:
          deviceClassName: gpu.example.com
---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu
  name: pod0
  labels:
    app: pod
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: gpus
  resourceClaims:
  - name: gpus
    resourceClaimTemplateName: multiple-gpus

example1

This example shows 1 pod with 1 container requesting 2 GPUs using the DRA mechanism.

kubectl  get pods -n gpu

NAME   READY   STATUS    RESTARTS   AGE
pod0   1/1     Running   0          81s

logs of this pod to verify that GPUs were allocated to them 

kubectl  logs -f -n gpu          pod0
declare -x DRA_RESOURCE_DRIVER_NAME="gpu.example.com"
declare -x GPU_DEVICE_3="gpu-3"
declare -x GPU_DEVICE_3_RESOURCE_CLAIM="c2d2a1d2-b52b-4b4c-b5ae-c1cace625493"
declare -x GPU_DEVICE_3_SHARING_STRATEGY="TimeSlicing"
declare -x GPU_DEVICE_3_TIMESLICE_INTERVAL="Default"
declare -x GPU_DEVICE_4="gpu-4"
declare -x GPU_DEVICE_4_RESOURCE_CLAIM="c2d2a1d2-b52b-4b4c-b5ae-c1cace625493"
declare -x GPU_DEVICE_4_SHARING_STRATEGY="TimeSlicing"
declare -x GPU_DEVICE_4_TIMESLICE_INTERVAL="Default"
declare -x HOME="/root"
declare -x HOSTNAME="pod0"
declare -x KUBERNETES_NODE_NAME="mks-demo"
declare -x KUBERNETES_PORT="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP_ADDR="10.96.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="10.96.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/"
declare -x SHLVL="1"
  • You have two device envs: GPU_DEVICE_3="gpu-3" and GPU_DEVICE_4="gpu-4".

Controlled GPU Sharing with DRA

One of DRA's most powerful capabilities is controlled GPU sharing, which allows multiple containers or pods to access the same GPU safely.

1. Intra-Pod GPU Sharing (Multiple Containers in One Pod)

Multiple containers within the same pod can reference a single ResourceClaim, giving them shared access to the same GPU.

---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: gpu0
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com

---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu0
  name: pod0
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  - name: ctr1
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  resourceClaims:
  - name: shared-gpu
    resourceClaimTemplateName: single-gpu

example1

This example shows 1 pod with 2 containers sharing 1 GPU in time using the Dynamic Resource Allocation (DRA) mechanism.

kubectl  get pods -n gpu0
NAME   READY   STATUS    RESTARTS   AGE
pod0   2/2     Running   0          9s

kubectl  logs -f -n  gpu0 pod0 -c ctr0
declare -x DRA_RESOURCE_DRIVER_NAME="gpu.example.com"
declare -x GPU_DEVICE_3="gpu-3"
declare -x GPU_DEVICE_3_RESOURCE_CLAIM="99944ed0-806f-496f-bdb5-e457b6a66a2d"
declare -x GPU_DEVICE_3_SHARING_STRATEGY="TimeSlicing"
declare -x GPU_DEVICE_3_TIMESLICE_INTERVAL="Default"
declare -x HOME="/root"
declare -x HOSTNAME="pod0"
declare -x KUBERNETES_NODE_NAME="mks-demo"
declare -x KUBERNETES_PORT="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP_ADDR="10.96.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="10.96.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/"
declare -x SHLVL="1"
root@mks-demo:/home/demo/dra-example-driver# kubectl  logs -f -n  gpu0 pod0 -c ctr1
declare -x DRA_RESOURCE_DRIVER_NAME="gpu.example.com"
declare -x GPU_DEVICE_3="gpu-3" # ← HIGHLIGHTED
declare -x GPU_DEVICE_3_RESOURCE_CLAIM="99944ed0-806f-496f-bdb5-e457b6a66a2d"
declare -x GPU_DEVICE_3_SHARING_STRATEGY="TimeSlicing"
declare -x GPU_DEVICE_3_TIMESLICE_INTERVAL="Default"
declare -x HOME="/root"
declare -x HOSTNAME="pod0"
declare -x KUBERNETES_NODE_NAME="mks-demo"
declare -x KUBERNETES_PORT="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP="tcp://10.96.0.1:443"
declare -x KUBERNETES_PORT_443_TCP_ADDR="10.96.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="10.96.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
declare -x PWD="/"
declare -x SHLVL="1"
  • GPU_DEVICE_3="gpu-3" is present in both ctr0 and ctr1

2. Inter-Pod GPU Sharing (Global Claim Across Pods)

You can create a global ResourceClaim and reference it across multiple pods — ideal for workloads that need coordinated access (like shared inference).

---
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  namespace: gpu1
  name: single-gpu
spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com

---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu1
  name: pod0
  labels:
    app: pod
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  resourceClaims:
  - name: shared-gpu
    resourceClaimName: single-gpu

---
apiVersion: v1
kind: Pod
metadata:
  namespace: gpu1
  name: pod1
  labels:
    app: pod
spec:
  containers:
  - name: ctr0
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["export; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: shared-gpu
  resourceClaims:
  - name: shared-gpu
    resourceClaimName: single-gpu

example1

This example shows 2 pods with 1 container each sharing 1 GPU in time using the Dynamic Resource Allocation mechanism.

Conclusion

The move from extended resources to Dynamic Resource Allocation represents a major leap in how Kubernetes manages GPUs and other accelerators.

DRA brings flexibility, fine-grained control, and vendor extensibility — making it the future of GPU scheduling in Kubernetes. Whether you're enabling fractional GPU usage, managing shared inference workloads, or defining custom device policies, DRA unlocks new capabilities that were never possible before.