Exposed Nvidia GPU monitors can reveal AI infrastructure secrets

What happened

Nvidia released a fix for the flaw in May when researchers at Lava Security found thousands of DCGM Exporter instances reachable from the internet, with about a quarter of them exposing the profiling endpoints associated with the vulnerability. A component of Nvidia’s GPU monitoring software that enterprises use to keep tabs on their AI training and inference infrastructure has been vulnerable to denial-of-service (DoS) and information disclosure attacks. NVIDIA is a semiconductor company based in Santa Clara, and its products and services include software.

The Nvidia DCGM Exporter contains an unauthenticated resource exhaustion vulnerability that could allow remote attackers to crash the monitoring service and potentially disrupt AI workloads running on the same host. “Across four scans, those servers reported more than 12,000 unique GPUs, representing an estimated $100 million in hardware.” DCGM Exporter is a telemetry agent Nvidia operates with to collect metrics from Nvidia GPUs and make them available to monitoring systems such as Prometheus.

Key facts

• Nvidia — released: a fix for the flaw in May when researchers at Lava Security — found: thousands of DCGM Exporter instances reachable from the internet, with about a quarter of them exposing the profiling endpoints associated with the vulnerability
• The Nvidia DCGM Exporter — includes: an unauthenticated resource exhaustion vulnerability that could allow remote attackers to crash the monitoring service and potentially disrupt AI workloads running on the same host
• DCGM Exporter is a telemetry agent Nvidia — uses: to collect metrics from Nvidia GPUs and make them available to monitoring systems such as Prometheus