Scaling AI Infrastructure: How Together AI is Tackling 'Day 2' GPU Cluster Operations

As large language model training scales to thousands of GPUs, the primary bottleneck for AI engineering teams is shifting from raw hardware acquisition to "day 2" operational reliability. In a recent update detailing new reliability and control features for its GPU clusters, Together AI highlights a critical industry pivot: standard cloud-native tooling is insufficient for large-scale AI orchestration, requiring purpose-built solutions for hardware failures, scheduler drift, and identity management.

The Reality of Scale: Passive Monitoring and Node Repair

The operational reality of running multi-day training jobs on massive GPU clusters is that hardware and schedulers will inevitably fail. GPUs drop off the PCIe bus, Xid errors take nodes offline, and thermal throttling can silently degrade job throughput. Together AI's update acknowledges that raw uptime metrics are less relevant than the speed of failure detection and the cleanliness of recovery. To address this, the company has introduced passive health checks that run continuously across cluster nodes. Unlike active synthetic tests run during provisioning, these passive checks monitor live workloads, logs, and metrics to identify degradation-such as thermal throttling or node drains-as it happens. Together AI claims this monitoring introduces near-zero overhead on running jobs.

Detection is coupled with a human-in-the-loop automated node repair system. When a failure signature is identified, the system recommends one of four remediation actions: an in-place Reboot for transient issues, a Reprovision to rebuild the node from a clean image, a Failover to move the workload to fresh bare-metal, or a Remove action for hardware requiring a Return Merchandise Authorization (RMA). By requiring operator approval before execution, the system balances automated diagnostics with the safety necessary to protect high-value training checkpoints and inference replicas from accidental deletion during automated drains.

Re-architecting Slurm on Kubernetes

Running Slurm-the de facto standard for high-performance computing workload management-on top of Kubernetes has historically been fraught with friction. Kubernetes was designed for ephemeral, stateless microservices, while Slurm expects static, stateful, and tightly coupled hardware environments. This impedance mismatch frequently results in crashing daemons, zombie processes, and scheduler drift. To mitigate these scale-at-failure headaches, Together AI has rebuilt its Slurm-on-Kubernetes stack based on a fork of the open-source Slinky project.

This architectural overhaul addresses several highly specific failure modes that plague AI infrastructure operators. First, it introduces self-healing worker daemons that auto-restart in place during transient failures, preserving long-running training sessions. Second, it tackles the persistent issue of zombie processes. Previously, daemonized child processes could escape Slurm's visibility upon job completion, holding GPU memory and /dev/shm segments hostage and forcing manual cleanup. The new stack implements kernel-level tracking of all job descendants, ensuring complete resource reclamation when a job ends.

Additionally, the update resolves critical state and accounting vulnerabilities. The Slurm sacct history, which previously resided on ephemeral storage and was vulnerable to pod restarts, has been migrated to durable, PVC-backed storage. This ensures that billing reconciliation and post-hoc analysis survive cluster lifecycle events. Furthermore, the stack now rebuilds Slurm's view of available GPUs fresh on every node start, eliminating the scheduler drift that occurs when stale GPU identifiers mismatch real hardware after a pod reschedule.

Enterprise Implications: Identity and Lifecycle Management

Beyond reliability, Together AI is positioning its platform as an enterprise-grade alternative to raw hyperscaler GPU rentals by addressing access and lifecycle management. As AI teams expand, the common practice of sharing a single admin kubeconfig becomes a severe security and operational liability. To rectify this, the platform now supports External OIDC for Kubernetes RBAC, integrating with identity providers such as Google, Okta, Auth0, and Microsoft Entra ID. This integration allows operators to enforce per-user audit trails, immediate revocation, and least-privilege access through standard Kubernetes ClusterRoleBindings, effectively reducing the admin kubeconfig to a break-glass utility.

Operational control is further enhanced by the introduction of lifecycle-triggered startup scripts. Operators can now define shell scripts that execute at specific cluster lifecycle events: at node boot for package installation, at job start for staging data and preparing scratch space, and at job end for cleanup and downstream notifications. By moving these configurations into self-serve, validated cluster definitions, Together AI eliminates the need for manual SSH interventions and support tickets, streamlining the deployment of custom monitoring agents and internal tooling.

Limitations and Open Architectural Questions

While the updates represent a significant maturation of Together AI's infrastructure offerings, several technical details remain opaque. The claim of "near-zero overhead" for passive health checks lacks specific performance metrics. In highly optimized distributed training environments, even minor CPU or network overhead from continuous log and metric observation can impact synchronization across thousands of GPUs.

Furthermore, the mechanics of the "Failover" repair action require deeper scrutiny. The documentation notes that this action moves the workload to a fresh bare-metal node and "clears local data." It remains unclear how the system manages the migration of state-such as weights currently in memory or checkpoints stored on local NVMe drives-without interrupting the training loop or requiring a complete rollback to the last distributed checkpoint. Finally, the exact architecture of the Slinky open-source fork, specifically how it bridges the fundamental state management differences between Kubernetes and Slurm, is not fully detailed, leaving questions about its compatibility with custom Kubernetes operators or advanced networking configurations.

The Maturation of AI Infrastructure

Together AI's latest cluster updates underscore a critical evolution in the AI hardware market. As the initial scramble for raw GPU availability subsides, the competitive frontier is shifting toward operational efficiency and enterprise compliance. By systematically addressing the painful, highly specific failure modes of large-scale AI orchestration-from kernel-level process leaks to identity management-Together AI is signaling that providing compute is no longer enough. The vendors that will dominate the next phase of AI infrastructure will be those that can successfully abstract the inherent brittleness of hybrid scheduling environments, allowing engineering teams to focus on model architecture rather than infrastructure triage.

Key Takeaways

• Together AI has upgraded its GPU clusters with passive health checks and a human-in-the-loop automated node repair system to address inevitable hardware failures at scale.
• A rebuilt Slurm-on-Kubernetes stack, based on an OSS Slinky fork, resolves persistent orchestration issues like zombie processes, scheduler drift, and ephemeral accounting loss.
• The introduction of External OIDC for Kubernetes RBAC eliminates the security risks of shared admin kubeconfigs, signaling a push toward enterprise-grade compliance.
• Questions remain regarding the specific performance overhead of passive monitoring and the technical mechanics of state management during bare-metal failovers.