The AI infrastructure market is entering a new phase. Early momentum centered on acquiring compute power—specifically GPU clusters—but enterprises are discovering that ownership alone doesn’t deliver value. The real challenge lies in transforming raw hardware into usable infrastructure that developers can consume without friction.
The 20x Gap Between Hardware and Service
Consider the economics: renting an Nvidia H100 GPU costs approximately $1.60 per hour. Yet accessing that same capacity through a hyperscaler’s AI model service runs $7 to $10 per hour, a 20X markup. This differential represents a significant opportunity for enterprises with their own infrastructure but capturing that value requires more than procurement.
The problem is execution complexity. Taking infrastructure “from the ground and running is an order of magnitude harder than people can tolerate.” Enterprises must deploy orchestration software, configure networking components, and operationalize specific models. At large financial institutions, developers reportedly spend 20-25% of their time managing infrastructure details—Terraform configurations, networking issues, and platform maintenance. The GPU hardware becomes a sunk cost without the sophisticated platform layer needed to make it accessible.
What Hyperscalers Understood First
Major cloud providers built their businesses on a fundamental insight: customers pay for managed services, not raw compute. IDC data shows that less than 20% of hyperscaler revenue comes from Infrastructure-as-a-Service. The bulk comes from specialized services like AWS Bedrock or SageMaker. These platforms succeed by making complex operations feel effortless—provisioning resources becomes self-service rather than ticket-based.
The hyperscaler model works because it abstracts infrastructure complexity behind automation and governance layers that took years and substantial engineering investment to build.
Rafay’s Infrastructure Automation Layer
At a recent AI Infrastructure Field Day, Rafay presented its approach to this challenge. The company positions itself as an AI cloud enabler—providing the platform layer that sits between raw infrastructure and developer consumption. Founded on the principle of self-service compute access, Rafay targets large enterprises, including those with sovereign AI requirements, who need to deliver cloud-like services using their own infrastructure.
The platform addresses what Rafay identifies as its primary competitor: the in-house approach where organizations build custom infrastructure orchestration. These internal systems often become complex and difficult to maintain. Rafay offers an alternative by automating the AI stack and extending the reach of platform engineering teams.
Core Platform Capabilities
- Multi-Tenancy and Self-Service: Rafay implements secure multi-tenancy by managing infrastructure controls: VM creation, public IP addressing, virtual routing and forwarding configurations, and network segmentation including east-west traffic controls. This architecture allows multiple teams to share GPU clusters securely without privilege escalation risks.
- Standardization Across Hybrid Environments: The platform standardizes deployments whether infrastructure runs on vSphere on-premises or managed Kubernetes services in the cloud. Built-in governance tracks GPU resource allocation—critical when these assets are finite and expensive. Policy management integrates with identity providers for authentication and can enforce workflows like mandatory ticketing before resource provisioning.
- Application-Centric Delivery: Rather than exposing bare infrastructure, Rafay presents applications through a catalog: Jupyter Notebooks, Nvidia Inference Microservices, Model-as-a-Service endpoints. The platform automatically provisions supporting infrastructure—namespaces, virtual clusters, DNS entries, allowing developers to begin work immediately.
- Cost Capture: By enabling enterprises to run AI services on proprietary hardware, Rafay helps organizations capture the cost differential between hyperscaler services and self-operated infrastructure.
Deployment Models and Constraints
Rafay operates as a software company without owned infrastructure. Enterprises choose between two deployment models:
- Smaller organizations typically use Rafay’s SOC2-compliant Software-as-a-Service offering, which runs on AWS or Oracle Cloud Infrastructure. Larger enterprises concerned with sovereignty deploy Rafay’s controller within their own data centers for air-gapped operation and complete control.
- For heterogeneous or disconnected environments, Rafay deploys an agent locally that communicates with the control plane, using Software-Defined Perimeter concepts to secure remote management without requiring firewall changes. Organizations with existing infrastructure-as-code configurations can integrate the IaC into the Rafay AI infrastructure environment, though migration requires conversion effort.
The Real Challenge in AI Infrastructure
The current bottleneck in enterprise AI isn’t GPU availability—it’s building governance and automation layers that transform hardware into a production platform. Enterprises without hyperscaler-level engineering resources face substantial time and cost investments to construct these capabilities internally.
Rafay’s value proposition centers on providing this missing layer: automation that standardizes infrastructure, enables secure resource sharing, and delivers AI capabilities through self-service interfaces. For organizations seeking to capitalize on GPU investments, accelerate AI application delivery, and maintain control of their infrastructure, Rafay offers a path to operationalizing AI infrastructure without years of internal development.
Dreamforce 2025 Developer Impact and the Rise of the Agentic Enterprise
The AI Infrastructure Paradox: When Buying GPUs Is Just the Beginning
Databricks’ Mooncake Move and OpenAI Pact
Developers Face Growing Consumer Distrust Over AI Code Security
Google Cloud Advances AI and Sovereign Infrastructure Across Public Sector
