Is Kubernetes Relevant now?

TL: DR — Kubernetes is still relevant and important now, but ‘to whom’ and ‘where in the stack’ matters. With most of the GenAI applications being (or to be) built using hosted foundational models, Kubernetes is irrelevant to GenAI Application Developers, but still relevant to Data Scientists/ AI Engineers, AI Developers, and Platform SREs. The focus of the Kubernetes community should be around making AI Engineers successful and enabling Platform SREs better than on improving/ building foundational models. The community should also see GenAI workloads more than just cloud-native workloads. Read further to learn more on why so.

[Image Credit: ChatGPT]

Introduction

Kubernetes, the vanguard of container orchestration, faces a pivotal moment as Generative AI (GenAI) begins to dominate the tech landscape. In the fast-evolving landscape of GenAI, the relevance of Kubernetes, the de-facto platform for automating the deployment, scaling, and management of containerized applications, is a topic of much discussion. As GenAI continues to reshape industries, driving innovation and streamlining operations, it prompts a reevaluation of the tools and technologies that underpin our digital infrastructure. This blog post delves into the intersection of Kubernetes and GenAI, exploring Kubernetes’ relevance in this new era.

KubeConEU — Quick Summary

KubeConEU, the premier European conference for Kubernetes and Cloud Native technologies, serves as a barometer for the current and future state of Kubernetes. The latest iteration of the conference held 19–22nd March in Paris had a healthy attendance of about 12000 attendees, one of the largest ever KubeCon shows. The show highlighted Kubernetes’ evolving ecosystem, showcasing advancements in security, observability, and automation, along with a growing emphasis on Kubernetes as a foundation for AI and machine learning workflows. The event also spotlighted the Kubernetes community’s efforts to simplify the platform, improve developer experience, and enhance performance and scalability — critical factors in the deployment and management of GenAI applications.

But, like everywhere else, AI captured the attention at the conference — with a lion’s share of keynotes (~ 45% of all keynotes and ALL of the first day’s keynotes all about AI), a dedicated AI Hub, and more. So much of AI that some felt ‘AI sucking the air out’ and other topics such as WASM were not represented well (only ~5% of keynote sessions were dedicated to WASM).

[James Governor’s endearing way of showing the domination of AI during the event]

The vibes within the Kubernetes community at both KubeCon Chicago and KubeCon Paris conferences reminded me of the times how the OpenStack community responded to the Containerization wave. Though the GenAI moment is not an existential threat to Kubernetes as containerization was to OpenStack, a sense of ‘defensive FOMO’ was palpable.

One of the questions I had going into the conference was Is Kubernetes relevant now? This also popped up during the QA session with the media/ analysts — Arun Gupta brought this up as an unasked question and interesting discussions ensued. There was rightful recognition of being more empathetic towards data scientists, making their workflows easier on top of Kubernetes.

Before we explore the relevance of Kubernetes to various personas involved in building GenAI systems, let us take a look at the operational aspects of building GenAI systems and LLMs.

MLOps/ GenAIOps/ LLMOps — A Quick Overview

[Simplified MLOps Workflow, Source: CNCF Cloud Native AI Whitepaper]

  • Machine Learning Operations (MLOps) refers to the tools, technologies and processes used to streamline and automate machine learning models’ deployment, monitoring, and management in production environments at scale.
  • GenAI Operations (GenAIOps) extends MLOps to develop and operationalize Generative AI applications/ solutions.
  • Large Language Model Operations (LLMOps) is a distinct type of GenAIOps focused specifically on developing and productionizing LLM-based solutions. LLMOps principles are equally applicable to SLMs as well.

Essentially, GenAIOps/ LLMOps can be seen as an extension/ adaptation of MLOps to build and operationalize GenAI systems/ foundational models. Though most of the LLMs are built on top of Kubernetes (such as OpenAI, LLama2), there is no standardized workflow yet with such large deployments needing tweaks/ custom implementations (check out NVIDIA’s keynote session at KubeConEU for an example). The following diagram depicts an LLMOps workflow based on an MLOps workflow:

[Source: Navigating the LLMOps landscape: What you need to know | Insight Partners, Oct 2023]

For each phase in the above diagram, multiple options/ offerings are available now — in other words, there is no standardized end-to-end LLMOps available yet. Customers are left to select, or build based on their needs, expertise, and preferences.

One can notice that Kubernetes is the common thread across all the phases above.

Is Kubernetes Relevant in the Era of GenAI and to Whom?

The relevance of Kubernetes in the GenAI era is shifting. To address the relevance of Kubernetes in the GenAI era, it’s essential to consider the unique demands GenAI applications place on infrastructure. GenAI applications, characterized by their need for extensive computational resources, dynamic scaling, and rapid iteration cycles, benefit from the orchestration capabilities of Kubernetes. Kubernetes facilitates the deployment of these applications by managing the complexity of scaling, networking, and storage in a way that’s both efficient and resilient.

A typical MLOps workflow involves multiple personas including Data Scientists, ML Engineers, Data Engineers, Platform Engineers/ SREs, and more. GenAIOps/ LLMOps, being an extension of MLOps, involves similar personas, however, at a much larger scale.

[Cloud Native AI Stack and User Personas, Source: CNCF Cloud Native AI Whitepaper]

The persona of an AI Engineer is an evolution of the persona of an ML Engineer in response to the scale and complexity involved in developing foundational models.

AI Engineer — a quick look

AI Engineers stand at the forefront of the GenAI revolution, crafting intelligent systems and driving innovation. Their work is multi-faceted, encompassing model development, data engineering, experimentation, and deployment. Kubernetes facilitates each of these areas in the following ways:

  • Model Development and Experimentation: AI Engineers benefit from Kubernetes’ ability to manage vast computational resources. They can run multiple training jobs in parallel, rapidly experiment with models, and scale resources as needed without worrying about the underlying infrastructure.
  • Data Engineering: Kubernetes supports complex data pipelines that are essential for AI model training. With its ability to handle persistent storage and stateful applications, AI Engineers can ensure data consistency and quality for training robust AI models.
  • Deployment: When it’s time to move models from development to production, Kubernetes provides a consistent environment for deployment. AI Engineers rely on Kubernetes to handle the scaling of inference services and to manage the lifecycle of AI applications in a production setting.

Platform SREs/Infra Engineers find a steadfast ally in Kubernetes, managing operational demands with finesse. ML/AI Engineers harness its orchestration for model development and deployment. For these personas, Kubernetes is as pertinent as ever and will continue to be so.

However, for GenAI Application Developers, Kubernetes is largely invisible and hence irrelevant. Let us explore the GenAI Stack to see why.

GenAI Stack and Where Kubernetes Is Relevant in the Stack?

The GenAI Stack depicted in the provided image below can be understood as a multilayered architecture, with each tier representing a critical segment of the Gen AI Stack. Please note that this a simplified abstraction common across current implementations of the stack. Given the pace of innovation in this space, this is bound to change/ evolve.

[Relevance of Kubernetes across the GenAI Stack and User Personas]

I have also simplified the user personas used for easy understanding. For the sake of discussions below, here are the definitions of the user personas.

  • Platform Site Reliability Engineer (SRE)**— The role of a Platform SRE is the combination of the roles of Platform Engineer and SRE/ Operator in the diagram from the CNCF whitepaper. These engineers are the stewards of system health, tasked with maintaining the reliability, availability, and performance of the infrastructure. They ensure that the foundational infrastructure (Kubernetes clusters, storage clusters, virtual machines, networking) operate seamlessly, to support the needs of LLMOps.
  • Data Scientist/ AI Engineer*— AI Engineers/ Data Scientists develop foundational models/ LLMs. These professionals are responsible for model development, data engineering, experimentation, and deployment of LLMs. Given the scale of data and compute required for LLM development, I have combined the roles of Data Scientist/ AI Engineer together here for simplicity.*
  • AI Developer*— AI Developers develop GenAI Platforms (as a service/ software). They use the services, and capabilities provided by the LLMOps layer to enable functionality and user experience.*
  • GenAI Application Developer*— GenAI Application Developer builds end-user facing business applications. They consume the capabilities that the GenAI platform provides to build such applications.*

Let’s take a closer look at the GenAI Stack:

  • GenAI Infrastructure (User Persona: Platform SRE) The foundational layer of the GenAI Stack is composed of the critical hardware (compute, storage, networking) resources that power and support model training, experimentation, and serving. It encompasses Compute with GPUs (and CPUs) for varied training and inference, TPUs for tensor computations, and DPUs for data-centric operations. The Storage component ensures data is securely held and easily retrievable, while Networking establishes the fast, reliable connections essential for data transmission and service delivery. This layer could be both on-premises or on cloud.
  • GenAI Ops/ LLMOps (User Persona: Data Scientist/ AI Engineer) This layer provides the foundational models and lifecycle management (CI/ CD) of these models. It includes Data Preparation for cleansing and structuring data sets, Model Training where algorithms learn and adapt from data, Model Serving/ Model Deployment to provide LLM endpoints, Model Monitoring to observe model performance, and Model Governance to uphold standards and ethical practices. LLM Testing / Evaluation ensures large language models meet performance criteria and Prompt Engineering is employed to fine-tune model interactions. Other components include Model Registry, Embeddings (and Vector databases to support embeddings), LLM Chains/ Orchestration, RLHF (Reinforcement Learning from Human Feedback), and LLM API Gateway.
  • GenAI Platform (User Persona: AI Developer) The GenAI Platform provides multi-modal GenAI capabilities as a service or as a software through multiple ways — APIs, Agents, Assistants, Chatbots, Playground, and/ or Copilots. This layer also includes tools for Retrieval-Augmented Generation (RAG) to enhance AI’s informational retrieval capabilities, Fine-Tuning mechanisms to customize model responses, Synthetic Data to address data privacy concerns, and Moderation tools to ensure the appropriateness of content. These capabilities technically fall under both the platform and LLMOps layer, but since they are exposed to end-users, they are included here. Developers utilize SDKs and APIs within this layer to create GenAI Applications.

Popular GenAI Platforms include as-a-service offerings such as OpenAI, Google Gemini, Azure OpenAI, Amazon Bedrock, Perplexity, and Abacus AI, with more such as OctoStack entering the market.

  • GenAI Applications (User Persona: GenAI Application Developer) At the apex of the stack are the GenAI Applications developed using the capabilities enabled by the GenAI Platform to meet specific business needs. Popular GenAI Applications include ChatGPT, Midjourney, DALL-E, Google Bard, the infamous Air Canada chatbot, and more.

The Value Gradient and Relevance of Kubernetes

As one progresses through the layers of the GenAI Stack, the value to the end-user increases — highlighting a clear gradient from operational efficiency to innovative application creation.

Various estimates of Generative AI Market opportunity float around. Bloomberg estimates this to grow from $40B in 2022 to $1.3T by 2032 at a GAGR of 42%. Mckinsey estimates the potential total economic benefits of GenAI to be in the range of $6.1T — $7.9T annually. Compare this with the estimates on cloud native technologies market whose estimates range from $6.5B in 2022 to about $53.6B by 2023 at a CAGR of 23.8% (more such estimates can be found here, here, and here). The estimates of $2.3T shared at the conference is from this report which includes PaaS, Kubernetes, service mesh, DevOps, and app development tools and hence are much higher.

Nevertheless, value is up the stack.

For Platform SREs at the infrastructure level, Kubernetes is critical for container orchestration and resource management.

As we move to GenAI Ops/ LLMOps, Kubernetes is still important. However, Data Scientists and AI Engineers can interact with Kubernetes-managed services indirectly rather than having to manage Kubernetes clusters directly.

Cases in point: Anthropic leveraging EKS or running Meta’s Llama2 on AKS.

At the GenAI Platform and GenAI Applications layers, the complexity of Kubernetes is largely abstracted away. Particularly, GenAI Application Developers work in environments that hide the intricacies of Kubernetes behind user-friendly interfaces, focusing instead on leveraging the platform to deliver sophisticated, value-driven AI applications. For them, Kubernetes is irrelevant.

While Kubernetes is relevant and important for Platform SREs and AI Engineers, it is irrelevant to GenAI Application developers.

Recommendations to the Kubernetes Community

Here are some recommendations to answer the question “Quo Vadis, Kubernetes?” based on the above observations:

  • Make AI Engineers successful!

Kubernetes is the de facto platform for cloud-native workloads. It has enabled developers to be successful by improving developer productivity through simplified deployments, consistent environments, automatic scaling, and self-healing. Kubernetes also supports continuous integration and continuous deployment (CI/ CD) processes further streamlining the development lifecycle. Such an experience for AI Engineers is currently missing. Almost all large scale LLM deployments on Kubernetes have fixes and customizations that need to be brought to Kubernetes. Bringing these fixes into Kubernetes upstream and adding capabilities to make the lives of AI Engineers easier is the need of the hour.

  • GenAI Workloads are not just Cloud-Native Workload, understand their needs better.

Kubernetes has been pivotal in accelerating AI/ ML innovations by providing a consistent, portable, and scalable environment to leverage GPUs. It is not hyperbolical to say that without Kubernetes, the industry would not have reached this stage of innovation in artificial intelligence.

However, GenAI workload is not just another cloud-native workload — it is a platform by itself. While GenAI capabilities will continue to be built on Kubernetes, seeing them just as another cloud-native workload puts the community potentially in a ‘defensive FOMO’ mode.

By treating GenAI/ LLMs as more than just a cloud-native workload, there is opportunity to understand what is needed for GenAI workloads than to assume the requirements of a cloud-native workload.

For example, see the writeup on how OpenAI scaled a single cluster to 7500 nodes. These workloads are semi-stateful, ever changing, need access to all of hardware resources on each node, different type of job scheduling, all at much larger scale, elasticity, throughput, and low latency.

  • Embrace the next platform — the GenAI Platform.

Civil infrastructure such as roads, bridges, railways, public utilities, electrical grid are boring — they don’t get talked about much unless something breaks down. They are not exciting, but they are vital for our survival. Operating systems are the tech equivalent of civil infrastructure. Kubernetes, by being the de facto cloud native platform, is getting there. There is nothing wrong about that.

Linux embraced Kubernetes/ the containerization wave, but that didn’t stop innovations in Linux. It is now time for Kubernetes to embrace the next platform — GenAI with continued focus on enabling Platform SREs (making the platform easier to manage, more secure, scalable, and energy-aware) and renewed focus on making AI Engineers successful.

Summary

Kubernetes’ relevance in the Generative AI (GenAI) era is nuanced, critical for Platform SREs and AI Engineers but less so for GenAI Application Developers due to the abstraction layers of technology stacks.

The evolving GenAI landscape presents an opportunity for the Kubernetes community to adapt and address the unique challenges of GenAI workloads.

By embracing the specific needs of GenAI workloads, Kubernetes can continue to facilitate technological advancement, supporting the development of applications that leverage AI to transform various sectors. The focus should be on making Kubernetes more accessible and efficient for AI Engineers while ensuring its orchestration capabilities are effectively utilized in the GenAI domain, thus maintaining its relevance and importance in the rapidly evolving tech ecosystem.

Disclaimer: The CNCF provided for my travel, accommodation, and conference pass, thank you CNCF!