Kubernetes adoption is rapidly growing among enterprises and global cloud builders. The push for digital transformation to modernize businesses for a cloud-native world has lifted both containerized applications and the Kubernetes container orchestration platform to an unprecedented level. The application landscape, especially the enterprise, is looking at Kubernetes to address several key areas:
- Scalability
- High performance
- Availability and Resiliency
- Security and Compliance
- Controllable Costs
- Simplified
The Persistent Storage Question
Enterprise applications such as relational databases, email servers, and even the cloud-native ones like NoSQL, analytics engines, demand a single data source of truth. Fundamentals properties such as ACID (atomicity, consistency, isolation, durability) and BASE (Basic Availability, Soft State, Eventual Consistency) have to have persistent storage as the foundational repository for the data. And thus, persistent storage has rallied under Container Storage Interface (CSI), and fast becoming a de facto standard for Kubernetes. At last count, there are more than 80 CSI drivers from 60+ storage and cloud vendors, each providing block-level storage to Kubernetes pods.
However, at this juncture, Kubernetes is still very engineering-centric. Persistent storage is equally as challenging, despite all the new developments and hype around it.
Kubernetes on-premises
Enterprises are presented with the Kubernetes dilemma. Many enterprise are not ready to do the lift-and-shift of legacy applications to the public clouds. There are enterprise applications developed on containers but faces concerns and risks of security and compliance. They are presented with choices, but these choices are limited to perhaps IBM® Red Hat® OpenShift and Ceph, or VMware® Tanzu with Cloud Foundations. Because Kubernetes takes a degree of technical expertises to scale, the talent resource pool is also limited.
Another option is to take up managed Kubernetes services such as Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS). However, not only does these managed K8s services are in the public cloud, their hybrid persistent storage plays are still very limited.
On-premises containerized enterprise applications running in on-premises Kubernetes pod demanding 100K IOPS with sub-milliseconds response time is almost impossible without the right persistent storage technology for Kubernetes in place.
Criteria for well-managed Kubernetes Persistent Storage
As we shape the requirements for persistent storage for Kubernetes, we list several critical properties as the criteria for well managed persistent storage for Kubernetes.
- On-premises
- Compliant with data policies and governance
- Integration into the internal cybersecurity best practices
- Congruent with established IT processes, practices and policies
- High performant and scalable
- Physical bare metal to reduce layers and overheads
- SSDs and future NVMe ready
- Ability to scale into hundreds of TBs in capacity, and linearly in storage performance
- Durable and resilient
- Highly available, continuous persistent storage service
- Self-healing when a node or service goes down
- Simple
- Minimal changes to existing server, network and storage infrastructure
- Well known protocols like iSCSI
- Managed service
- Closing the Kubernetes skills gap
- Bespoke, white-glove monitoring and support services
- Metered usage and cost control
- Pay as you grow
- Good price/performance and price/feature
The rise of managed bare metals
As Kubernetes evolve, another rapidly rising infrastructure play is running it on bare metal physical servers. The demands of higher performance to address new types of workload such AI, big and deep data analytics and machine learning, along with traditional enterprise applications are fuelling a more integrated and advanced Kubernetes platform that can take advantage of SmartNICs, GPUs and persistent memory technology such as Intel® Optane™.
Whilst there is apparent performance gain, the management and automation of Kubernetes and persistent storage on bare metal also present new and unknown challenges. Thus, using a managed service to operate, manage and monitor the persistent storage to Kubernetes on bare metal is more prudent, rather than jumping through hoops and hurdles to get it right. And the errors and mistakes in operating and supporting persistent storage on bare metal Kubernetes can be costly, especially when in-house skilled and experienced resources are extremely rare. Therefore, having plans to have managed services for persistent storage to serve bare metal Kubernetes platform may be the wisest choice to realize the benefits of scaling enterprise applications, and without the costly errors and risks.
Managing and controlling costs
At the end of all this, regardless about all the feel good points stated, the management of any organization will simply ask “How much is this going to cost over a period of time?“.
The OPEX model is definitely supplanting the traditional CAPEX model. Even so, the costs of building and operating persistent storage for Kubernetes in a cloud and cloud-like model are unpredictable especially when uncertainties abound. Removing unpredictable concerns such as capacity planning and sizing, hardware and performance testing, ensuring data recoverability and protection and many others will clear one of the most challenging obstacles for Kubernetes. This will eventually but surely translate into more controlled and manageable operating costs of persistent, giving the C-levels the best possible advantage in running the business – reducing unpredictability.
Source: Storage Gaga