Building Tiny But Mighty: OpenShift Virtualization on Minimal SNO Clusters

Single Node OpenShift promised an elegant solution: full OpenShift capabilities in a minimal footprint, perfect for labs, edge sites, and home setups. But the moment you add operators, especially OpenShift Virtualization, you realize the hidden cost: memory contention. Every service fights for the same finite pool. We discovered that most of OpenShift's default components are unnecessary for virtualization-only workloads, and that Cluster Capabilities lets us strip away the excess. Here's how we built a lean, functional OpenShift Virtualization cluster on SNO 4.21, configured HostPathProvisioner for storage, and deployed a VM with minimal resource overhead.

Reducing the Cluster Footprint at Install Time

Cluster capabilities are optional built-in components that can be enabled or disabled at install time via the capabilities section of install-config.yaml. Two fields control what gets installed: baselineCapabilitySet, a named set (None, v4.11 through v4.18, or vCurrent) that selects a starting group of operators, and additionalEnabledCapabilities, a list of individual capabilities to layer on top. The default baseline is vCurrent, which enables everything. Setting the baseline to None and adding back only what you need is the most effective way to shrink the operator footprint before the cluster ever boots.

One important constraint: capabilities can be enabled post-install, but they cannot be removed once enabled. That makes the baseline choice a one-time decision. For a minimal SNO cluster, starting from None and selectively enabling only what your workloads require keeps idle memory pressure low from day one.

Prerequisites

You need a running SNO 4.21 cluster installed with the following capabilities configuration in install-config.yaml:

capabilities:
  baselineCapabilitySet: None
  additionalEnabledCapabilities:
  - OperatorLifecycleManager
  - Console
  - Ingress

This starts from an empty baseline and enables only OLM (required to install operators), the web console, and the cluster ingress. For the full list of available capabilities and their dependencies, see the Cluster capabilities documentation.

Note: Ingress is not listed as a named Cluster Capability in the table above or in the OpenShift Documentation, however, as of OCP 4.18 and later it must be explicitly included in additionalEnabledCapabilities. Omitting Ingress will result in the following error when performing the installation:

INFO Configuration has 1 master replicas, 0 arbiter replicas, and 0 worker replicas FATAL failed to fetch Agent Manifests: failed to load asset "Install Config": invalid install-config configuration: capabilities: Invalid value: {"baselineCapabilitySet":"None","additionalEnabledCapabilities":["OperatorLifecycleManager","Console"]}: the Ingress capability is required

From my testing, deploying these three Cluster Capabilities will result in an SNO using around 9,500Mi of RAM with ~65 Running Pods.

Verify all cluster operators are healthy before proceeding:

oc get co

Enable the Red Hat Operator Catalog

Without the Marketplace capability enabled, there are no catalog sources configured. Without a catalog source, OLM has no index to resolve packages from.

To resolve this issue we will manually create the openshift-marketplace namespace and then create the Red Hat catalog.

Save the following as ns-openshift-marketplace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-marketplace
spec: {}
status: {}

Apply the manifest by running:

oc apply -f ns-openshift-marketplace.yaml

Save the following as cs-redhat-catalogsource.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operators
  namespace: openshift-marketplace
spec:
  displayName: Red Hat Operators
  image: registry.redhat.io/redhat/redhat-operator-index:v4.21
  publisher: Red Hat
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 12h

Apply the manifest by running:

oc apply -f cs-redhat-catalogsource.yaml

Wait until the source is ready before continuing:

oc get catalogsource redhat-operators -n openshift-marketplace \
  -o jsonpath='{.status.connectionState.lastObservedState}'
# Expected: READY

Now that we have the catalog configured, we can move on to installing OpenShift Virtualization.

Installing OpenShift Virtualization

OpenShift Virtualization is delivered via the kubevirt-hyperconverged OLM package. Three resources bootstrap the install: a Namespace, an OperatorGroup, and a Subscription.

Save the following as openshift-virtualization.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.21.6
  channel: stable

Apply the manifest by running:

oc apply -f openshift-virtualization.yaml

OLM automatically creates and approves an InstallPlan. Once the CSV reaches Succeeded, the HCO operator pod is running and waiting for its configuration.

Minimizing your OpenShift Virtualization Footprint

The default HCO configuration is designed for multi-node clusters with HA in mind. On a SNO cluster, several of those defaults waste memory that your VMs need. Thankfully, when deploying to a SNO 4.21 cluster, the infra pods are automatically scaled down to 1.

You will see the following events in your web console for the virt-exportproxy, virt-controller, and virt-api.

applying custom number of infra replica. this is an advanced feature that prevents auto-scaling for core kubevirt components. Please use with caution!

Disable Automatic Golden Image Imports

As soon as the HyperConverged CR is applied, SSP's DataImportCron controller starts importing golden OS images (Fedora, CentOS Stream 9/10, RHEL 8/9/10) into the openshift-virtualization-os-images namespace. Each image claims a 30 Gi PVC, and the importer pods consume additional memory. On a node with limited storage, this exhausts capacity quickly.

If you do not plan on using the Common Boot Images and wish to save memory used by those pods, then be sure to uncheck enableCommonBootImageImport when configuring your HCO.

Configuring HostPathProvisioner

HCO deploys the hostpath-provisioner-operator automatically. You just need to give it a path on the node to use and create the HostPathProvisioner CR to activate it.

A secondary disk is recommended but not required. If you plan to run real workloads, it is worth considering the impact of placing VM disk images on the same device that etcd writes to. etcd is sensitive to storage latency, and heavy VM I/O on a shared disk can introduce the kind of fsync delays that cause etcd to log apply took too long messages and leader is overloaded likely from slow disk warnings, and can ultimately result in API timeouts, context deadline exceeded errors, and pod restarts. A dedicated disk eliminates that contention entirely.

Preparing the Dedicated Disk

If you have a dedicated disk for HostPathProvisioner storage, use a MachineConfig to format and mount it before configuring HPP. If not, skip ahead to Deploy the HostPathProvisioner.

Two systemd units do the work: one formats the disk as XFS on first boot (skipping the format step if a filesystem already exists), and one mounts it persistently at /var/mnt/hostpath. Because this goes through MachineConfig, it survives node reboots and reprovisioning without manual intervention.

Replace /path/to/disk with your actual device path (for example, /dev/nvme0n1p1) before applying.

Save the following as mc-hostpath-mount.yaml:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 50-master-hostpath-mount
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Format /path/to/disk for hostpath provisioner
          Before=var-mnt-hostpath.mount
          DefaultDependencies=no
          ConditionPathExists=/path/to/disk

          [Service]
          Type=oneshot
          RemainAfterExit=yes
          ExecStart=/bin/bash -c 'blkid /path/to/disk | grep -q TYPE || /usr/sbin/mkfs.xfs /path/to/disk'

          [Install]
          WantedBy=local-fs.target
        enabled: true
        name: format-hostpath.service
      - contents: |
          [Unit]
          Description=Mount /path/to/disk at /var/mnt/hostpath
          After=format-hostpath.service
          Requires=format-hostpath.service

          [Mount]
          What=/path/to/disk
          Where=/var/mnt/hostpath
          Type=xfs
          Options=defaults

          [Install]
          WantedBy=local-fs.target
        enabled: true
        name: var-mnt-hostpath.mount

Apply the manifest by running:

oc apply -f mc-hostpath-mount.yaml

The MachineConfig triggers a rolling reboot of the control plane node. On SNO that means a single node reboot. Wait until the node returns to Ready before continuing:

oc get nodes -w

Once the node is back, confirm the disk is mounted at /var/mnt/hostpath:

oc debug node/<node-name> -- chroot /host findmnt /var/mnt/hostpath
TARGET                SOURCE        FSTYPE OPTIONS
/var/mnt/hostpath     /path/to/disk xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

Deploy the HostPathProvisioner

The HostPathProvisioner CR tells the operator which directory on the node to expose as storage. It maps a named storage pool to the mount point prepared in the previous section. The HPP daemonset then runs on the node and watches that directory, dynamically provisioning subdirectories as PVCs are requested.

Save the following as hostpath-provisioner.yaml

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  storagePools:
    - name: data-pool
      path: /var/mnt/hostpath
  workload:
    nodeSelector:
      kubernetes.io/os: linux

Apply the manifest by running:

oc apply -f hostpath-provisioner.yaml

Wait for the HPP daemonset to become available:

oc get hostpathprovisioner hostpath-provisioner \
  -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'
# Expected: True

Create the StorageClass

The Cluster Storage Operator sets OpenShift Container Platform cluster-wide storage defaults. It ensures a default storageclass exists for OpenShift Container Platform clusters. It also installs Container Storage Interface (CSI) drivers which enable your cluster to use various storage backends.

Since we are using the HostPathProvisioner for storage, we opted out of enabling the Storage operator on this cluster and are configuring the StorageClass manually.

Therefore, once the HPP daemonset is running, create a StorageClass that binds to the data-pool storage pool and is marked as the cluster default.

Save the following as sc-hpp.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-provisioner
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagePool: data-pool

Apply the manifest by running:

oc apply -f sc-hpp.yaml

volumeBindingMode: WaitForFirstConsumer defers PVC binding until a pod is scheduled, which is the correct behaviour on a single-node cluster where all storage is local.

Creating a Virtual Machine

With storage in place, you can create a VM. This example uses a DataVolume to import the Fedora 41 cloud image from quay.io/containerdisks/fedora:41 into a 20 Gi PVC backed by the hostpath-provisioner StorageClass. The disk is persistent: it survives VM restarts and deletion of the VM object only removes the VM definition, not the volume.

cloud-init configures the fedora user and enables SSH password authentication. A NodePort Service exposes port 22 on the node IP.

Save the following as fedora-vm.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: vms
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora
  namespace: vms
spec:
  runStrategy: Always
  dataVolumeTemplates:
    - metadata:
        name: fedora-disk
      spec:
        storage:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
          storageClassName: hostpath-provisioner
        source:
          registry:
            url: docker://quay.io/containerdisks/fedora:41
  template:
    metadata:
      labels:
        kubevirt.io/domain: fedora
    spec:
      domain:
        cpu:
          cores: 2
        memory:
          guest: 1024Mi
        devices:
          disks:
            - name: fedora-disk
              disk:
                bus: virtio
            - name: cloudinit
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: fedora-disk
          dataVolume:
            name: fedora-disk
        - name: cloudinit
          cloudInitNoCloud:
            userData: |
              #cloud-config
              password: fedora
              chpasswd:
                expire: false
              ssh_pwauth: true
---
apiVersion: v1
kind: Service
metadata:
  name: fedora-ssh
  namespace: vms
spec:
  type: NodePort
  selector:
    kubevirt.io/domain: fedora
  ports:
    - name: ssh
      port: 22
      targetPort: 22
      protocol: TCP

Apply the manifest by running:

oc apply -f fedora-vm.yaml

SSH Access

Once the VMI phase reaches Running, find the assigned NodePort and connect:

oc get svc fedora-ssh -n vms
ssh fedora@<node-ip> -p <nodeport>
# password: fedora

Conclusion

Getting OpenShift Virtualization running on a minimal SNO cluster is less about fighting the platform and more about knowing which defaults to opt out of. Setting baselineCapabilitySet: None at install time is the single highest-leverage decision as it keeps a dozen operators from consuming memory your VMs need before you've even logged in. From there, manually wiring up the Red Hat catalog, disabling golden image imports, and pointing HostPathProvisioner at a local directory gets you to a functional virtualization platform without anything you didn't ask for.

The setup here is intentionally bare. Just a node, a directory, and KubeVirt doing what it does well. That's enough to run real workloads, iterate on VM configurations, and learn the OpenShift Virtualization API without the overhead of a full cluster humming in the background.

An added benefit of starting from None is that you retain the option to grow. Capabilities can be enabled at any time post-install, so you can selectively add what you need:

oc patch clusterversion/version --type merge -p \
  '{"spec":{"capabilities":{"additionalEnabledCapabilities":["openshift-samples","marketplace"]}}}'

Or enable the full default baseline in one shot:

oc patch clusterversion version --type merge -p \
  '{"spec":{"capabilities":{"baselineCapabilitySet":"vCurrent"}}}'

Just keep in mind that this is a one-way door and once a capability is enabled or the baseline is raised, it cannot be reverted.