多平台部署minikube+kubevirt


多平台部署minikube+kubevirt

[TOC]

在不同的平台快速启动研发环境.

考虑到平台支持的情况,打算使用minikube+kubevirt.

目前在 x86 红帽 REHL 8 windows 10 macOS Monterey测试部署过.

对比

kubevirt

后Kubernetes时代的虚拟机管理技术之kubevirt篇

cloud-init

安装cloud-init (aliyun.com) cloud-init 只支持安装在linux系统上.

virtlet

KubeVirt vs Virtlet: Performance Guide

KubeVirt is a virtual machine management add-on for Kubernetes providing control of VMs as Kubernetes Custom Resources. Virtlet, on the other hand, is a CRI (Container Runtime Interface) implementation, which means that Kubernetes sees VMs in the same way it sees Docker containers.

Virtlet is a CRI implementation, so all VMs are defined as Kubernetes Pods and treated as first-class citizens, so to speak. The advantage of this architecture is that anything you can do with Pods can be done with Virtlet VMs, right out of the box.

但是virtle只支持linux平台,并且最后一次提交是2019年.

而kubevirt + minikube 可以在支持嵌套虚拟化的平台上运行.

安装minikube

minikube 虚拟化驱动

具体介绍可以看官网文档:minikube drive

  • 可以在linux平台使用kvm逻辑安装
  • 在windows 10 平台,开启hyperv嵌套虚拟化,安装部署成一台hyperv虚拟机

    //开启hyperv嵌套虚拟化,进去minikube虚拟机可以看到它支持虚拟化.
    PS C:\WINDOWS\system32> minikube.exe ssh
                           _             _
              _         _ ( )           ( )
    ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
    /' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
    | ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
    (_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
    
    $ lsmod |grep kvm
    kvm_intel             233472  3
    kvm                   671744  1 kvm_intel
    irqbypass              16384  8 kvm
    $ exit
    
  • 在macos monterey上,使用vmware fusion2 开启嵌套虚拟化,安装部署成一台虚拟化.

点击下载安装minikube => minikube start | minikube (k8s.io)

安装完成后, 启动集群会自动,下载相应的镜像

linux

在linux平台建议裸机部署, 尝试过使用kvm驱动部署, 虽然开启了嵌套虚拟化,最后, kubevirt 启动虚拟机失败.

  1. 裸机安装需要先按照docker-ce, 按照官网安装
  2. 如果是REHL 8 平台需要修改docker repo中[docker-ce-stable] 内容为centos .

    [wjh@node1 ~]$ cat /etc/yum.repos.d/docker-ce.repo |grep -ie '\[docker-ce-stable\]' -A 6
    [docker-ce-stable]
    name=Docker CE Stable - $basearch
    baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
    enabled=1
    gpgcheck=1
    gpgkey=https://download.docker.com/linux/centos/gpg
    
  3. 启动docker
    // 考虑到不要使用root权限启动minikube, 需要给当前用户wjh,赋权执行docker 命令
    sudo usermod  -aG docker wjh
    // exit 重新ssh登录账号.
    
    // 启动docker
    sudo systemctl start docker.service
    sudo systemctl status docker.service
    
  4. 启动minikube集群
    // driver none 代表裸机安装
    minikube start --driver=none
    
    // 查看集群状态
    minikube status
    

windows

管理员身份启动powershell

  • powershell中设置路径, 启动集群,自动下载驱动.
    PS C:\WINDOWS\system32> $oldPath = [Environment]::GetEnvironmentVariable('Path', [EnvironmentVariableTarget]::Machine)
    PS C:\WINDOWS\system32> if ($oldPath.Split(';') -inotcontains 'C:\minikube'){ `
    >>   [Environment]::SetEnvironmentVariable('Path', $('{0};C:\minikube' -f $oldPath), [EnvironmentVariableTarget]::Machine) `
    >> }
    
  • 在windows平台使用hyperv驱动,minikube集群会被安装到hyperv的一台虚拟机上. 安装完成之后, 需要关闭集群,设置该虚拟机支持嵌套虚拟化.
    PS C:\WINDOWS\system32> minikube start --hyperv-virtual-switch=minikube_switch --driver=hyperv
    
    * Microsoft Windows 10 Enterprise 10.0.19044 Build 19044 上的 minikube v1.24.0                                          
    * 根据用户配置使用 hyperv 驱动程序
    * 正在下载 VM boot image...
      > minikube-v1.24.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
      > minikube-v1.24.0.iso: 225.58 MiB / 225.58 MiB [ 100.00% 11.14 MiB p/s 20s
    * Starting control plane node minikube in cluster minikube
    * Downloading Kubernetes v1.22.3 preload ...
      > preloaded-images-k8s-v13-v1...: 501.73 MiB / 501.73 MiB  100.00% 11.07 Mi
    * Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
    ! This VM is having trouble accessing https://k8s.gcr.io
    * To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
    * 正在 Docker 20.10.8 中准备 Kubernetes v1.22.3…
    - Generating certificates and keys ...
    - Booting up control plane ...
    - Configuring RBAC rules ...
    * Verifying Kubernetes components...
    - Using image gcr.io/k8s-minikube/storage-provisioner:v5
    * Enabled addons: storage-provisioner, default-storageclass
    * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
    
  • 关闭minikube集群, 执行命令让虚拟机支持嵌套虚拟化
    minikube stop
    
    Get-VMProcessor -VMName minikube
    
    VMName   Count CompatibilityForMigrationEnabled CompatibilityForOlderOperatingSystemsEnabled
    ------   ----- -------------------------------- --------------------------------------------
    minikube 2     False                            False
    
    ;虚拟机开启嵌套虚拟化
    Set-VMProcessor -ExposeVirtualizationExtensions $true -VMName minikube
    
  • 查看集群情况
    PS C:\WINDOWS\system32> minikube.exe status
    minikube
    type: Control Plane
    host: Running
    kubelet: Running
    apiserver: Running
    kubeconfig: Configured
    

https://kubevirt.io/quickstart_minikube/

Nested virtualization

If the minikube cluster runs on a virtual machine consider enabling nested virtualization.

当minikube 集群运行在虚拟机的时候, 需要该虚拟机支持嵌套虚拟化,才能使用kubevirt部署和管理虚拟机.

macOS

  • vmware fusion 12.2 个人用户可以免费使用.支持嵌套虚拟化.需要自己手动开启
  • Parallels® Desktop 17或许可以但是没有测试.
  • virtualbox, hyperkit 不支持嵌套虚拟化
minikube start --driver=vmware

安装kubevirt

minikube 安装 kubevirt

一条命令安装kubevirt

minikube addons enable kubevirt

// 如果出现 kubevirt-install-manager 拉取镜像失败,耐心等待半小时,或者可以考虑使用多条命令安装.
[wjh@node1 ~]$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   50m
[wjh@node1 ~]$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS             RESTARTS      AGE
kube-system   coredns-78fcd69978-fqd54           1/1     Running            1 (10m ago)   50m
kube-system   etcd-minikube                      1/1     Running            1 (10m ago)   50m
kube-system   kube-apiserver-minikube            1/1     Running            1 (10m ago)   50m
kube-system   kube-controller-manager-minikube   1/1     Running            1 (10m ago)   50m
kube-system   kube-proxy-6d25g                   1/1     Running            1 (10m ago)   50m
kube-system   kube-scheduler-minikube            1/1     Running            1 (10m ago)   50m
kube-system   kubevirt-install-manager           0/1     ImagePullBackOff   0             5m44s
kube-system   storage-provisioner                1/1     Running            1 (10m ago)   50m

多条命令式安装kubevirt

  • 最好先下载kubevirt-operator.yaml和kubevirt-cr.yaml
C:\Users\wjh>kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/kubevirt-operator.yaml
namespace/kubevirt created
customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io created
priorityclass.scheduling.k8s.io/kubevirt-cluster-critical created
clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator created
serviceaccount/kubevirt-operator created
role.rbac.authorization.k8s.io/kubevirt-operator created
rolebinding.rbac.authorization.k8s.io/kubevirt-operator-rolebinding created
clusterrole.rbac.authorization.k8s.io/kubevirt-operator created
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator created
deployment.apps/virt-operator created

C:\Users\wjh>
C:\Users\wjh>kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/kubevirt-cr.yaml
C:\Users\wjh>
C:\Users\wjh>
C:\Users\wjh>kubectl create -f Downloads\kubevirt-cr.yaml
kubevirt.kubevirt.io/kubevirt created

C:\Users\wjh>kubectl get all -n kubevirt
NAME                                                                  READY   STATUS    RESTARTS   AGE
pod/kubevirt-7184739af3ff450da8cf9df6eb8ebffa3fae18c0-jobfq5s4pw625   1/1     Running   0          3s
pod/virt-operator-7d787566d5-9sd5v                                    1/1     Running   0          5m9s
pod/virt-operator-7d787566d5-tnb7w                                    1/1     Running   0          5m9s

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/virt-operator   2/2     2            2           5m9s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/virt-operator-7d787566d5   2         2         2       5m9s

NAME                                                                   COMPLETIONS   DURATION   AGE
job.batch/kubevirt-7184739af3ff450da8cf9df6eb8ebffa3fae18c0-jobfq5s4   0/1           3s         3s

NAME                            AGE   PHASE
kubevirt.kubevirt.io/kubevirt   13s   Deploying

下载virtctl工具

kubevirt 最新releases

例如下载virtctl-v0.48.1

https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-darwin-amd64

https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-linux-amd64

https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-windows-amd64.exe

查看安装情况

PS C:\WINDOWS\system32> minikube kubectl -- get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}"
Deploying
PS C:\WINDOWS\system32> minikube kubectl -- get all -n kubevirt
NAME                                   READY   STATUS              RESTARTS   AGE
pod/virt-api-79c76787cb-5x44n          1/1     Running             0          2m17s
pod/virt-api-79c76787cb-mhv2x          1/1     Running             0          2m17s
pod/virt-controller-8486c8d5cb-bzrf7   0/1     ContainerCreating   0          82s
pod/virt-controller-8486c8d5cb-gth4x   0/1     ContainerCreating   0          82s
pod/virt-handler-qf2jj                 0/1     Init:0/1            0          82s
pod/virt-operator-7d787566d5-jd8q9     1/1     Running             0          3m24s
pod/virt-operator-7d787566d5-wzllf     1/1     Running             0          3m24s

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubevirt-operator-webhook     ClusterIP   10.105.221.167   <none>        443/TCP   2m20s
service/kubevirt-prometheus-metrics   ClusterIP   10.103.77.79     <none>        443/TCP   2m20s
service/virt-api                      ClusterIP   10.102.101.254   <none>        443/TCP   2m20s

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/virt-handler   1         1         0       1            0           kubernetes.io/os=linux   82s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/virt-api          2/2     2            2           2m17s
deployment.apps/virt-controller   0/2     2            0           82s
deployment.apps/virt-operator     2/2     2            2           3m24s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/virt-api-79c76787cb          2         2         2       2m17s
replicaset.apps/virt-controller-8486c8d5cb   2         2         0       82s
replicaset.apps/virt-operator-7d787566d5     2         2         2       3m24s

NAME                            AGE     PHASE
kubevirt.kubevirt.io/kubevirt   3m22s   Deploying

安装kubevirt完成

PS C:\WINDOWS\system32> minikube.exe kubectl -- get all -n kubevirt
NAME                                   READY   STATUS    RESTARTS        AGE
pod/virt-api-79c76787cb-6fcph          1/1     Running   2 (4d15h ago)   4d17h
pod/virt-api-79c76787cb-g2gj2          1/1     Running   2 (4d15h ago)   4d17h
pod/virt-controller-8486c8d5cb-hkjfv   1/1     Running   1 (4d15h ago)   4d17h
pod/virt-controller-8486c8d5cb-ht859   1/1     Running   1 (4d15h ago)   4d17h
pod/virt-handler-9wtk4                 1/1     Running   1 (4d15h ago)   4d17h
pod/virt-operator-7d787566d5-cjrkt     1/1     Running   2 (4d15h ago)   4d17h
pod/virt-operator-7d787566d5-h2dcx     1/1     Running   2 (4d15h ago)   4d17h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubevirt-operator-webhook     ClusterIP   10.104.57.84     <none>        443/TCP   4d17h
service/kubevirt-prometheus-metrics   ClusterIP   10.108.227.106   <none>        443/TCP   4d17h
service/virt-api                      ClusterIP   10.103.220.122   <none>        443/TCP   4d17h

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/virt-handler   1         1         1       1            1           kubernetes.io/os=linux   4d17h

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/virt-api          2/2     2            2           4d17h
deployment.apps/virt-controller   2/2     2            2           4d17h
deployment.apps/virt-operator     2/2     2            2           4d17h

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/virt-api-79c76787cb          2         2         2       4d17h
replicaset.apps/virt-controller-8486c8d5cb   2         2         2       4d17h
replicaset.apps/virt-operator-7d787566d5     2         2         2       4d17h

NAME                            AGE     PHASE
kubevirt.kubevirt.io/kubevirt   4d17h   Deployed

kubevirt 启动虚拟机失败排查

使用kubectl describe 排查

  • vms
[root@node1 ~]#  kubectl describe vms testvm| head -n 10
Name:         testvm
Namespace:    default
Labels:       <none>
Annotations:  kubevirt.io/latest-observed-api-version: v1
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1
Kind:         VirtualMachine
Metadata:
  Creation Timestamp:  2021-12-30T05:35:02Z
  Generation:          14
[root@node1 ~]#  kubectl describe vms testvm| tail -n 20
    Enabled:  false
    Name:     containerdisk
    Reason:   Snapshot is not supported for this volumeSource type [containerdisk]
    Enabled:  false
    Name:     cloudinitdisk
    Reason:   Snapshot is not supported for this volumeSource type [cloudinitdisk]
Events:
  Type    Reason            Age                   From                       Message
  ----    ------            ----                  ----                       -------
  Normal  SuccessfulDelete  35m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance d74fd308-ced6-45a2-b32f-42a1754f36e2
  Normal  SuccessfulDelete  34m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance a34a4887-17f7-4c23-a854-4fd72c6743ca
  Normal  SuccessfulDelete  33m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance 25a1884c-742f-47fe-a0b1-1d5843005109
  Normal  SuccessfulDelete  31m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance 258154c1-e9e7-4aa5-811d-8fdcaa9fe6c7
  Normal  SuccessfulDelete  27m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance 609cde18-6625-4eb9-ac38-c6e9902f20dc
  Normal  SuccessfulDelete  21m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance d15ffdd4-f79b-41b6-b95d-d7749ae4657b
  Normal  SuccessfulDelete  20m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance 89217783-e691-43a9-b5b8-271bc2ea3cd4
  Normal  SuccessfulDelete  19m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance e87bc90c-24b0-4370-b8a7-7fe921589e55
  Normal  SuccessfulDelete  19m                   virtualmachine-controller  Stopped the virtual machine by deleting the virtual machine instance cb7ba9ed-6477-49e3-aa5f-a5119e99fc62
  Normal  SuccessfulCreate  9m37s (x19 over 35m)  virtualmachine-controller  Started the virtual machine by creating the new virtual machine instance testvm
  Normal  SuccessfulDelete  5m21s (x12 over 18m)  virtualmachine-controller  (combined from similar events): Stopped the virtual machine by deleting the virtual machine instance 0ba4bc8f-fcd1-43fb-87da-166b9a229d29

  • vmi
[root@node1 ~]# kubectl describe vmi testvm
Name:         testvm
Namespace:    default
Labels:       kubevirt.io/domain=testvm
              kubevirt.io/nodeName=minikube
              kubevirt.io/size=small
Annotations:  kubevirt.io/latest-observed-api-version: v1
              kubevirt.io/storage-observed-api-version: v1alpha3
API Version:  kubevirt.io/v1
Kind:         VirtualMachineInstance
Metadata:
  Creation Timestamp:             2021-12-30T06:04:59Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2021-12-30T06:05:20Z
  Finalizers:
    foregroundDeleteVirtualMachine
  Generation:  9
  Managed Fields:
    API Version:  kubevirt.io/v1alpha3
....
Status:
  Active Pods:
    8e517875-c61c-4c75-b262-bf1b04e1b6bb:  minikube
  Conditions:
    Last Probe Time:       2021-12-30T06:05:21Z
    Last Transition Time:  2021-12-30T06:05:21Z
    Message:               virt-launcher pod is terminating
    Reason:                PodTerminating
    Status:                False
    Type:                  Ready
    Last Probe Time:       <nil>
    Last Transition Time:  <nil>
    Status:                True
    Type:                  LiveMigratable
  Guest OS Info:
  Migration Method:     BlockMigration
  Migration Transport:  Unix
  Node Name:            minikube
  Phase:                Failed
  Phase Transition Timestamps:
    Phase:                        Pending
    Phase Transition Timestamp:   2021-12-30T06:04:59Z
    Phase:                        Scheduling
    Phase Transition Timestamp:   2021-12-30T06:04:59Z
    Phase:                        Scheduled
    Phase Transition Timestamp:   2021-12-30T06:05:20Z
    Phase:                        Failed
    Phase Transition Timestamp:   2021-12-30T06:05:20Z
  Qos Class:                      Burstable
  Virtual Machine Revision Name:  revision-start-vm-686e130e-1d4e-47ab-922d-0739c0ee7920-14
  Volume Status:
    Name:    cloudinitdisk
    Target:
    Name:    containerdisk
    Target:
Events:
  Type     Reason            Age   From                       Message
  ----     ------            ----  ----                       -------
  Normal   SuccessfulCreate  23s   virtualmachine-controller  Created virtual machine pod virt-launcher-testvm-jcpbz
  Warning  SyncFailed        2s    virt-handler               failed to configure vmi network: failed plugging phase1 at nic 'eth0': Critical network error: Couldn't configure ip nat rules
  Warning  Stopped           2s    virt-handler               The VirtualMachineInstance crashed.
  Normal   SuccessfulDelete  2s    virtualmachine-controller  Deleted virtual machine pod virt-launcher-testvm-jcpbz

kubevirt 控制虚拟机

kubevirt vm life cycle

kubectl create -f vmi.yaml

kubectl get vmis

kubectl get vmis testvmi

kubectl get vms

$ kubectl delete -f vmi.yaml
# OR
$ kubectl delete vmis testvmi

$ kubectl get vmi testvm -o=jsonpath='{.status.conditions[?(@.type=="Paused")].message}'


//启动虚拟机
virtctl start testvm
virtctl stop testvm
//连接虚拟机
virtctl console testvm

virtctl rename vm_name new_vm_name

NOTE

  • 安装过程出错执行

    minikube delete

    minikube delete -h

  • 安装kubevirt 拉取镜像失败
    • 不使用代理
    • 不使用阿里云镜像源

    minikube delete

    minikube start –image-mirror-country=”

  • minikube开启debug日志

    minikube start –alsologtostderr -v=7

  • go 环境配置proxy

    七牛云 – Goproxy.cn


Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: