多平台部署minikube+kubevirt
[TOC]
在不同的平台快速启动研发环境.
考虑到平台支持的情况,打算使用minikube+kubevirt.
目前在 x86 红帽 REHL 8 windows 10 macOS Monterey测试部署过.
对比
kubevirt
后Kubernetes时代的虚拟机管理技术之kubevirt篇
cloud-init
安装cloud-init (aliyun.com) cloud-init 只支持安装在linux系统上.
virtlet
KubeVirt vs Virtlet: Performance Guide
KubeVirt is a virtual machine management add-on for Kubernetes providing control of VMs as Kubernetes Custom Resources. Virtlet, on the other hand, is a CRI (Container Runtime Interface) implementation, which means that Kubernetes sees VMs in the same way it sees Docker containers.
Virtlet is a CRI implementation, so all VMs are defined as Kubernetes Pods and treated as first-class citizens, so to speak. The advantage of this architecture is that anything you can do with Pods can be done with Virtlet VMs, right out of the box.
但是virtle只支持linux平台,并且最后一次提交是2019年.
而kubevirt + minikube 可以在支持嵌套虚拟化的平台上运行.
安装minikube
minikube 虚拟化驱动
具体介绍可以看官网文档:minikube drive
- 可以在linux平台使用kvm逻辑安装
-
在windows 10 平台,开启hyperv嵌套虚拟化,安装部署成一台hyperv虚拟机
//开启hyperv嵌套虚拟化,进去minikube虚拟机可以看到它支持虚拟化. PS C:\WINDOWS\system32> minikube.exe ssh _ _ _ _ ( ) ( ) ___ ___ (_) ___ (_)| |/') _ _ | |_ __ /' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\ | ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/ (_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____) $ lsmod |grep kvm kvm_intel 233472 3 kvm 671744 1 kvm_intel irqbypass 16384 8 kvm $ exit - 在macos monterey上,使用vmware fusion2 开启嵌套虚拟化,安装部署成一台虚拟化.
点击下载安装minikube => minikube start | minikube (k8s.io)
安装完成后, 启动集群会自动,下载相应的镜像
linux
在linux平台建议裸机部署, 尝试过使用kvm驱动部署, 虽然开启了嵌套虚拟化,最后, kubevirt 启动虚拟机失败.
- 裸机安装需要先按照docker-ce, 按照官网安装
-
如果是REHL 8 平台需要修改docker repo中[docker-ce-stable] 内容为centos .
[wjh@node1 ~]$ cat /etc/yum.repos.d/docker-ce.repo |grep -ie '\[docker-ce-stable\]' -A 6 [docker-ce-stable] name=Docker CE Stable - $basearch baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg - 启动docker
// 考虑到不要使用root权限启动minikube, 需要给当前用户wjh,赋权执行docker 命令 sudo usermod -aG docker wjh // exit 重新ssh登录账号. // 启动docker sudo systemctl start docker.service sudo systemctl status docker.service - 启动minikube集群
// driver none 代表裸机安装 minikube start --driver=none // 查看集群状态 minikube status
windows
- windows开启嵌套虚拟化
-
管理员身份启动powershell
- powershell中设置路径, 启动集群,自动下载驱动.
PS C:\WINDOWS\system32> $oldPath = [Environment]::GetEnvironmentVariable('Path', [EnvironmentVariableTarget]::Machine) PS C:\WINDOWS\system32> if ($oldPath.Split(';') -inotcontains 'C:\minikube'){ ` >> [Environment]::SetEnvironmentVariable('Path', $('{0};C:\minikube' -f $oldPath), [EnvironmentVariableTarget]::Machine) ` >> } - 在windows平台使用hyperv驱动,minikube集群会被安装到hyperv的一台虚拟机上. 安装完成之后, 需要关闭集群,设置该虚拟机支持嵌套虚拟化.
PS C:\WINDOWS\system32> minikube start --hyperv-virtual-switch=minikube_switch --driver=hyperv * Microsoft Windows 10 Enterprise 10.0.19044 Build 19044 上的 minikube v1.24.0 * 根据用户配置使用 hyperv 驱动程序 * 正在下载 VM boot image... > minikube-v1.24.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s > minikube-v1.24.0.iso: 225.58 MiB / 225.58 MiB [ 100.00% 11.14 MiB p/s 20s * Starting control plane node minikube in cluster minikube * Downloading Kubernetes v1.22.3 preload ... > preloaded-images-k8s-v13-v1...: 501.73 MiB / 501.73 MiB 100.00% 11.07 Mi * Creating hyperv VM (CPUs=2, Memory=4000MB, Disk=20000MB) ... ! This VM is having trouble accessing https://k8s.gcr.io * To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ * 正在 Docker 20.10.8 中准备 Kubernetes v1.22.3… - Generating certificates and keys ... - Booting up control plane ... - Configuring RBAC rules ... * Verifying Kubernetes components... - Using image gcr.io/k8s-minikube/storage-provisioner:v5 * Enabled addons: storage-provisioner, default-storageclass * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default - 关闭minikube集群, 执行命令让虚拟机支持嵌套虚拟化
minikube stop Get-VMProcessor -VMName minikube VMName Count CompatibilityForMigrationEnabled CompatibilityForOlderOperatingSystemsEnabled ------ ----- -------------------------------- -------------------------------------------- minikube 2 False False ;虚拟机开启嵌套虚拟化 Set-VMProcessor -ExposeVirtualizationExtensions $true -VMName minikube - 查看集群情况
PS C:\WINDOWS\system32> minikube.exe status minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured
https://kubevirt.io/quickstart_minikube/
Nested virtualization
If the minikube cluster runs on a virtual machine consider enabling nested virtualization.
当minikube 集群运行在虚拟机的时候, 需要该虚拟机支持嵌套虚拟化,才能使用kubevirt部署和管理虚拟机.
macOS
- vmware fusion 12.2 个人用户可以免费使用.支持嵌套虚拟化.需要自己手动开启
- Parallels® Desktop 17或许可以但是没有测试.
- virtualbox, hyperkit 不支持嵌套虚拟化
minikube start --driver=vmware
安装kubevirt
一条命令安装kubevirt
minikube addons enable kubevirt
// 如果出现 kubevirt-install-manager 拉取镜像失败,耐心等待半小时,或者可以考虑使用多条命令安装.
[wjh@node1 ~]$ kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 50m
[wjh@node1 ~]$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcd69978-fqd54 1/1 Running 1 (10m ago) 50m
kube-system etcd-minikube 1/1 Running 1 (10m ago) 50m
kube-system kube-apiserver-minikube 1/1 Running 1 (10m ago) 50m
kube-system kube-controller-manager-minikube 1/1 Running 1 (10m ago) 50m
kube-system kube-proxy-6d25g 1/1 Running 1 (10m ago) 50m
kube-system kube-scheduler-minikube 1/1 Running 1 (10m ago) 50m
kube-system kubevirt-install-manager 0/1 ImagePullBackOff 0 5m44s
kube-system storage-provisioner 1/1 Running 1 (10m ago) 50m
多条命令式安装kubevirt
- 最好先下载kubevirt-operator.yaml和kubevirt-cr.yaml
C:\Users\wjh>kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/kubevirt-operator.yaml
namespace/kubevirt created
customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io created
priorityclass.scheduling.k8s.io/kubevirt-cluster-critical created
clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator created
serviceaccount/kubevirt-operator created
role.rbac.authorization.k8s.io/kubevirt-operator created
rolebinding.rbac.authorization.k8s.io/kubevirt-operator-rolebinding created
clusterrole.rbac.authorization.k8s.io/kubevirt-operator created
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator created
deployment.apps/virt-operator created
C:\Users\wjh>
C:\Users\wjh>kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/kubevirt-cr.yaml
C:\Users\wjh>
C:\Users\wjh>
C:\Users\wjh>kubectl create -f Downloads\kubevirt-cr.yaml
kubevirt.kubevirt.io/kubevirt created
C:\Users\wjh>kubectl get all -n kubevirt
NAME READY STATUS RESTARTS AGE
pod/kubevirt-7184739af3ff450da8cf9df6eb8ebffa3fae18c0-jobfq5s4pw625 1/1 Running 0 3s
pod/virt-operator-7d787566d5-9sd5v 1/1 Running 0 5m9s
pod/virt-operator-7d787566d5-tnb7w 1/1 Running 0 5m9s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/virt-operator 2/2 2 2 5m9s
NAME DESIRED CURRENT READY AGE
replicaset.apps/virt-operator-7d787566d5 2 2 2 5m9s
NAME COMPLETIONS DURATION AGE
job.batch/kubevirt-7184739af3ff450da8cf9df6eb8ebffa3fae18c0-jobfq5s4 0/1 3s 3s
NAME AGE PHASE
kubevirt.kubevirt.io/kubevirt 13s Deploying
下载virtctl工具
例如下载virtctl-v0.48.1
https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-darwin-amd64
https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-linux-amd64
https://github.com/kubevirt/kubevirt/releases/download/v0.48.1/virtctl-v0.48.1-windows-amd64.exe
查看安装情况
PS C:\WINDOWS\system32> minikube kubectl -- get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}"
Deploying
PS C:\WINDOWS\system32> minikube kubectl -- get all -n kubevirt
NAME READY STATUS RESTARTS AGE
pod/virt-api-79c76787cb-5x44n 1/1 Running 0 2m17s
pod/virt-api-79c76787cb-mhv2x 1/1 Running 0 2m17s
pod/virt-controller-8486c8d5cb-bzrf7 0/1 ContainerCreating 0 82s
pod/virt-controller-8486c8d5cb-gth4x 0/1 ContainerCreating 0 82s
pod/virt-handler-qf2jj 0/1 Init:0/1 0 82s
pod/virt-operator-7d787566d5-jd8q9 1/1 Running 0 3m24s
pod/virt-operator-7d787566d5-wzllf 1/1 Running 0 3m24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubevirt-operator-webhook ClusterIP 10.105.221.167 <none> 443/TCP 2m20s
service/kubevirt-prometheus-metrics ClusterIP 10.103.77.79 <none> 443/TCP 2m20s
service/virt-api ClusterIP 10.102.101.254 <none> 443/TCP 2m20s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/virt-handler 1 1 0 1 0 kubernetes.io/os=linux 82s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/virt-api 2/2 2 2 2m17s
deployment.apps/virt-controller 0/2 2 0 82s
deployment.apps/virt-operator 2/2 2 2 3m24s
NAME DESIRED CURRENT READY AGE
replicaset.apps/virt-api-79c76787cb 2 2 2 2m17s
replicaset.apps/virt-controller-8486c8d5cb 2 2 0 82s
replicaset.apps/virt-operator-7d787566d5 2 2 2 3m24s
NAME AGE PHASE
kubevirt.kubevirt.io/kubevirt 3m22s Deploying
安装kubevirt完成
PS C:\WINDOWS\system32> minikube.exe kubectl -- get all -n kubevirt
NAME READY STATUS RESTARTS AGE
pod/virt-api-79c76787cb-6fcph 1/1 Running 2 (4d15h ago) 4d17h
pod/virt-api-79c76787cb-g2gj2 1/1 Running 2 (4d15h ago) 4d17h
pod/virt-controller-8486c8d5cb-hkjfv 1/1 Running 1 (4d15h ago) 4d17h
pod/virt-controller-8486c8d5cb-ht859 1/1 Running 1 (4d15h ago) 4d17h
pod/virt-handler-9wtk4 1/1 Running 1 (4d15h ago) 4d17h
pod/virt-operator-7d787566d5-cjrkt 1/1 Running 2 (4d15h ago) 4d17h
pod/virt-operator-7d787566d5-h2dcx 1/1 Running 2 (4d15h ago) 4d17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubevirt-operator-webhook ClusterIP 10.104.57.84 <none> 443/TCP 4d17h
service/kubevirt-prometheus-metrics ClusterIP 10.108.227.106 <none> 443/TCP 4d17h
service/virt-api ClusterIP 10.103.220.122 <none> 443/TCP 4d17h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/virt-handler 1 1 1 1 1 kubernetes.io/os=linux 4d17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/virt-api 2/2 2 2 4d17h
deployment.apps/virt-controller 2/2 2 2 4d17h
deployment.apps/virt-operator 2/2 2 2 4d17h
NAME DESIRED CURRENT READY AGE
replicaset.apps/virt-api-79c76787cb 2 2 2 4d17h
replicaset.apps/virt-controller-8486c8d5cb 2 2 2 4d17h
replicaset.apps/virt-operator-7d787566d5 2 2 2 4d17h
NAME AGE PHASE
kubevirt.kubevirt.io/kubevirt 4d17h Deployed
kubevirt 启动虚拟机失败排查
使用kubectl describe 排查
- vms
[root@node1 ~]# kubectl describe vms testvm| head -n 10
Name: testvm
Namespace: default
Labels: <none>
Annotations: kubevirt.io/latest-observed-api-version: v1
kubevirt.io/storage-observed-api-version: v1alpha3
API Version: kubevirt.io/v1
Kind: VirtualMachine
Metadata:
Creation Timestamp: 2021-12-30T05:35:02Z
Generation: 14
[root@node1 ~]# kubectl describe vms testvm| tail -n 20
Enabled: false
Name: containerdisk
Reason: Snapshot is not supported for this volumeSource type [containerdisk]
Enabled: false
Name: cloudinitdisk
Reason: Snapshot is not supported for this volumeSource type [cloudinitdisk]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulDelete 35m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance d74fd308-ced6-45a2-b32f-42a1754f36e2
Normal SuccessfulDelete 34m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance a34a4887-17f7-4c23-a854-4fd72c6743ca
Normal SuccessfulDelete 33m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance 25a1884c-742f-47fe-a0b1-1d5843005109
Normal SuccessfulDelete 31m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance 258154c1-e9e7-4aa5-811d-8fdcaa9fe6c7
Normal SuccessfulDelete 27m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance 609cde18-6625-4eb9-ac38-c6e9902f20dc
Normal SuccessfulDelete 21m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance d15ffdd4-f79b-41b6-b95d-d7749ae4657b
Normal SuccessfulDelete 20m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance 89217783-e691-43a9-b5b8-271bc2ea3cd4
Normal SuccessfulDelete 19m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance e87bc90c-24b0-4370-b8a7-7fe921589e55
Normal SuccessfulDelete 19m virtualmachine-controller Stopped the virtual machine by deleting the virtual machine instance cb7ba9ed-6477-49e3-aa5f-a5119e99fc62
Normal SuccessfulCreate 9m37s (x19 over 35m) virtualmachine-controller Started the virtual machine by creating the new virtual machine instance testvm
Normal SuccessfulDelete 5m21s (x12 over 18m) virtualmachine-controller (combined from similar events): Stopped the virtual machine by deleting the virtual machine instance 0ba4bc8f-fcd1-43fb-87da-166b9a229d29
- vmi
[root@node1 ~]# kubectl describe vmi testvm
Name: testvm
Namespace: default
Labels: kubevirt.io/domain=testvm
kubevirt.io/nodeName=minikube
kubevirt.io/size=small
Annotations: kubevirt.io/latest-observed-api-version: v1
kubevirt.io/storage-observed-api-version: v1alpha3
API Version: kubevirt.io/v1
Kind: VirtualMachineInstance
Metadata:
Creation Timestamp: 2021-12-30T06:04:59Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2021-12-30T06:05:20Z
Finalizers:
foregroundDeleteVirtualMachine
Generation: 9
Managed Fields:
API Version: kubevirt.io/v1alpha3
....
Status:
Active Pods:
8e517875-c61c-4c75-b262-bf1b04e1b6bb: minikube
Conditions:
Last Probe Time: 2021-12-30T06:05:21Z
Last Transition Time: 2021-12-30T06:05:21Z
Message: virt-launcher pod is terminating
Reason: PodTerminating
Status: False
Type: Ready
Last Probe Time: <nil>
Last Transition Time: <nil>
Status: True
Type: LiveMigratable
Guest OS Info:
Migration Method: BlockMigration
Migration Transport: Unix
Node Name: minikube
Phase: Failed
Phase Transition Timestamps:
Phase: Pending
Phase Transition Timestamp: 2021-12-30T06:04:59Z
Phase: Scheduling
Phase Transition Timestamp: 2021-12-30T06:04:59Z
Phase: Scheduled
Phase Transition Timestamp: 2021-12-30T06:05:20Z
Phase: Failed
Phase Transition Timestamp: 2021-12-30T06:05:20Z
Qos Class: Burstable
Virtual Machine Revision Name: revision-start-vm-686e130e-1d4e-47ab-922d-0739c0ee7920-14
Volume Status:
Name: cloudinitdisk
Target:
Name: containerdisk
Target:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 23s virtualmachine-controller Created virtual machine pod virt-launcher-testvm-jcpbz
Warning SyncFailed 2s virt-handler failed to configure vmi network: failed plugging phase1 at nic 'eth0': Critical network error: Couldn't configure ip nat rules
Warning Stopped 2s virt-handler The VirtualMachineInstance crashed.
Normal SuccessfulDelete 2s virtualmachine-controller Deleted virtual machine pod virt-launcher-testvm-jcpbz
kubevirt 控制虚拟机
kubectl create -f vmi.yaml
kubectl get vmis
kubectl get vmis testvmi
kubectl get vms
$ kubectl delete -f vmi.yaml
# OR
$ kubectl delete vmis testvmi
$ kubectl get vmi testvm -o=jsonpath='{.status.conditions[?(@.type=="Paused")].message}'
//启动虚拟机
virtctl start testvm
virtctl stop testvm
//连接虚拟机
virtctl console testvm
virtctl rename vm_name new_vm_name
NOTE
- 安装过程出错执行
minikube delete
minikube delete -h
- 安装kubevirt 拉取镜像失败
- 不使用代理
-
不使用阿里云镜像源
minikube delete
minikube start –image-mirror-country=”
- minikube开启debug日志
minikube start –alsologtostderr -v=7
- go 环境配置proxy