在内网要做macos的编译 CI流水线,需要在zstack平台装一台虚拟机(它使用的是KVM,类似openstack)
尝试了好几种方式,最后选择了3
-
https://github.com/sickcodes/Docker-OSX之有试过docker里跑kvm的一个方法,确实可行,但不知道怎么设置网络,ping不通其他机器,作罢
-
参考之前做linux windows虚拟机的经验,以为vmware装一个macos,做出来的vmdk 再用qemu-img转成qcow2可以在zstack上跑,无法启动,失败
-
https://github.com/foxlet/macOS-Simple-KVM这个可行,开始折腾了半天libvirtd 的vnc密码,后面发现空密码不行,必须要设置一个密码,才能vnc;其他就是一台物理机centos7上的qemu版本有点问题,跑不起来这个,换了一个zstack上的虚拟机,套娃跑这个虚拟机,可行,稍微调整一下配置文件里的qemu kvm二进制路径,usb设备等就行了
-
还是想在zstack上直接跑这个macos,毕竟zstack做快照、备份方便点,要不就得自己人工做;但折腾了一下发现卡在clover选磁盘这里,对比了下zstack libvirtd qemu的配置,与macos-simple-kvm的UEFI固件有区别,[Solved] Clover hangs on "4630 scan entries" | tonymacx86.com结合这个,推断确实zstack上不能直接跑macos
-
其实还有一种方式是gitlab runner的libvirtd exectuor方式,但只看到linux虚拟机的实践,mac和windows的没有看到
使用macOS-Simple-KVM,把仓库克隆,进入目录:
-
外网找一台ubuntu22的机器 装好qemu等依赖
-
./jumpstart.sh --catalina
下载苹果的catalina
镜像 BaseSystem.img -
创建一个空磁盘
qemu-img create -f qcow2 MyDisk.qcow2 256G
-
给basic.sh 添加两行, 然后运行basic.sh 安装macos后关虚拟机, 设置用户名密码(比如设置用户是user,密码是12345)
-drive id=SystemDisk,if=none,file=MyDisk.qcow2 \
-device ide-hd,bus=sata.4,drive=SystemDisk \
-
https://github.com/foxlet/macOS-Simple-KVM/issues/566按照这个这个编辑ESP.qcow2中的config.plist 让clover自动选磁盘(diskutil list 拿到 /dev/disk3 vm后面的一个名称 ),不然每次启动还要去选择从哪里启动
-
把项目整个目录拿到内网,找个欧拉2003sp3的虚拟机(其他较新的系统也行,centos7似乎有问题),装好livirtd以及qemu-kvm一堆程序, 整个项目目录记得chown成qemu用户的,放到比如/opt/os下(qemu能访问到), 运行
./make.sh --add
, 再用virsh edit macOS-Simple-KVM
添加磁盘 -
配置类似(根据你qemu的版本,需要调整路径等,微调),这里设置qemu的vnc密码为123
msu@msu-System-Product-Name:~$ sudo cat /etc/libvirt/qemu/macOS-Simple-KVM.xml
[sudo] password for msu:
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit macOS-Simple-KVM
or other application using the libvirt API.
-->
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>macOS-Simple-KVM</name>
<uuid>d06d502a-904a-4b34-847d-debf1a3d76c7</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
<loader readonly='yes' type='pflash'>/media/msu/本地磁盘/macos/macOS-Simple-KVM/firmware/OVMF_CODE.fd</loader>
<nvram>/media/msu/本地磁盘/macos/macOS-Simple-KVM/firmware/OVMF_VARS-1024x768.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='custom' match='exact' check='none'>
<model fallback='forbid'>qemu64</model>
<topology sockets='1' dies='1' cores='4' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/media/msu/本地磁盘/macos/macOS-Simple-KVM/ESP.qcow2'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/media/msu/本地磁盘/macos/macOS-Simple-KVM/BaseSystem.img'/>
<target dev='sdb' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/media/msu/本地磁盘/macos/macOS-Simple-KVM/MyDisk.qcow2'/>
<target dev='sdc' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='network'>
<mac address='52:54:00:92:d4:7b'/>
<source network='default'/>
<model type='e1000-82545em'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='keyboard' bus='usb'>
<address type='usb' bus='0' port='2'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us' passwd='123'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<sound model='ich9'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='spice'/>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='4'/>
</redirdev>
<memballoon model='none'/>
</devices>
<qemu:commandline>
<qemu:arg value='-cpu'/>
<qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/>
<qemu:arg value='-device'/>
<qemu:arg value='isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'/>
<qemu:arg value='-smbios'/>
<qemu:arg value='type=2'/>
</qemu:commandline>
</domain>
-
最后
virsh start macOS-Simple-KVM
跑起来 -
网络是nat的 在宿主机上 删除libvirtd的规则
sudo iptables -D LIBVIRT_FWI 2
, 与10的iptables配置冲突了 -
在mac里增加屏幕共享(vnc),远程访问(ssh)添加转发规则 ,宿主机2222端口转发到虚拟机22端口,5999端口转发到虚拟机5900端口, 10.113.6.118为宿主机IP;qemu自身的vnc监听在宿主机的5900端口,但这个vnc不太好用,鼠标太飘
iptables -t nat -A PREROUTING -p tcp -d 10.113.6.118/32 --dport 2222 -j DNAT --to-destination 192.168.122.186:22
iptables -t nat -A PREROUTING -p tcp -d 10.113.6.118/32 --dport 5999 -j DNAT --to-destination 192.168.122.186:5900