概述
文章目录
- 应用场景
- 冷迁移(也叫静态迁移)
- 热迁移(也叫动态迁移)
- P2V迁移(企业迁移案例)
- 1 VM在KVM集群服务器之间的迁移(冷迁移技术)
- 2 VM在KVM集群服务器之间的迁移(webvirtmgr)
- 3 VMWare ESXi服务器迁移到KVM
- 4 XEN服务器迁移到KVM
应用场景
VMware ESXi–> KVM
KVM –> VMware ESXi
Xen –> VMware ESXi
XEN–>KVM
KVM –>KVM
0x1 v2v(虚拟机到虚拟机的迁移virtual machine to virtual machine)
冷迁移(也叫静态迁移)
静态迁移(offline migration)也叫做常规迁移,离线迁移。在迁移之前将虚拟机暂停,同时拷贝虚拟机镜像和状态到目的主机。相比较于在线迁移(online migration),其缺点就是静态迁移方式的过程需要显式的停止虚拟机的运行,而在线迁移的过程仅有非常短暂的停机时间,保证迁移过程中虚拟机服务的持续可用;静态迁移的优点是兼容性强,不需要VMM的同意API接口,可以借助第三方工具辅助迁移。由于目前VMM的开发没有统一的标准,不同的虚拟化厂商,尤其是闭源的厂商开发的VMM各成一套,静态迁移是解决不同类型虚拟机之间迁移的有效办法。
优点:虚拟机不需要位于共享存储器上,丢失数据的概率小。
缺点:需要关闭电源,业务中断。
名称 主机规划 内存 热迁移技术实现方式 网络 虚拟化
名称 | 主机规划 | 内存 | 热迁移技术实现方式 | 网络 | 虚拟化 |
---|---|---|---|---|---|
KVM01 | 10.0.0.11 | 4G | KVM静态迁移 | NAT | KVM |
KVM02 | 10.0.0.12 | 4G | KVM静态迁移 | NAT | KVM |
热迁移(也叫动态迁移)
热迁移(Live Migration),又叫动态迁移、实时迁移,即虚拟机保存/恢复,通常是将整个虚拟机的运行状态完整保存下来,同时可以快速的恢复到原有硬件平台甚至是不同硬件平台上。恢复以后,虚拟机仍旧平滑运行,用户不会察觉到任何差异。
如果源宿主机和目的宿主机共享存储系统,则只需要通过网络发送客户机的 vCPU 执行状
态、内存中的内容、虚机设备的状态到目的主机上。否则,还需要将客户机的磁盘存储发到目的主
机上。共享存储系统指的是源和目的虚机的镜像文件目录是在一个共享的存储上的。
在基于 共享存储系统 时,KVM 动态迁移的具体过程为:
1、迁移开始时,客户机依然在宿主机上运行,与此同时,客户机的内存页被传输到目的主机上。
2、QEMU/KVM 会监控并记录下迁移过程中所有已被传输的内页的任何修改,并在所有内存页都传输完成后即开始传输在前面过程中内存页的更改内容。
3、QEMU/KVM 会估计迁移过程中的传输速度,当剩余的内存数据量能够在一个可以设定的时间周期(默认 30 毫秒)内传输完成时,QEMU/KVM 会关闭源宿主机上的客户机,再将剩余的数据量传输到目的主机上,最后传输过来的内存内容在目的宿主机上恢复客户机的运行状态。
4、至此,KVM 的动态迁移操作就完成了。迁移后的客户机尽可能与迁移前一致,除非目的主机上缺少一些配置,比如网桥等。注意,当客户机中内存使用率非常大而且修改频繁时,内存中数据不断被修改的速度大于KVM能够传输的内存速度时,动态迁移的过程是完成不了的,这时候只能静态迁移
优点:软件和硬件系统的维护升级,不会影响用户的关键服务,提高了服务的高可用性和 用户的满意度。
缺点:过程不可中断,操作复杂(良好的网络环境,要不有可能丢包)。
名称 | 主机规划 | 内存 | 热迁移技术实现方式 | 网络 | 虚拟化 |
---|---|---|---|---|---|
KVM01 | 10.0.0.11 | 4G | virt-manager 进行动态迁移 | NAT | KVM |
KVM02 | 10.0.0.12 | 2G | quem+tcp 进行动态迁移 | NAT | KVM |
KVM03 | 10.0.0.13 | 2G | NFS进行动态迁移 | NAT | KVM |
KVM04 | 10.0.0.14 | 2G | webvirtmgr进行动态迁移 | NAT | KVM |
KVM05 | 10.0.0.15 | 2G | virt-v2v 进行动态迁移 | NAT | KVM |
P2V迁移(企业迁移案例)
1 VM在KVM集群服务器之间的迁移(冷迁移技术)
步骤
第一步 virsh list --all查看当前虚拟机状态
第二步 virsh shutdowm 你要迁移的虚拟机
第三步 virsh domblklist 虚拟机名称 获得虚拟机虚拟硬盘算在位置
第四步 virsh dumpxml 导出需要迁移的虚拟机配置文件
第五步 rsync -avz -P /vm_data/disk/xxx root@10.0.0.12:/vm_data/disk
第六步 virsh define xxx.xml virsh start xxx
2 VM在KVM集群服务器之间的迁移(webvirtmgr)
名称 | 主机规划 | 内存 | 软件需求 | 网络 | 虚拟化 |
---|---|---|---|---|---|
KVM01 | 10.0.0.143 | 2Core 2G | KVM+WebVirtMgr | NAT | KVM |
KVM02 | 10.0.0.144 | 4Core 8G | KVM | NAT | KVM |
KVM03 | 10.0.0.145 | 4Core 8G | KVM | NAT | KVM |
步骤一:
需要在kvm01 kvm02 kvm03上配置host文件本地解析
vim /etc/hosts
10.0.0.143 kvm01
10.0.0.144 kvm02
10.0.0.145 kvm03
步骤二:
在kvm01机器上免密登陆kvm02,kvm03。
[root@localhost conf.d]# su -s /bin/bash nginx
bash-4.2$ ssh-copy-id root@10.0.0.144 #kvm02机的免密
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/var/lib/nginx/.ssh/id_rsa.pub"
The authenticity of host '10.0.0.143 (10.0.0.143)' can't be established.
ECDSA key fingerprint is SHA256:N99fAx5OoDgmMhJEIaDDT4aPn4cusNFjs5xCmD9cDwc.
ECDSA key fingerprint is MD5:e3:d3:fc:9b:51:56:0d:20:5b:16:1e:1c:b4:38:a1:38.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.0.0.143's password:
Number of key(s) added: 1
Now try logging into the machine, with:
"ssh 'root@10.0.0.144'"
and check to make sure that only the key(s) you wanted were added.
bash-4.2$ ssh-copy-id root@10.0.0.145 #kvm03机器免密
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/var/lib/nginx/.ssh/id_rsa.pub"
The authenticity of host '10.0.0.143 (10.0.0.143)' can't be established.
ECDSA key fingerprint is SHA256:N99fAx5OoDgmMhJEIaDDT4aPn4cusNFjs5xCmD9cDwc.
ECDSA key fingerprint is MD5:e3:d3:fc:9b:51:56:0d:20:5b:16:1e:1c:b4:38:a1:38.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.0.0.143's password:
Number of key(s) added: 1
Now try logging into the machine, with:
"ssh 'root@10.0.0.145'"
and check to make sure that only the key(s) you wanted were added.
webvirtmgr管理三台服务器 kvm01-当管理机器,纳管kvm02 kvm03机,形成KVM集群
方法就是浏览器输入10.0.0.143这是kvm01管理机器的IP地址
点击右上角绿色 Add Connection按钮 添加连接
注意:这里选择SSH连接,我们已经做过免密登陆了。
Label:10.0.0.144
FQDN/IP:10.0.0.144
用户名:root
添加完毕以后重复该步骤
Label:10.0.0.145
FQDN/IP:10.0.0.145
用户名:root
添加完毕以后效果如图
步骤三:点击10.0.0.144红色区域
出现以后点击要迁移的虚拟机名称
进入以后点击启动按钮开启虚拟机
启动虚拟机
开始迁移
迁移过程中会关闭源服务器的虚拟机,在目标服务器上打开
点击控制台
3 VMWare ESXi服务器迁移到KVM
virt-v2v是将外部的虚拟化平台上的虚拟机转化到可以运行的KVM平台上。它可以读取在VMware、Xen运行Hyper-V和其他虚拟机管理程序上的Windows和Linux的虚拟机,并将其转换为KVM的libvirt,OpenStack等几种方式
yum install libvirt* virt-* qemu-kvm* -y
[root@kvm02 ~]# yum install libvirt* virt-* qemu-kvm* -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink
| 9.4 kB
00:00:00
* base: mirrors.aliyun.com
* epel: mirrors.yun-idc.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
base
| 3.6 kB
00:00:00
epel
| 5.3 kB
00:00:00
extras
| 2.9 kB
00:00:00
updates
| 2.9 kB
00:00:00
(1/2): epel/x86_64/updateinfo
| 1.0 MB
00:00:00
(2/2): epel/x86_64/primary_db
| 6.9 MB
00:00:02
Package libvirt-glib-1.0.0-1.el7.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-logical-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-qemu-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-core-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-nwfilter-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-lxc-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-secret-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-config-nwfilter-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-interface-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-bash-completion-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-mpath-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-python-4.5.0-1.el7.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-gluster-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-disk-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-scsi-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-client-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-config-network-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-network-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-libs-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-kvm-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-rbd-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-storage-iscsi-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package libvirt-daemon-driver-nodedev-4.5.0-23.el7_7.3.x86_64 already installed and latest version
Package virt-manager-common-1.5.0-7.el7.noarch already installed and latest version
Package virt-manager-1.5.0-7.el7.noarch already installed and latest version
Package virt-install-1.5.0-7.el7.noarch already installed and latest version
Package virt-what-1.18-4.el7.x86_64 already installed and latest version
Package virt-viewer-5.0-15.el7.x86_64 already installed and latest version
Package 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 already installed and latest version
Package 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 already installed and latest version
#次数省略N多行
Dependency Installed:
OVMF.noarch 0:20180508-6.gitee3198e672e2.el7
PyYAML.x86_64 0:3.10-11.el7
augeas.x86_64 0:1.4.0-9.el7
copy-jdk-configs.noarch 0:3.3-10.el7_5
giflib.x86_64 0:4.1.6-9.el7
glib2-devel.x86_64 0:2.56.1-5.el7
gtk2.x86_64 0:2.24.31-1.el7
java-1.8.0-openjdk.x86_64 1:1.8.0.232.b09-0.el7_7
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.232.b09-0.el7_7
javapackages-tools.noarch 0:3.4.1-11.el7
jna.x86_64 0:3.5.2-8.el7
libcmpiutil.x86_64 0:0.5.7-3.el7
libfontenc.x86_64 0:1.1.3-3.el7
libguestfs-winsupport.x86_64 0:7.2-3.el7
libxml2-devel.x86_64 0:2.9.1-6.el7_2.3
libyaml.x86_64 0:0.1.4-11.el7_0
lksctp-tools.x86_64 0:1.0.17-2.el7
lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7
m2crypto.x86_64 0:0.21.1-17.el7
nbdkit.x86_64 0:1.8.0-1.el7
nbdkit-plugin-python-common.x86_64 0:1.8.0-1.el7
nbdkit-plugin-python2.x86_64 0:1.8.0-1.el7
nbdkit-plugin-vddk.x86_64 0:1.8.0-1.el7
net-snmp-agent-libs.x86_64 1:5.7.2-43.el7
net-snmp-libs.x86_64 1:5.7.2-43.el7
openslp.x86_64 1:2.0.0-8.el7_7
pcre-devel.x86_64 0:8.32-17.el7
pcsc-lite-libs.x86_64 0:1.8.8-8.el7
python-dateutil.noarch 0:1.5-7.el7
python-javapackages.noarch 0:3.4.1-11.el7
python-lxml.x86_64 0:3.2.1-4.el7
python-suds.noarch 0:0.4.1-5.el7
sanlock.x86_64 0:3.7.3-1.el7
sanlock-lib.x86_64 0:3.7.3-1.el7
subscription-manager-rhsm.x86_64 0:1.24.13-3.el7.centos
subscription-manager-rhsm-certificates.x86_64 0:1.24.13-3.el7.centos
systemd-python.x86_64 0:219-67.el7_7.2
tog-pegasus.x86_64 2:2.14.1-7.el7
tog-pegasus-libs.x86_64 2:2.14.1-7.el7
ttmkfdir.x86_64 0:3.0.9-42.el7
tzdata-java.noarch 0:2019c-1.el7
unzip.x86_64 0:6.0-20.el7
xorg-x11-font-utils.x86_64 1:7.5-21.el7
xorg-x11-fonts-Type1.noarch 0:7.5-9.el7
xz-devel.x86_64 0:5.2.2-1.el7
zlib-devel.x86_64 0:1.2.7-18.el7
Dependency Updated:
glib2.x86_64 0:2.56.1-5.el7
systemd.x86_64 0:219-67.el7_7.2
systemd-libs.x86_64 0:219-67.el7_7.2
systemd-sysv.x86_64 0:219-67.el7_7.2
Complete!
#至此KVM+QEMU所有组件安装完毕!
#执行
virt-v2v -help
[root@kvm02 ~]# virt-v2v -help
virt-v2v: convert a guest to use KVM
virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi -os imported esx_guest
virt-v2v -ic vpx://vcenter.example.com/Datacenter/esxi esx_guest -o rhv -os rhv.nfs:/export_domain --network ovirtmgmt
virt-v2v -i libvirtxml guest-domain.xml -o local -os /var/tmp
virt-v2v -i disk disk.img -o local -os /var/tmp
virt-v2v -i disk disk.img -o glance
There is a companion front-end called "virt-p2v" which comes as an
ISO or CD image that can be booted on physical machines.
A short summary of the options is given below.
For detailed help please
read the man page virt-v2v(1).
Options:
-b, --bridge <in:out>
Map bridge ‘in’ to ‘out’
--color, --colors, --colour, --colours
Use ANSI colour sequences even if not tty
--compressed
Compress output file (-of qcow2 only)
--debug-overlay, --debug-overlays
Save overlay files
--echo-keys
Don’t turn off echo for passphrases
--help
Display brief help
-i <disk|libvirt|libvirtxml|ova|vmx>
Set input mode (default: libvirt)
-ic <uri>
Libvirt URI
-if <format>
Input format (for -i disk)
-io <option[=value]>
Set option for input mode
-ip <filename>
Use password from file to connect to input hypervisor
-it <transport>
Input transport
--key <SELECTOR>
Specify a LUKS key
--keys-from-stdin
Read passphrases from stdin
--mac <mac:network|bridge:out>
Map NIC to network or bridge
--machine-readable[=format]
Make output machine readable
-n, --network <in:out>
Map network ‘in’ to ‘out’
--no-copy
Just write the metadata
--no-trim <->
Ignored for backwards compatibility
-o <glance|libvirt|local|null|openstack|qemu|rhv|rhv-upload|vdsm>
Set output mode (default: libvirt)
-oa <sparse|preallocated>
Set output allocation mode
-oc <uri>
Output hypervisor connection
-of <raw|qcow2>
Set output format
-on <name>
Rename guest when converting
-oo <option[=value]>
Set option for output mode
-op <filename>
Use password from file to connect to output hypervisor
-os <storage>
Set output storage location
--password-file <filename>
Same as ‘-ip filename’
--print-estimate
Estimate size of source and stop
--print-source
Print source and stop
-q, --quiet
Don’t print progress messages
--root <ask|... >
How to choose root filesystem
-V, --version
Display version and exit
-v, --verbose
Enable libguestfs debugging messages
--vddk-config <filename>
Same as ‘-io vddk-config=filename’
--vddk-cookie <cookie>
Same as ‘-io vddk-cookie=filename’
--vddk-libdir <libdir>
Same as ‘-io vddk-libdir=libdir’
--vddk-nfchostport <nfchostport>
Same as ‘-io vddk-nfchostport=nfchostport’
--vddk-port <port>
Same as ‘-io vddk-port=port’
--vddk-snapshot <snapshot-moref>
Same as ‘-io vddk-snapshot=snapshot-moref’
--vddk-thumbprint <thumbprint>
Same as ‘-io vddk-thumbprint=thumbprint’
--vddk-transports <transports>
Same as ‘-io vddk-transports=transports’
--vdsm-compat <0.10|1.1>
Same as ‘-oo vdsm-compat=0.10|1.1’
--vdsm-image-uuid <uuid>
Same as ‘-oo vdsm-image-uuid=uuid’
--vdsm-ovf-flavour <ovirt|rhvexp>
Same as ‘-oo vdsm-ovf-flavour=flavour’
--vdsm-ovf-output <dir>
Same as ‘-oo vdsm-ovf-output=dir’
--vdsm-vm-uuid <uuid>
Same as ‘-oo vdsm-vm-uuid=uuid’
--vdsm-vol-uuid <uuid>
Same as ‘-oo vdsm-vol-uuid=uuid’
--vmtype <->
Ignored for backwards compatibility
-x
Enable tracing of libguestfs calls
[root@kvm02 ~]#
#证明安装virt-v2v成功!
通过V2V方式迁移
cd /vm_data/v2v/
virt-v2v-copy-to-local -ic esx://root@10.0.0.130?no_verify=1 Centosx64_ESXi
virt-v2v -i libvirtxml Centosx64_ESXi.xml -o local -os /vm_data/v2v/ -of raw
[root@kvm02 v2v]# virt-v2v-copy-to-local -ic esx://root@10.0.0.130?no_verify=1 Centosx64_ESXi
[
0.0] Fetching the remote libvirt XML metadata ...
2020-01-08 15:17:16.837+0000: 13731: info : libvirt version: 4.5.0, package: 23.el7_7.3 (CentOS BuildSystem <http://bugs.centos.org>, 2019-12-02-17:45:06, x86-02.bsys.centos.org)
2020-01-08 15:17:16.837+0000: 13731: info : hostname: kvm02
2020-01-08 15:17:16.837+0000: 13731: warning : esxConnectOpen:857 : Ignoring unexpected path '' for non-vpx scheme 'esx'
Enter root's password for 10.0.0.130:
[
7.9] Parsing the remote libvirt XML metadata ...
Enter host password for user 'root':
Enter host password for user 'root':
[
18.8] Copying remote disk 1/1 to Centosx64_ESXi-disk1
% Total
% Received % Xferd
Average Speed
Time
Time
Time
Current
Dload
Upload
Total
Spent
Left
Speed
100 10.0G
100 10.0G
0
0
55.6M
0
0:03:03
0:03:03 --:--:-- 50.9M
[ 202.8] Writing libvirt XML metadata to Centosx64_ESXi.xml ...
[ 202.8] Finishing off
[root@kvm02 v2v]# ls
Centosx64_ESXi-disk1
Centosx64_ESXi.xml
[root@kvm02 v2v]# virt-v2v -i libvirtxml Centosx64_ESXi.xml -o local -os /vm_data/v2v/ -of raw
[
0.0] Opening the source -i libvirtxml Centosx64_ESXi.xml
[
0.0] Creating an overlay to protect the source from being modified
[
0.1] Opening the overlay
[
4.2] Inspecting the overlay
[
11.7] Checking for sufficient free disk space in the guest
[
11.7] Estimating space required on target for each disk
[
11.7] Converting CentOS Linux release 7.6.1810 (Core)
to run on KVM
virt-v2v: This guest has virtio drivers installed.
[
41.3] Mapping filesystem data to avoid copying unused and blank areas
[
41.8] Closing the overlay
[
42.1] Assigning disks to buses
[
42.1] Checking if the guest needs BIOS or UEFI to boot
[
42.1] Initializing the target -o local -os /vm_data/v2v/
[
42.1] Copying disk 1/1 to /vm_data/v2v/Centosx64_ESXi-sda (raw)
(100.00/100%)
[
50.6] Creating output metadata
[
50.6] Finishing off
迁移完毕
virsh define Centosx64_ESXi.xml
[root@kvm02 v2v]# virsh list --all
Id
Name
State
----------------------------------------------------
-
Centos7x64_1810
shut off
-
Centosx64_ESXi
shut off
【友情提示】需要注意迁移虚拟机的网络状态,源虚拟机如果是桥接,迁移后需要编辑XML文件确定网络的连接方式。
<interface type='bridge'>
<mac address='00:0c:29:7a:fd:a7'/>
<source bridge='VM Network'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
改成默认网络模式即可上网Ping 通百度
<interface type='network'>
<mac address='52:54:00:e8:c3:2b'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
4 XEN服务器迁移到KVM
方法同ESXi迁移方式!!!
最后
以上就是懦弱凉面为你收集整理的Centos7KVM虚拟化Migration-企业迁移案例应用场景冷迁移(也叫静态迁移)热迁移(也叫动态迁移)P2V迁移(企业迁移案例)的全部内容,希望文章能够帮你解决Centos7KVM虚拟化Migration-企业迁移案例应用场景冷迁移(也叫静态迁移)热迁移(也叫动态迁移)P2V迁移(企业迁移案例)所遇到的程序开发问题。
如果觉得靠谱客网站的内容还不错,欢迎将靠谱客网站推荐给程序员好友。
发表评论 取消回复