PVE+TrueNAS+ZFS+10G内网方案

想要一个NAS，于是用上了TrueNAS，又想要虚拟化，无奈TrueNAS的虚拟化功能太弱了。于是在PVE上安装了TrueNAS，再将虚拟机安装到NAS上，达到all in one的目的，又能利用zfs的快照功能，加强虚拟机的安全性。

这个架构需要使用硬件直通+NFS。如果你的硬件条件不满足下面4个条件，个人建议不要尝试这个方案。如果你不需要将虚拟机存放到nas，也不需要尝试这个方案。

• 内存 ≥16G ---zfs吃内存
• CPU ≥i5 级别以上 ---zfs需要cpu去做计算，all in one得需要强力CPU
• 主板支持直通 ---zfs需要直接控制硬盘
• 主板支持nvme/或者有额外的sas/sata控制器/或者在U盘上装了系统 ---需要把硬盘直通给VM，所以需要一个系统盘。

一、方案

在PVE上新建TrueNAS虚拟机。同时创建2个网桥：

一个桥接到物理网口，以用于外部文件共享
一个不桥接任何网口，作为一个虚拟接口，用于TrueNAS和Proxmox nfs通信。（这个网口最高可以达到40G的带宽）

将非系统盘的硬盘控制器直通给TrueNAS，此时TrueNAS可以直接读取到硬盘，创建一个zfs池，再新建一个nfs共享，通过虚拟接口共享给PVE，作为VM的存储池。

 

二、本文实验环境：

类型	详细信息
CPU	QQLT
主板	华硕B150
内存	24G
板载SATA	3T*2
板载M.2	240G NVME(系统)
直通卡	m5110
直通卡硬盘	2T sas*4
PVE版本	7.0.9

三、开搞

1、创建2个网桥。

2、开启iommu

在/etc/default/grub的启动行，添加intel_iommu=on

随后进行update-grub，
再进行 ，update-initramfs -u，重启生效。

3、创建TrueNAS

创建的时候，建议使用UEFI和virio设备，同时添加2个不同网桥的网卡，一定要选Q35例下图

4、直通sata控制器给VM

添加PCIe设备

根据自己的需求，选择直通设备。反正不要把自己系统盘直通去了就行。注意，板载SATA名一般是芯片组+sata controller。

我这里要直通sata控制器和直通卡。再看一下配置

5、安装TrueNAS

记得选择UEFI，如果是传统模式，就要选bios

安装好重启就可以了。

6、配置TrueNAS，ip

计划是将正常的网卡配置局域网ip，将虚拟网桥接口配置到和pve一个网段的。

我这里正常的网口已经dhcp了，就直接上网页配置把

改成中文，修改了需要save一下。

进入网络配置网络接口

保存之后，如下

再选择test change，做更改

在pve上面，ping下这个truenas地址

虚拟接口链路是通的

正常线路也是通的。

这里使用iperf3 来测试一下虚拟接口的带宽。

在pve上安装iperf3，apt install iperf3 -y

root@pve3:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

iperf3已经集成在了TrueNAS上，可以直接使用

这里测试结果出来了，有30G的带宽。非常优秀。

root@pve3:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.10.2, port 25838
[  5] local 10.10.10.1 port 5201 connected to 10.10.10.2 port 31584
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.44 GBytes  29.5 Gbits/sec                  
[  5]   1.00-2.00   sec  3.77 GBytes  32.4 Gbits/sec                  
[  5]   2.00-3.00   sec  3.78 GBytes  32.5 Gbits/sec                  
[  5]   3.00-4.00   sec  3.78 GBytes  32.5 Gbits/sec                  
[  5]   4.00-5.00   sec  3.69 GBytes  31.7 Gbits/sec                  
[  5]   5.00-6.00   sec  3.72 GBytes  32.0 Gbits/sec                  
[  5]   6.00-7.00   sec  3.77 GBytes  32.4 Gbits/sec                  
[  5]   7.00-8.00   sec  3.76 GBytes  32.3 Gbits/sec                  
[  5]   8.00-9.00   sec  3.83 GBytes  32.9 Gbits/sec                  
[  5]   9.00-10.00  sec  3.85 GBytes  33.1 Gbits/sec                  
[  5]  10.00-10.07  sec   265 MBytes  32.6 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.07  sec  37.7 GBytes  32.1 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

对于30G的带宽，预估要20块hdd才能达到。所以对于一般的用户来说，这个接口完全不会成这个方案的瓶颈。

7、创建ZFS池和nfs共享

我这里使用2块3T硬盘创建了一个raid1

8、在PVE上添加nfs储存

在其他主机上，复制一个镜像到nfs存储

使用scp复制，可以看到速度有100M/S，这是把我的千兆网跑满了。

 

9、安装虚拟机测试。

拉跨的4k速度

4K速度太拉跨了。

测试一下存储移动的速度。将虚拟机关机，移动磁盘到本地存储。

create full clone of drive virtio0 (nfs:101/vm-101-disk-0.qcow2)
Formatting '/var/lib/vz/images/101/vm-101-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=42949672960 lazy_refcounts=off refcount_bits=16
transferred 0.0 B of 40.0 GiB (0.00%)
transferred 409.6 MiB of 40.0 GiB (1.00%)
transferred 819.2 MiB of 40.0 GiB (2.00%)
transferred 1.2 GiB of 40.0 GiB (3.00%)
transferred 1.6 GiB of 40.0 GiB (4.00%)
transferred 2.0 GiB of 40.0 GiB (5.00%)
transferred 2.4 GiB of 40.0 GiB (6.01%)
transferred 2.8 GiB of 40.0 GiB (7.01%)
transferred 3.2 GiB of 40.0 GiB (8.01%)
transferred 3.6 GiB of 40.0 GiB (9.01%)
transferred 4.0 GiB of 40.0 GiB (10.01%)
transferred 4.4 GiB of 40.0 GiB (11.02%)
transferred 4.8 GiB of 40.0 GiB (12.02%)
transferred 5.2 GiB of 40.0 GiB (13.02%)
transferred 5.6 GiB of 40.0 GiB (14.02%)
transferred 6.0 GiB of 40.0 GiB (15.02%)
transferred 6.4 GiB of 40.0 GiB (16.02%)
transferred 6.8 GiB of 40.0 GiB (17.02%)
transferred 7.2 GiB of 40.0 GiB (18.02%)
transferred 7.6 GiB of 40.0 GiB (19.02%)
transferred 8.0 GiB of 40.0 GiB (20.02%)
transferred 8.4 GiB of 40.0 GiB (21.02%)
transferred 8.8 GiB of 40.0 GiB (22.02%)
transferred 9.2 GiB of 40.0 GiB (23.02%)
transferred 9.6 GiB of 40.0 GiB (24.03%)
transferred 10.0 GiB of 40.0 GiB (25.04%)
transferred 10.4 GiB of 40.0 GiB (26.04%)
transferred 10.8 GiB of 40.0 GiB (27.04%)
transferred 11.2 GiB of 40.0 GiB (28.04%)
transferred 11.6 GiB of 40.0 GiB (29.04%)
transferred 12.0 GiB of 40.0 GiB (30.04%)
transferred 12.4 GiB of 40.0 GiB (31.04%)
transferred 12.8 GiB of 40.0 GiB (32.04%)
transferred 13.2 GiB of 40.0 GiB (33.04%)
transferred 13.6 GiB of 40.0 GiB (34.04%)
transferred 14.0 GiB of 40.0 GiB (35.04%)
transferred 14.4 GiB of 40.0 GiB (36.04%)
transferred 14.8 GiB of 40.0 GiB (37.05%)
transferred 15.2 GiB of 40.0 GiB (38.05%)
transferred 15.6 GiB of 40.0 GiB (39.05%)
transferred 16.0 GiB of 40.0 GiB (40.05%)
transferred 16.4 GiB of 40.0 GiB (41.05%)
transferred 16.8 GiB of 40.0 GiB (42.05%)
transferred 17.2 GiB of 40.0 GiB (43.05%)
transferred 17.6 GiB of 40.0 GiB (44.05%)
transferred 18.0 GiB of 40.0 GiB (45.05%)
transferred 18.4 GiB of 40.0 GiB (46.05%)
transferred 18.8 GiB of 40.0 GiB (47.05%)
transferred 19.2 GiB of 40.0 GiB (48.05%)
transferred 19.6 GiB of 40.0 GiB (49.06%)
transferred 20.0 GiB of 40.0 GiB (50.06%)
transferred 20.4 GiB of 40.0 GiB (51.06%)
transferred 20.8 GiB of 40.0 GiB (52.06%)
transferred 21.2 GiB of 40.0 GiB (53.06%)
transferred 21.6 GiB of 40.0 GiB (54.06%)
transferred 22.0 GiB of 40.0 GiB (55.06%)
transferred 22.4 GiB of 40.0 GiB (56.06%)
transferred 22.8 GiB of 40.0 GiB (57.06%)
transferred 23.2 GiB of 40.0 GiB (58.06%)
transferred 23.6 GiB of 40.0 GiB (59.06%)
transferred 24.0 GiB of 40.0 GiB (60.07%)
transferred 24.4 GiB of 40.0 GiB (61.07%)
transferred 24.8 GiB of 40.0 GiB (62.08%)
transferred 25.2 GiB of 40.0 GiB (63.08%)
transferred 25.6 GiB of 40.0 GiB (64.08%)
transferred 26.0 GiB of 40.0 GiB (65.08%)
transferred 26.4 GiB of 40.0 GiB (66.08%)
transferred 26.8 GiB of 40.0 GiB (67.08%)
transferred 27.2 GiB of 40.0 GiB (68.09%)
transferred 27.6 GiB of 40.0 GiB (69.09%)
transferred 28.0 GiB of 40.0 GiB (70.09%)
transferred 28.4 GiB of 40.0 GiB (71.09%)
transferred 28.8 GiB of 40.0 GiB (72.09%)
transferred 29.2 GiB of 40.0 GiB (73.09%)
transferred 29.6 GiB of 40.0 GiB (74.09%)
transferred 30.0 GiB of 40.0 GiB (75.09%)
transferred 30.4 GiB of 40.0 GiB (76.09%)
transferred 30.8 GiB of 40.0 GiB (77.10%)
transferred 31.2 GiB of 40.0 GiB (78.11%)
transferred 31.6 GiB of 40.0 GiB (79.12%)
transferred 32.0 GiB of 40.0 GiB (80.12%)
transferred 32.4 GiB of 40.0 GiB (81.12%)
transferred 32.8 GiB of 40.0 GiB (82.12%)
transferred 33.3 GiB of 40.0 GiB (83.13%)
transferred 33.7 GiB of 40.0 GiB (84.13%)
transferred 34.1 GiB of 40.0 GiB (85.14%)
transferred 34.5 GiB of 40.0 GiB (86.15%)
transferred 34.9 GiB of 40.0 GiB (87.16%)
transferred 35.3 GiB of 40.0 GiB (88.16%)
transferred 35.7 GiB of 40.0 GiB (89.16%)
transferred 36.1 GiB of 40.0 GiB (90.17%)
transferred 36.5 GiB of 40.0 GiB (91.17%)
transferred 36.9 GiB of 40.0 GiB (92.17%)
transferred 37.3 GiB of 40.0 GiB (93.17%)
transferred 37.7 GiB of 40.0 GiB (94.17%)
transferred 38.1 GiB of 40.0 GiB (95.17%)
transferred 38.5 GiB of 40.0 GiB (96.17%)
transferred 38.9 GiB of 40.0 GiB (97.17%)
transferred 39.3 GiB of 40.0 GiB (98.17%)
transferred 39.7 GiB of 40.0 GiB (99.18%)
transferred 40.0 GiB of 40.0 GiB (100.00%)
transferred 40.0 GiB of 40.0 GiB (100.00%)
TASK OK

从本地移动到nfs

create full clone of drive virtio0 (local:101/vm-101-disk-0.qcow2)
Formatting '/mnt/pve/nfs/images/101/vm-101-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=42949672960 lazy_refcounts=off refcount_bits=16
transferred 0.0 B of 40.0 GiB (0.00%)
transferred 413.7 MiB of 40.0 GiB (1.01%)
transferred 823.3 MiB of 40.0 GiB (2.01%)
transferred 1.2 GiB of 40.0 GiB (3.02%)
transferred 1.6 GiB of 40.0 GiB (4.04%)
transferred 2.0 GiB of 40.0 GiB (5.04%)
transferred 2.4 GiB of 40.0 GiB (6.05%)
transferred 2.8 GiB of 40.0 GiB (7.07%)
transferred 3.2 GiB of 40.0 GiB (8.08%)
transferred 3.6 GiB of 40.0 GiB (9.09%)
transferred 4.0 GiB of 40.0 GiB (10.09%)
transferred 4.4 GiB of 40.0 GiB (11.11%)
transferred 4.8 GiB of 40.0 GiB (12.12%)
transferred 5.2 GiB of 40.0 GiB (13.12%)
transferred 5.7 GiB of 40.0 GiB (14.14%)
transferred 6.1 GiB of 40.0 GiB (15.15%)
transferred 6.5 GiB of 40.0 GiB (16.15%)
transferred 6.9 GiB of 40.0 GiB (17.16%)
transferred 7.3 GiB of 40.0 GiB (18.17%)
transferred 7.7 GiB of 40.0 GiB (19.17%)
transferred 8.1 GiB of 40.0 GiB (20.18%)
transferred 8.5 GiB of 40.0 GiB (21.18%)
transferred 8.9 GiB of 40.0 GiB (22.20%)
transferred 9.3 GiB of 40.0 GiB (23.21%)
transferred 9.7 GiB of 40.0 GiB (24.22%)
transferred 10.1 GiB of 40.0 GiB (25.23%)
transferred 10.5 GiB of 40.0 GiB (26.24%)
transferred 10.9 GiB of 40.0 GiB (27.25%)
transferred 11.3 GiB of 40.0 GiB (28.25%)
transferred 11.7 GiB of 40.0 GiB (29.26%)
transferred 12.1 GiB of 40.0 GiB (30.28%)
transferred 12.5 GiB of 40.0 GiB (31.29%)
transferred 12.9 GiB of 40.0 GiB (32.31%)
transferred 13.3 GiB of 40.0 GiB (33.32%)
transferred 13.7 GiB of 40.0 GiB (34.33%)
transferred 14.1 GiB of 40.0 GiB (35.34%)
transferred 14.5 GiB of 40.0 GiB (36.35%)
transferred 14.9 GiB of 40.0 GiB (37.36%)
transferred 15.3 GiB of 40.0 GiB (38.36%)
transferred 15.7 GiB of 40.0 GiB (39.37%)
transferred 16.2 GiB of 40.0 GiB (40.38%)
transferred 16.6 GiB of 40.0 GiB (41.38%)
transferred 17.0 GiB of 40.0 GiB (42.39%)
transferred 17.4 GiB of 40.0 GiB (43.40%)
transferred 17.8 GiB of 40.0 GiB (44.41%)
transferred 18.2 GiB of 40.0 GiB (45.42%)
transferred 18.6 GiB of 40.0 GiB (46.44%)
transferred 19.0 GiB of 40.0 GiB (47.45%)
transferred 19.4 GiB of 40.0 GiB (48.46%)
transferred 19.8 GiB of 40.0 GiB (49.47%)
transferred 20.2 GiB of 40.0 GiB (50.49%)
transferred 20.6 GiB of 40.0 GiB (51.50%)
transferred 21.0 GiB of 40.0 GiB (52.52%)
transferred 21.4 GiB of 40.0 GiB (53.53%)
transferred 21.8 GiB of 40.0 GiB (54.54%)
transferred 22.2 GiB of 40.0 GiB (55.56%)
transferred 22.6 GiB of 40.0 GiB (56.56%)
transferred 23.0 GiB of 40.0 GiB (57.57%)
transferred 23.4 GiB of 40.0 GiB (58.58%)
transferred 23.8 GiB of 40.0 GiB (59.58%)
transferred 24.2 GiB of 40.0 GiB (60.59%)
transferred 24.6 GiB of 40.0 GiB (61.61%)
transferred 25.0 GiB of 40.0 GiB (62.61%)
transferred 25.5 GiB of 40.0 GiB (63.63%)
transferred 25.9 GiB of 40.0 GiB (64.63%)
transferred 26.3 GiB of 40.0 GiB (65.64%)
transferred 26.7 GiB of 40.0 GiB (66.66%)
transferred 27.1 GiB of 40.0 GiB (67.67%)
transferred 27.5 GiB of 40.0 GiB (68.68%)
transferred 27.9 GiB of 40.0 GiB (69.68%)
transferred 28.3 GiB of 40.0 GiB (70.70%)
transferred 28.7 GiB of 40.0 GiB (71.71%)
transferred 29.1 GiB of 40.0 GiB (72.73%)
transferred 29.5 GiB of 40.0 GiB (73.74%)
transferred 29.9 GiB of 40.0 GiB (74.75%)
transferred 30.3 GiB of 40.0 GiB (75.76%)
transferred 30.7 GiB of 40.0 GiB (76.77%)
transferred 31.1 GiB of 40.0 GiB (77.79%)
transferred 31.5 GiB of 40.0 GiB (78.79%)
transferred 31.9 GiB of 40.0 GiB (79.80%)
transferred 32.3 GiB of 40.0 GiB (80.80%)
transferred 32.7 GiB of 40.0 GiB (81.82%)
transferred 33.1 GiB of 40.0 GiB (82.83%)
transferred 33.5 GiB of 40.0 GiB (83.84%)
transferred 33.9 GiB of 40.0 GiB (84.85%)
transferred 34.3 GiB of 40.0 GiB (85.86%)
transferred 34.8 GiB of 40.0 GiB (86.88%)
transferred 35.2 GiB of 40.0 GiB (87.89%)
transferred 35.6 GiB of 40.0 GiB (88.91%)
transferred 36.0 GiB of 40.0 GiB (89.92%)
transferred 36.4 GiB of 40.0 GiB (90.93%)
transferred 36.8 GiB of 40.0 GiB (91.94%)
transferred 37.2 GiB of 40.0 GiB (92.94%)
transferred 37.6 GiB of 40.0 GiB (93.94%)
transferred 38.0 GiB of 40.0 GiB (94.94%)
transferred 38.4 GiB of 40.0 GiB (95.94%)
transferred 38.8 GiB of 40.0 GiB (96.95%)
transferred 39.2 GiB of 40.0 GiB (97.97%)
transferred 39.6 GiB of 40.0 GiB (98.98%)
transferred 40.0 GiB of 40.0 GiB (99.99%)
transferred 40.0 GiB of 40.0 GiB (100.00%)
transferred 40.0 GiB of 40.0 GiB (100.00%)
TASK OK

从测试结果来，顺序读写都有400M/S。算是把2块盘性能拉到顶了。

应该是大部分数据都写入到了内存，TrueNAS因为内存不足都已经卡死了。