The old fileserver was filled with a mixture of 3TB and 4TB disks, which were starting to become increasingly unreliable. The ZFS pool couldn’t expand until all the 3TB disks were replaced with 4TB, the SAS card they were connected to was obtained suspiciously cheaply as it was an uncommon chipset variant that’s not supported very well in Linux, the onboard SATA had persistently many read errors on one of the ports, I never quite liked the HTPC case it was in, and fixing any of these problems would inevitably require replacing most of entire server.

So a new fileserver was built in a 2RU rackmount case. 8x 12TB drives fill the front panel which has drive bays rather than the drives being internally mounted. 64GB of ECC RAM is fitted to the motherboard, with all the drives attached to 8 onboard SATA ports. It’s running an AMD CPU, because the current generation allows ECC on all CPUs without onboard graphics if the motherboard supports it (unlike Intel where only the high end Xeon line supports it). Two NVMe SSDs are installed on the motherboard - a 256GB one for the OS to boot from, and a 1TB one for L2ARC. A NVidia GT710 provides graphics, should the console ever need to be used - and maybe I can use it for hardware H.264 decoding/encoding to transcode some media later on - although I’m not totally sure it’ll be faster than doing so on the CPU.

It’s much more interesting with the top off…

I did, however, need a low profile 10Gbit network card. You may recall a long time ago, I picked up a bunch of Mellanox ConnectX-2 cards for about $35 each. Now, nearly four years on, the later ConnectX-3 cards are now selling for about $50, so I bought two with full height brackets, and a third with a low profile bracket (for a few dollars more). All three subsequently turned up with both kinds of bracket included.

Top left: Intel dual-port gigabit. Top centre: Mellanox ConnectX-2. Bottom row: Mellanox ConnectX-3, times 3.

The ConnectX-3 cards offload a little more of the work, draw less power, and are a PCIe 3.0 x4 rather than the PCIe 2.0 x8 of the ConnectX-2. This makes fitting one into a system a bit easier especially given the number of micro-ATX boards which are x16/x4 in arrangement - now both of our desktops are connected to the 10Gbit side of the LAN.

But what about getting my Macbook onto something faster than gigabit? I’ve currently got a mid-2014 Macbook Pro, which due to its age features Thunderbolt 2. It’s fairly ancient and it’s about time for an upgrade - so I’m planning on buying a 15" Apple Silicon Mac pretty much as soon as one exists. Forwards compatibility with this hypothetical new machine is essential.

I considered the Solo10G adaptor, which comes in a Thunderbolt 2 variant, but also a Thunderbolt 3 variant. The TB2 variant is significantly more expensive, however. I’m not sure why - it may be economies of scale, as Thunderbolt 3 is a lot more popular in PC laptops than Thunderbolt 2 was.

The Thunderbolt 3 version of the Solo10G adaptor also has a SFP+ version in addition to the 10GBaseT version. However, the SFP+ version is way more expensive - maybe it’s a media converter internally. I figured, though, that it would be best to get a 10GBaseT adaptor at this point - it’d probably be compatible with more networks going forward given the increasing popularity of NBaseT.

Given this, we’d need a way to connect 10GBaseT to the Mikrotik CRS317 switch. I’ve ordered a Mikrotik S+RJ10 - this fits into a SFP+ slot and provides a NBaseT/10GBaseT port. Because it’s limited in how much power it can draw from the SFP slot, it can only do 10Gbit for 30m, as opposed to the usual 100m, but my desk isn’t that far away from the server.

So I ended up buying the OWC Thunderbolt 3 10G adaptor. It’s got a cable on the input and supports 10GBaseT. I’m not entirely sure how I’d get it connected to my current MBP 15" mid-2014. I found out that the Apple Thunderbolt 3 to 2 adaptor dongle actually supports being used in reverse - with a Thunderbolt 2 host being connected to a Thunderbolt 3 device. However, it does not provide USB power to the USB C Thunderbolt 3 end, so the device must be self-powered, which the OWC network adaptor is not.

Some have reported success using the Thunderbolt 2-3 adaptor connected to a powered Thunderbolt 3 dock like the CalDigit TS3 Plus, then connecting the network adaptor to that. At the price for one of those, it’s an expensive workaround. What I’d like is a cable that allows Thunderbolt to pass through, but injects USB power. I’m still looking for one - I wonder if I’ll find one or buy the M2 MacBook first…

However, I wondered what else there was out there. NBaseT provides 5 and 2.5GBit speeds, which although not the full 10 would still be an upgrade over gigabit. 5Gbit USB 3 adaptors do exist - but cost about $150 to $200. Given the 10Gbit adaptor cost $250, this may not be entirely worth it. Users aren’t entirely happy with them, and they do seem to push the limits of USB 3, and performance is often CPU limited by the host.

However, I found something else. Plugable make a USB 3 ethernet adaptor that supports 1Gbit and 2.5GBit with an RTL8156 chipset. At a cost of $31, it’s not much more expensive than a USB 3 gigabit adaptor - and with an attached USB C to USB A adaptor, it’s usable on old and new computers. The downside is that being a USB device, it does take up one of the two USB ports (I wish the MacBook had more than two, there’s certainly room). The thunderbolt adaptor doesn’t - instead connecting to one of the display ports. Guess I might need to carry a USB hub too.

Top: Plugable USB 3 to 2.5Gbit adaptor. Bottom left: Mikrotik S+RJ10 SFP+ to 10GBaseT/NBaseT adaptor. Bottom right: Apple Thunderbolt 2 to Gigabit Ethernet adaptor.

So let’s compare it to the Apple Thunderbolt to Gigabit Ethernet adaptor. Connected to the gigabit “management” port on the CRS317 switch, by a 15m Cat6 SFP cable (a little overkill), we run iperf to the fileserver, which is connected to the switch by a Mellanox ConnectX-3 card on an SFP+ direct-attach copper cable. No need for fibre, as it’s just under the switch.

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec
[  5]  0.0-10.0 sec  1.10 GBytes   942 Mbits/sec

This achieves roughly 940mbit both ways - 95% of the theoretical maximum, which isn’t bad. Let’s swap it out for the Plugable adaptor.

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   906 MBytes   760 Mbits/sec
[  5]  0.0-10.0 sec  1.10 GBytes   940 Mbits/sec

Same 940mbit on the downlink side, but only 760mbit on the uplink side, which isn’t so great.

Trying the iperf transfer with two threads results in the difference disappearing. We check this against the Apple adaptor first to establish a baseline, then try with the plugable adaptor.

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   564 MBytes   473 Mbits/sec
[  4]  0.0-10.0 sec   563 MBytes   472 Mbits/sec
[SUM]  0.0-10.0 sec  1.10 GBytes   945 Mbits/sec

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   564 MBytes   473 Mbits/sec
[  5]  0.0-10.0 sec   563 MBytes   473 Mbits/sec
[SUM]  0.0-10.0 sec  1.10 GBytes   946 Mbits/sec

Now let’s swap out the gigabit port on the switch for the S+RJ10 and see how it goes.

First we’ll do a baseline test with the Apple Gigabit adaptor. The S+RJ10 will support “legacy” gigabit just fine:

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec
[  5]  0.0-10.0 sec  1.09 GBytes   938 Mbits/sec

It’s pretty much the same as before. Before we run iperf on the Plugable adaptor, I wanted to see if it was actually using 2.5Gbit. Checking ifconfig says it’s not, and the supported media types don’t even list support for it.

% ifconfig -m en5
en5: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	capabilities=400<CHANNEL_IO>
	ether XX:XX:XX:XX:XX:XX 
	inet6 fe80::10ba:61c2:1718:ea5a%en5 prefixlen 64 secured scopeid 0xd 
	inet 172.16.0.68 netmask 0xffffff00 broadcast 172.16.0.255
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
	supported media:
		media none
		media autoselect
		media 10baseT/UTP mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 1000baseT mediaopt full-duplex

Checking the Mikrotik control panel, on the other hand, shows that the SFP+ side is reporting it is running at 2.5GBit. If it is actually only in gigabit mode on the NBaseT side, we should see limited performance.

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   811 MBytes   680 Mbits/sec
[  5]  0.0-10.0 sec  2.48 GBytes  2.13 Gbits/sec

Although it’s capable of getting 2.13Gbit down, it appears that the same outgoing bandwidth limitation that applied on gigabit also applies here. I noticed that CPU load was pretty high during the test - it may indeed be CPU limited.

Doing two threads, which helped last time, is actually worse this time, only achieving 663mbit.

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   396 MBytes   332 Mbits/sec
[  5]  0.0-10.0 sec   395 MBytes   331 Mbits/sec
[SUM]  0.0-10.0 sec   791 MBytes   663 Mbits/sec

I decided to see how well it goes in the real world. I tried copying a few files back and forward over the network, and MenuMeters was showing this, as I pushed a few tens of gigabytes back and forward, noticably faster than before:

244MB/s (1.95Gbit) copying to the fileserver, 218MB/s (1.74Gbit) copying from it.

I guess this shows benchmarks aren’t everything. Interesting that for this workload, the outbound direction isn’t so slow.

Just as a final check, I borrowed an M1 13" Macbook Pro - and although it’s faster, the outbound bandwidth still seemed limited testing in iperf.

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.76 GBytes  1.51 Gbits/sec
[  5]  0.0-10.0 sec  2.65 GBytes  2.27 Gbits/sec

Unlike on the 2014 MBP, multi-threaded was faster outbound.

[  6]  0.0-10.0 sec  1.08 GBytes   925 Mbits/sec
[  5]  0.0-10.0 sec  1.08 GBytes   930 Mbits/sec
[SUM]  0.0-10.0 sec  2.16 GBytes  1.85 Gbits/sec

So I’m willing to say given the results in Finder, iperf may not be telling the full story. As file copying is likely to be most of what this adaptor will be used for, I’m happier with the real-world numbers.

The final annoyance is that it has a very bright network activity light, which for some reason is a bright white LED which literally lights up half the wall behind it when it flashes. I’m not sure why it was fitted with this, but it’s getting taped over…