FreeBSD Networking with Emulex OneConnect® Ethernet Adapters

By Steve Perkins

A few months back, the question of “tuning” the Emulex FreeBSD driver came up and it took me back to the days when I would spend time “tuning” largely unroadworthy 1960’s and 1970’s British cars on weekends and evenings (I live in the UK so it seemed like a good idea at the time!). It was a belief that this was “performance tuning” but in reality, if the thing started without a push, it was a bonus. But it always felt like hours spent tweaking timings, gapping spark plugs and balancing Skinner Unions(SU) carburettors with a variety of tubes and tuning gadgets was worth all the time and blood lost. Network card driver tuning – what could be more fun?

If we look at the traditional customer base for Emulex products, it has been the sort of enterprise level data centres who usetraditional operating systems (OS) from the likes of Red Hat, SuSE, Microsoft, VMware and OEM UNIX derivatives. These “paid for” OSes (money up front and continuing support) have formed the backbone of our IT world and have been the focus of our driver development for Fibre Channel and Ethernet products.

But the IT world is changing and we are seeing new dynamic types of customer who are willing and able to take open source software to build new data centres for the world of big data and cloud solutions. One way Emulex has responded to this is to increase OS support outside of the “usual suspects” to embrace not only the community versions of Red Hat (Centos) but also Debian, Ubuntu and FreeBSD.

FreeBSD is an interesting OS that is often seen as a less showy alternative to the myriad of Linux distributions. Just getting its’ head down and getting on with the job, FreeBSD is quietly running everything from small home routers through TVs, switches, storage systems to data centres as well as being the basis for Apple’s OSX.

Emulex has Ethernet driver support for FreeBSD for our range of 10GbE OneConnect (OCe1110x) Network Interface Cards (NICs) available for download here. But do the drivers “just” work or do they “really” work? Have we kept the performance capabilities we have built into the OneConnect NICs for Linux, Windows, ESXi etc. for FreeBSD users, or are we just another NIC port? This was a question we’ve had a few times recently in EMEA from customers looking for FreeBSD support. Well, actually the conversation usually goes along the lines of:

Customer: We use FreeBSD,- do you have any drivers for your NICs?

Emulex: Yes we do, you can get them at ….

Customer: Are they any good?

Emulex: Yes, they’re good quality and dependable drivers.

Customer: No, really?

Emulex: Yes, we have embedded RISC CPUs which offload TCP/IP …

Customer: Hmmm…

So fuelled with the customers’ healthy scepticism that we are simply paying lip service to FreeBSD and all they get is basic NIC support, we thought it would be useful to check out and document not only the installation and configuration of the Emulex OneConnect NICs, but demonstrate the sort of performance that could be achieved.

Setting up a test environment always risks an argument, as we are faced with an almost infinite world of network configurations. Are we looking for database performance in a data centre, setting up video streaming servers, running a cloud data centre – the list goes on. Accepting that we’re not going to build a model of the Internet in the humble lab of the Emulex UK office, we settled on the very simplest configuration of 2x servers connected back-to-back, so we could simply look at the raw capabilities of the OneConnect NICs with FreeBSD. Using the industry standard Netperf tool is a bit like using IOmeter for testing storage I/O. There is a view that it is irrelevant to a real-world application – just a trade show trick – but I believe that it is of real value to strip back the complexities of the whole ecosystem to the raw components of performance. If the basic connectivity is broken, Netperf (or IOmeter) very quickly shows that something is not set up correctly. For example, if you have plugged a NIC in to a PCIe x1 slot by mistake, it is just never going to give you full 10GbE line rate. Drive your car in first gear, your journey is going to be long and loud.

This type of testing is important as we needed to understand the baseline capabilities before analysing the wider network environment in the final system. We aimed to have a very simple and repeatable setup that could be quickly replicated as a starting point for broader system tuning.

We produced an application note on the whole exercise at www.ImplementersLab.com which goes into detail on the installation of the driver, how the tests were done and the results. We’ve even included the script we used to run the tests so we could hit “go” and come back from lunch to a nice set of results.

I have already alluded to the “humble” nature of the Emulex UK labs so the tests were run on fairly average level systems – nothing exotic levered from Intel’s back door here! Even so, we could easily get very close to line rate transfers without any great effort (see chart)

Interestingly enough, the default Maximum Transmission Unit (MTU) size of 1500 bytes, which showed very respectable performance compared to enabling jumbo (9K) frames under TCP tests, although streaming with UDP really got value from larger frames.

Performance tuning of a system is a subject that can generate long email trails, but we need to consider the difference between tuning driver parameters and the broader OS parameters that are typically more relevant to the final application. Fortunately, the Emulex driver defaults are optimised as a default to maximise the use of hardware offload within the NIC CPU. Although the OneConnect NICs are theoretically capable of full TCP Offload Engine (TOE), this has fallen out of favour, and so we use stateless TCP offloads to effectively grab subroutines required in TCP/IP and process them in hardware. This approach still gives offload performance but compared to a full TOE implementation, allows the OS and applications access to all layers of the stack without a rewrite of the OS kernel. These stateless offload functions, such as hardware checksum calculations, VLAN tagging, TCP Segmentation Offload are managed using the good old ifconfig command on the NIC. Running “ifconfig –m” on a NIC port will give you the capabilities and which ones are enabled. For example:

capabilities=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO, VLAN_HWFILTER,VLAN_HWTSO>

ether 00:00:c9:e6:67:a8

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

media: Ethernet autoselect

status: no carrier

supported media:

media autoselect

This is the Emulex oce driver default configuration, and we can see that all hardware offload options are enabled. Apart from playing with the MTU, there is no point in changing things as all you are going to do is load up your CPU by moving hardware accelerated functions back on to the host system.

So that was easy. To the car analogy, these days my car just works. It is all computer controlled to run at optimum performance, the right idle speed, perfect timing, and it starts the first time! Somewhere in my garage, I have a collection of timing lights, Colortune plugs, tubes for balancing carbs etc. but evenings of frozen and skinned knuckles spent tinkering just to stand a chance of completing a journey are thankfully over. My car is German.

For more information on going beyond the basic performance validation of a system, there is some good background information around here which is based on work done by the developer of the Nginx web server and the FreeBSD wiki . This should all be considered “work in progress” as we all know that the world of IT never stops evolving, so in the field of performance, the raw capabilities will be the building blocks of understanding the whole system.

To complete the analogy (in the style of a blog!), getting from A to B in the fastest time is not about “does the car start” (without a push) and “can I keep most of the cylinders firing to complete a journey”. The car works as it should do just as the NIC runs predictably and reliably, but journey times are about how I choose the best route. I have GPS to find routes and live traffic data to guide me through the crowded road network and it is these tools that make the difference. Happy tinkering!