Enterprise and technical customers place a diverse set of requirements on server I/O networks. In the past, no single network type has been able to satisfy all of these requirements. As a result several network types evolved and several interconnects emerged to satisfy a subset of the requirements. Recently several technologies have emerged that enable a single interconnect to be used in more than one fabric type. This paper will describe the requirements customers place on server I/O networks; the various fabric types and interconnects that have been used to satisfy those requirements; the technologies that are enabling network convergence; and how these new technologies are being deployed on various network families.

Periodic order-of-magnitude jumps in Ethernet bandwidth regularly reawaken interest in TCP/IP transport protocol offload. This time the jump to 10-Gigabit Ethernet coincides with the emergence of new network storage protocols (iSCSI and DAFS), and vendors are combining these with offload NICs to position IP as a competitor to FibreChannel and other SAN interconnects. But what benefits will offload show for application performance?

Several recent studies have presented conflicting data to argue that offload either does or does not benefit applications. But the evidence from empirical studies is often little better than anecdotal. The principles that determine the results are not widely understood, except for the first principle: Your Mileage May Vary.

This paper outlines fundamental performance properties of transport offload and other techniques for low-overhead I/O in terms of four key ratios that capture the CPU-intensity of the application and the relative speeds of the host, NIC device, and network path. The study also reflects the role of offload as an enabler for direct data placement, which eliminates some communication overheads rather than merely shifting them to the NIC. The analysis applies to Internet services, streaming data, and other scenarios in which end-to-end throughput is limited by network bandwidth or processing overhead rather than latency.

We evaluate the performance of a user-space Direct Access File System (DAFS) server and Oracle Disk Manager (ODM) client using two synthetic test codes as well as the Oracle database. Tests were run on 4-processor Intel Xeon-based systems running Windows 2000. The systems were connected with ServerNet II, a Virtual Interface Architecture (VIA) compliant system area network. We compare the performance of DAFS/ODM and local-disk based I/O, measuring I/O bandwidth and latency. We also compare the runtime and CPU utilization of the Oracle database running the TPC-H benchmark over DAFS/ODM and local disk.

The NFS filesystem was designed as a work-group filesystem, making a central file store available to and shared between a number of client workstations. However, more recently NFS has grown in popularity in the server room, connecting large application servers with back-end file servers. In this environment, where high-speed access to data is critical, high capacity interconnects like gigabit Ethernet, Fibre Channel and Infiniband are to be expected. With RDMA technology we can fully utilize the data capacity of these interconnects, while providing relief for host CPU and memory buses from the demands of managing a "fire hose" of data.

Here we describe the use of RDMA as an RPC transport layer. We show that the NFS protocol running over this new transport achieves higher throughput than a conventional TCP transport along with a reduction in CPU utilization. The benefits of an RDMA transport are enjoyed not only by the applications that use NFS mounts, but extend to any RPC service that requires high speed and efficient transfer of large volumes of data.

The iSCSI protocol is the IETF standard that maps the SCSI family of application protocols onto TCP/IP enabling convergence of storage traffic on to standard TCP/IP fabrics. The ability to efficiently transfer and place the data on TCP/IP networks is crucial for this convergence of the storage traffic. The iWARP protocol suite provides Remote Direct Memory Access (RDMA) semantics over TCP/IP networks and enables efficient memory-to-memory data transfers over an IP fabric. This paper studies the design process of iSCSI Extensions for RDMA (iSER), a protocol that maps the iSCSI protocol over the iWARP protocol suite. As part of this study, this paper shows how iSER enables efficient data movement for iSCSI using generic RDMA hardware and then presents a discussion of the iWARP architectural features that were conceived during the iSER design. These features potentially enable highly efficient realizations of other I/O protocols as well.

The design of modern operating systems is based around the concept of memory as a cache for data that flows between applications, storage, and I/O devices. With the increasing disparity between I/O bandwidth and CPU performance, this architecture exposes the processor and memory subsystems as the bottlenecks to system performance. Furthermore, this design does not easily lend itself to exploitation of new capabilities in peripheral devices, such as programmable network cards or special-purpose hardware accelerators, capable of card-to-card data transfers.

We propose a new operating system architecture that removes the memory and CPU from the data path. The role of the operating system becomes that of data-flow management, while applications operate purely at the signaling level. This design parallels the evolution of modern network routers, and has the potential to enable high-performance I/O for end-systems, as well as fully exploit recent trends in programmability of peripheral (I/O) devices.

As networks and I/O systems converge and the bandwidth of networks increases, conventional approaches to networking are struggling to deliver the performance and flexibility required.

CLAN (Collapsed LAN) is a high performance user-level network targeted at the server room. It supports RDMA and programmed I/O (PIO). We have implemented a set of IP based protocols at user level, and shown how true zero copy transmission (without modifying the sockets API) and reception can be achieved.

In this paper we discuss the problems associated with placing protocol stacks at user level and the architectural decisions required to obtain high performance. We also introduce our work using the network gateway which connects CLAN to the Internet to assist a server cluster in protocol processing.

Modern desktop and server computer systems use multiple processors: general purpose CPU(s), graphic processor (GPU), network processors (NP) on Network Interface Cards (NICs), RAID controllers, and signal processors on sound cards and modems. Some of these processors traditionally have been special purpose processors but there is a trend towards replacing some of these with embedded general purpose processors. At the same time main CPUs become more powerful; desktop CPUs start featuring Simultaneous Multi-Threading (SMT); and Symmetric Multi-Processing (SMP) systems are widely used in server systems. However, the structure of operating systems has not really changed to reflect these trends --- different types of processors evolve at different timescales (largely driven by market forces) requiring significant changes to operating systems kernels to reflect the appropriate tradeoffs.

In this position paper we propose to re-vitalise the old idea of channel processors by encapsulating operating system I/O subsystems in Virtual Channel Processors (VCPs). VCPs perform I/O operations on behalf of an OS. They provide similar development, performance, and fault isolation as dedicated (embedded) I/O processors do while offering the flexibility to split functionality between the main processor(s) and dedicated processors without affecting the rest of the OS. If part of a VCP is executed on the main processor, we propose to make use of virtual machine technology and SMT/SMP features to isolate its performance from that of the rest of the system and to protect the system from faults within the VCP.