Why is a connectionless protocol best for Ethernet SAN?

SAN traffic is simple commands to either write or read data, to persistent storage (usually disk drives) connected to a network—and this flow of data needs to be fast.

When speed is important, it matters how you transport data over the network.  Wasted steps and added complexity slow data flow.

It’s clear, Ethernet has won the technology race as the preferred transport mechanism for Storage Area Networks (SAN).  Ethernet is simpler, faster, and much less expensive than Fibre Channel.

But Ethernet defines an electrical method and basic frame/packet structure to carry data, it doesn’t define how data is to be transported.  Ethernet was designed to be a lossy network, primarily because early hardware implementations experienced network collisions, and some packets were lost in transmission.  So, applications that use Ethernet need to cope with occasional dropped packets.

Wrapping each packet in TCP guarantees lost packets are resent and delivered without corruption, but TCP also adds other features that are not always needed, especially on an Ethernet SAN.

For example, TCP establishes a “connection path” between the two ends of the network conversation.  This connect results in unintended consequences when TCP is used in a SAN.  To mitigate these consequences adds complexity.  I’ll explain more later.

ATA-over-Ethernet (AoE):   SAN without Complexity

AoE is an IEEE registered Ethernet frame type (0x88A2), and the AoE protocol includes important features that enable Ethernet to transport data efficiently and reliably.  AoE accomplishes some of the things provided by TCP, but without all the unnecessary stuff for a SAN application.

First each AoE frame (packet) is given a unique tag number.  All AoE packets, initiated by the host (initiator) are acknowledged by the EtherDrive storage device (target) echoing the packets tag number.  If a packet has not been acknowledged the host will resend it with a new reference tag number.  Standard Ethernet frame CRC will detect corrupt packets and discard them.

That’s it!  We don’t need to reorder the sequence of received data since the data is already identified by its block address, and we don’t need to route the packets with IP addresses when they stay within the local network.

Okay, so what happens when you use AoE as the storage protocol?

As I mentioned earlier, everyone wants the data flow between the storage device and the host (server) to be as fast as possible.  GigE and 10GigE links are used in an Ethernet SAN fabric.  More ports at the initiator and more ports at the target, provide more bandwidth for more data flow.  And multiple paths through the network also provide redundancy in case a link is broken.

In an iSCSI SAN, the TCP part of the protocol sets up a connection from one port on the initiator to one port on the target.  If more than one port is available on the initiator or target, the TCP connection won’t use it without additional software to manage these multi-path connections.  And as the number of available ports connected to the SAN is increased, the number of connection possibilities grows exponentially.  Each possible path presents a new instance of the same target LUN, making management much more complicated.

Connectionless Simplicity

AoE is a connectionless protocol and has the advantage of being able to flood the network with as many packets as the ports and paths between initiator and target can handle.

The AoE initiator will round robin packets between all available ports, load balancing the flow, for maximum throughput speed.  The LUNs are presented as a single instance to the host, no mater how many paths are being used for the data flow.  If the fabric of the SAN is changed, or new paths added/removed, AoE just naturally works.  Links can fail and be restored and the AoE flow automatically adapts to the available bandwidth.

That’s why AoE is better than iSCSI, it’s connectionless.  It's simpler to understand, and easier to use.  It just works.....and that's what makes EtherDrive storage so fast.

Confessions of an EtherDrive user.

Some time ago, I was speaking with an EtherDrive storage user about how he was using the product.

Well, she said, it all started several years ago.  I'm a system integrator and I was contracted by a nearby University to come up with a storage solution for some class room lectures that had been recorded, and used periodically across the campus.  It was considered a low priority project, so the system cost was important. 

We started this first project with EtherDrive SR1521's attached to a Linux server.  It went together quickly and worked exactly as planned.  The organization was surprised at how inexpensive the system was.  EtherDrive has built its reputation by providing great value, and reliability.  As is normal in campus life, word spreads when something new is tried and the experience is good.   

Soon I had another department at the University asking if EtherDrive might work for the campus security system.  They were expanding the number of security cameras around the campus and needed something that could expand easily.  Again we used EtherDrive SR storage and a Linux server, to build a really nice way to store and retrieve recorded videos from hundreds of IP cameras. And as the storage needs grew, we just added more EtherDrive SR's to the SAN.  We didn't need to change the Linux server, it just had more storage to work with. 

Word spread and the next request came from the main IT department of the University.  They needed some low cost backup disk storage, and they were willing to give EtherDrive a try.  They were sure, it wasn't going to be fast enough or good enough for their tier 1 applications, but for a lowly backup job, they decided to give EtherDrive a try.  There were many skeptics and some real technology bigots in that organization, so the trial had lots of critical eyes watching.  The system installed so quickly and the testing ran so smoothy, many in the IT department became EtherDrive fans. 

Next was the medical science department.  They had huge storage needs for a cluster of Linux servers sharing SAN storage, for their new genome sequencing projects.  This was a high performance application that needed to scale, but still be affordable.  A 10GigE SAN was constructed and they installed more than a dozen EtherDrive SRX storage appliances.  This same architecture has been replicated and is being used by many Universities around the world.  The genome sequencing people must talk a lot about what they are doing. 

As the University started moving into server virtualizing, they went with VMware, and after initially using iSCSI, they switched to EtherDrive SRX Storage.  EtherDrive storage can be configured to satisfy all the redundancy and reliability that virtualization environments require. 

The bottom line was, that each and every application could be supported with EtherDrive storage.  Its just simple to understand when you realize EtherDrive is just a disk (or LUN) on the network.  

Since that early conversation with my system integrator friend, I have yet to hear of any application that can not be satisfied by EtherDrive storage.  Users can install SATA drives and achieve lowest cost per TeraByte, or install SAS or SSD disks and achieve maximum performance and still have the lowest cost.