For the last year or so, I have been involved in several client projects and in a lot of VC advisory discussions, where the success of a particular development program or start-up company depends largely on the viability of the iSCSI market in terms of timing and eventual size. Therefore I have been studying the applications, opportunities, and roadblocks for iSCSI for some time, trying to figure out how to forecast its prospects.  While I am continuing to gather information and hone my thinking, I do believe I have become dangerous in the sense that I now have a point-of-view.  I don’t think the iSCSI market is going to amount to much.  I’ll explain why to you in a series of two Musings, this one being the first.

Let’s start with some background information.  The main purpose of the iSCSI protocol is to allow block-level SCSI commands from servers to disk and tape drives and the ensuing returned data to be transported over an IP network.  Although there are alternative and competing protocol options, one potential application of iSCSI is the interconnection of geographically separate Fibre Channel SANs via an IP WAN.  This application has some real promise (as we will discuss in a future Musing in this series).  But the application that really has vendors salivating and has attracted mountains of investment dollars is the creation of native Ethernet-attached storage devices to enable Ethernet-based local SANs.

Why is there so much interest in iSCSI?  On the surface, it’s because it is projected that 8-12 million new devices will be attached to SANs per year by 2006.  To put this in perspective, in 2002 there will be a little more than one million new SAN connections made (essentially all Fibre Channel).  So a lot of growth is expected.  SAN connections tend to be high speed so you can probably count on a total connection cost (switch plus endpoint HBA) of about $600 -$1,000 per port in 2006.  Therefore, in terms of connectivity equipment alone, the SAN market could end up being somewhere between $5B and $12B in 2006.  Today, the market is smaller than this (about $2B), is growing reasonably well (for the times), and is almost totally dominated by Fibre Channel.  But numerous semiconductor and systems vendors are investing a lot of money in the development of iSCSI and IP/Ethernet SANs in the hope that much of the SAN growth will be in Ethernet and not Fibre Channel.

So, in reality, the interest in iSCSI is not really related to its technical promise or market potential so much, but to the hope that it will cause a disruption that will shift the balance of power among vendors, creating new opportunities for both start-ups and established companies who are presently locked out of the SAN market by strong incumbents.  This is the kind of situation that can attract a lot of investment dollars if there appears to be any reality beyond the hopes.

Of course, the promoters of Ethernet-based SANs don’t talk about it in terms of how it might benefit them.  They prefer to speak in terms of advantages to end users.  In doing so, they propose several value propositions:

• IP/Ethernet is more easily managed than Fibre Channel.

• IP/Ethernet is cheaper than Fibre Channel.

• IP/Ethernet allows for the elimination of a separate SAN network entirely, which plays into the two factors above.

• The above characteristics make iSCSI especially suitable as a choice for small-to-medium businesses that haven’t built SANs yet and haven’t invested in Fibre Channel yet.

• Not always specifically stated, but most importantly, IP/Ethernet has a Manifest Destiny – sooner or later it takes over everything.

Based on these tenets, it’s pretty obvious to see (say the promoters of iSCSI) that Ethernet-based SANs are going to be a huge market, dwarfing the size of my connectivity-only projections above when you add in the opportunity for value added security, management, SAN services, wide area, and other products. Maybe it’s a $50B market!  Maybe $100B!  Give us some money and let’s go find out!

But wait.  Let’s not concede this point of view just yet.

I think most of the pro-Ethernet SAN arguments are hogwash except for the ones related to the perception that Ethernet has to be the solution because it’s well ….. Ethernet.  Just that perception alone may be enough to drive iSCSI to critical mass.  But before we get into that, let’s discuss the other claims.

Claim
iSCSI is easier to manage because an Ethernet/IP network is more familiar to network engineers and because there are more management tools available to manage Ethernet/IP devices and networks than are available to manage Fibre Channel networks.

Reality
Even if Ethernet were a more easily manageable network protocol than Fibre Channel, it would be irrelevant to the overall SAN management discussion.  SANs are about moving stored data off of dedicated servers and onto boxes that can be accessed by a wider range of devices.  At least 90-95% of the daily operations tasks are related to managing this critical data and its dependent applications, most of which are completely transparent as to whether the underlying network glue is Fibre Channel, Ethernet, or benevolent elves.

However, to the degree that this is relevant, all of the existing SAN Management tools assume Fibre Channel as the underlying infrastructure (simply because that has been a reasonable assumption for the past few years) and they will have to be modified somewhat to work in iSCSI environments.  But that is a transient issue.  What happens when such tools are equally available and powerful for both iSCSI and Fibre Channel, won’t managing IP/Ethernet be easier?   Well, that’s not clear to me.  Fibre Channel is by its nature a more self-managing protocol in terms of traffic flow management, congestion avoidance, and performance tuning.  In a sense it’s not really fair to compare the management of Fibre Channel to Ethernet.  You have to compare it to the combined management of Ethernet, IP, and TCP in an environment where reliability is critical and performance is very important.  Early IP/Ethernet SAN adopters, and maybe even ones far into the future, will have to exercise a lot more monitoring and management effort to ensure the performance and quality of Ethernet-based SANs.  Therefore the opposite of this claim may be true - it will take some time and vendor development effort before Ethernet SANs can be managed as efficiently as Fibre Channel SANs.

Claim
Ethernet is a lot cheaper than Fibre Channel.

Reality
If you look at an Ethernet switch, it is cheaper per equivalent speed port than a Fibre Channel switch.  If you look at an Ethernet NIC today it is cheaper than a Fibre Channel HBA today. But these two facts aren’t really all that relevant to a discussion about the cost of an Ethernet SAN vs. a Fibre Channel SAN.  There are several reasons for this.

First, much of the difference in prices of Fibre Channel and Ethernet prices today are not due to fundamental cost considerations.  Much of the actual cost of Ethernet and Fibre Channel switches and NICs today are in the cost of the transceivers that drive the optical fibers or copper wires, and these components are identical for both Ethernet and Fibre Channel.

On the NIC/HBA side, you have to remember that you probably won’t be using a standard Ethernet NIC in a SAN application.  If you do, you’ll be putting a lot of burden on the host CPU to terminate TCP at high speeds.  Generally, some form of TCP off-load will be implemented on the NIC, either via another CPU or a so-called TCP Offload Engine (TOE) or Storage Network Processor (SNP).  Either way, this requires more high-speed components and a lot more buffer memory on the NIC than your average garden-variety Ethernet NIC.   Most of this extra cost is not needed on a Fibre Channel HBA because many of the functions that require TCP in the Ethernet environment are already handled by Fibre Channel in hardware in the MACs.

So, it’s reasonable to believe that Ethernet SAN HBAs will actually have Costs-of-Goods-Sold (COGS) that are greater than FC HBAs.

For switches, after the transceiver, the next biggest costs are related to switching silicon, memory, and processing horsepower.  There is more of a merchant silicon industry supporting Ethernet switches than Fibre Channel, so the Ethernet vendors may have an edge because of the purchasing power of volume. But there is another factor that needs to be considered here, which is the cost of over-provisioning. This could be a rather lengthy discussion, but I’ll try to make it shorter by over-simplifying.

Generally, in storage networks you don’t want to drop packets because end-to-end latency will dramatically increase, whether it is due to TCP retransmission latencies or higher layer Fibre Channel recovery mechanisms.  When this happens, the race is on because SCSI timers can time-out, which can cause entire operations to shut down and attempt to re-start.  And in SANs, congestion can occur quickly.  Although it may take 10 milliseconds or more for a high-end disk drive to spin up and find the data you want, once it does, unless otherwise throttled, it might send that back to you at 40-50 Mbytes/sec.  A couple of such drives spinning up relatively simultaneously and attempting to send over the same 1 Gbit/sec link within the network will congest that link.

Fibre Channel deals with that in part by implementing a link-by-link credit-based flow control mechanism.  Basically, a transmitting port is restricted to a maximum number of in-flight packets by the number of “credits” it has been issued, and the receiving port can further restrict the transmitter by pacing the rate at which it sends “READYs” back to the transmitter. In a good Fibre Channel switch, READYs are only sent back to the transmitter if the receiving port knows it can send the packets it has already received down the line to the next switch or to the ultimate destination device.  Thus, as congestion occurs on output ports, the traffic sources that are combining to create that congestion are naturally flow controlled.  This mechanism works so well that it is not uncommon for Fibre Channel links to hum along at 90-95% congestion with no packet (frame) drops.

In Ethernet, it is hard to achieve the same characteristics with many presently shipping switches. Some higher end Ethernet switches may implement a lot of packet buffering in an attempt to absorb periods of short-term congestion, and they may implement some form of XON/XOFF flow control to downstream devices to try to eliminate the congestion.  These are some of the characteristics I would define as necessary to be called a “Storage Grade” Ethernet switch.  Ethernet switches with such characteristics cost more than simple, basic switches.  Without these characteristics, the only way to avoid congestion is to try to drain buffered packets off of output buffers before the continual input load causes those buffers to overflow.  It means you need to overprovision for much higher speed links than may be needed to handle average traffic flow conditions. But over provisioning has a price.  If a 2 Gbps Fibre Channel link is effectively as good as a 10 Gbps Ethernet link (which it’s not, but it’s not that far off), which one do you think is cheaper to manufacture?

All of the above is to say I don’t think there is a fundamental COGS difference to manufacturers between Ethernet and Fibre Channel.  However, the overall price a customer has to pay to build a SAN may still be lower for Ethernet for two reasons:

1)   Fibre Channel vendors price to get 60-75% gross margins on their products. Additionally, these products generally arrive at your door through an OEM or Systems Integrator who jacks up the street price to you in order to maintain their profit margins and to allow them to provide the higher level of support generally needed in a SAN application vs. your garden variety Ethernet deployment.

2)   Ethernet NIC and switch vendors have become accustomed to gross margins in the 10-50% range (with the lower end more related to NICs and the upper end to switches), and the channel partners that delivery those products to you are also accustomed to much lower mark ups and margins – but also to less installation and support effort due to the relative “cookie cutter” nature of most Ethernet installs today.

If Fibre Channel vendors or channel partners were to shave their margins, or if Ethernet vendors or channel partners were to beef theirs up, the present street price difference between Fibre Channel and Ethernet would disappear.  We’ll discuss the probability of these events in the next Musing.

Claim
iSCSI will make network topologies simpler by eliminating the need for a separate network to connect servers to storage devices.

Reality
I’ll believe it when I see it.  So far there are not many converged local Ethernet/IP networks.  Even those few who use Voice-over-IP in their intranet usually provision and manage a separate network just for that traffic.  And as critical as voice is, stored data is even more essential, as it relates to the core assets of the company.  Plus, a big value of a SAN is precisely to create a separate network so that back-up, restoral, and mirroring operations can be performed without interfering with or being interfered by the normal client-to-application functions running in the enterprise.  Combined with the fact that in larger companies the responsibility for the applications and data management is fairly far distanced organizationally from the people responsible for the enterprise network, and you don’t really have the menu required to politically and technically create a single unified network that supports all traffic types at the level of service and reliability required for each traffic type.

Think of an Ethernet SAN as just that.  A specific network built to serve a specific need.  It may or may not share technical characteristics with the front-end network that has been built and optimized for very different requirements.

Claim
Ethernet is the choice for small-to-medium business.

Reality
I guess if you believe that Ethernet SANs are cheaper and easier to manage, this claim is direct fallout of that.  But since I don’t believe those basic tenets, I am driven to investigate this one a little bit further.

My main concern with this claim is that it assumes that small-to-medium businesses will implement SANs.  I’m not so sure of that.  The basic values of SANs are associated with issues of large scale – being able to add a lot more storage to meet the needs of a particular application, saving storage across lots of application silos, managing storage over a wide range of different systems in a unified and efficient manner, and enabling fast and efficient back-ups without disrupting normal network operations.  The spreadsheet that drives the decision “To SAN Or Not To SAN” usually requires pretty large numbers in the application silo, total storage capacity, # different systems, and personnel involved in data management columns to justify implementing a SAN.  Now, Small Business doesn’t necessarily mean Small Data, but there is some correlation. I just don’t see most small businesses and many medium-sized businesses justifying a SAN - FC or Ethernet or otherwise.  For these environments a low number of small-to-medium sized self-contained NAS boxes or Database Servers usually meets the needs.  Think of these alternative solutions as “SANs-in-a-box,” because they contain many of the components of a SAN pre-integrated and delivered in a more plug-and-play configuration for the end user.  These products generally interwork with lots of associated tools for back-up and restore, disaster recovery, archiving and other critical data management functions without the user ever having to get involved directly with block mode operations, SCSI protocols, and other things that SANs expose to the end user.

There you have it - I don’t believe that any of the commonly espoused technical or business related value propositions of native Ethernet SANs (championed almost exclusively by those who have a selfish interest in killing Fibre Channel) hold any water.   But there is still the issue of the Manifest Destiny of IP and Ethernet, which I believe creates a very real psychological advantage for iSCSI.  If the world assumes the answer might very well turn out to be X, the answer may turn out to be X.    But, in my next Musing I’ll tell you why I think the lessons of history actually work against iSCSI, and then I’ll give you my prediction of how the iSCSI market will really develop (or not).

(volume 5, number 8)