They say those who don’t study history are doomed to repeat it.  But the same is true of those who learn the wrong lessons in their historical analysis.

Once you get past all of the hand waving associated with arguable technical and business justifications for why Ethernet (running IP, TCP, and iSCSI) should replace or displace Fibre Channel as the preferred technology for SANs (see my previous Musing on this subject at www.acuitive.com/musings/hmv5-8.htm), you come to an argument that goes something like this:

“Ethernet and IP are irresistible forces, engulfing every application in their path.  The low cost, flexibility, simplicity, and ‘good enough-ness’ of IP and Ethernet have been chosen by the marketplace time after time.  Look at what Ethernet did to Token Ring, FDDI, and ATM.  Look at what IP did to Decnet, Appletalk, IPX, and OSI.  It can’t be stopped.  It will happen.”

While it is true Ethernet fended off all competitors to become the de facto Layer 2 LAN protocol, and IP did the same to become the universal Layer 3 protocol for packet-based messaging networks, that doesn’t necessarily mean these protocols have a Manifest Destiny or Divine Right to be deployed in all situations under all circumstances.  Various factors at work at that time allowed Ethernet and IP to win the protocols battles they have won to date.  If we look at those factors and apply them to the ensuing Fibre Channel/iSCSI wars, we can see the lessons of history actually favoring FC not iSCSI.

First, let’s review some basics.  When new networking challenges arise, the world favors the development of multiple alternative solutions.  The world then sorts through these choices by various means, including trying them out.  But in the end, the world likes to “vote” on a particular solution (and, in fact one or two leading vendors of that technology) so applications can be built on a stable platform and so management bells and whistles can be put into place.  This “vote” is usually taken long before most of the market opportunity has even developed.  This is why time-to-market is so critical in high tech products and why so many people learn the hard way that they entered markets after the “vote” was already taken (but the results not yet divulged), and they were never a viable candidate.

The winning protocol or technology is not always the best one, but it is always “good enough.”  All aspirant “good enough” technologies race to see who can achieve critical market mass soonest.   Critical market mass is achieved when end users assume a specific solution as “best practice” and the vendor community rallies around it instead of dissipating energy around multiple solutions.  When one technology or protocol reaches critical market mass, it is the winner.  That’s it.  There are no recounts or hanging chads.  The winning technology stays in vogue until a new approach with significantly better ROI emerges.

In the case of Ethernet and ATM, ATM came along after Ethernet was well established in the LAN.  It didn’t have a chance.  The cost of introducing ATM was just too high and there wasn’t much of an ROI to offset the cost.  The technical complexity of ATM resulted in an elongated technical gestation process, which allowed the Ethernet community to invent switched Ethernet, Fast Ethernet, Layer 3 switching, and Link Aggregation, the combination of which blew away any small pockets of support for ATM in the LAN.   As we look at the Ethernet-based vs. Fibre Channel based-SAN battle today, FC takes on the role more like Ethernet had in the past vs. ATM – it’s good enough, and there is widespread support in terms of storage products, management product, etc.  It’s also fairly widely deployed.  There are roughly two million production FC ports in operation today.  And while the iSCSI community sweats to get their initial products out and prove interoperability, basic performance, reliability, and manageability, the FC community is working on important incremental improvements and system enhancements, building on the base technology deployed and the lessons learned over the past three or four years.

The case of Ethernet vs. Token Ring was a little different since both technologies became available on the market at roughly the same time.  But Ethernet was adopted and supported by a broad vendor community and was promoted hand-in-hand with the adoption of TCP/IP as an extra boost.  Token Ring was mostly an IBM initiative with a little bit of Proteon thrown in, and wasn’t supported extensively by the TCP/IP community.   Those two facts resulted in different growth rates for the adoption of the technologies, which over the years compounded to create a large installed base differential between the technologies.  Vendors of a wide variety of related products - NIC cards, drivers, routers and bridges, servers, application suites, management tools – all recognized a bigger and more immediate market opportunity in Ethernet environments vs. Token Ring and either focused their products in those environments or at least tested and supported them first for Ethernet with lagging support for Token Ring.  Over time, end users applied “pattern recognition” and understood that even if they felt Token Ring was a better technology, overall solutions were cheaper and better if they used Ethernet because they had access to a wider variety of related products and solutions.  This is similar to why Macintosh lost out to Windows – the ISVs prioritized application development for Windows because it represented a bigger market for them and that eventually (almost) killed Macintosh.  In this case, Fibre Channel is again analogous to the Ethernet of the past, and Ethernet-based SANs are analogous to Token Ring.  The “whole product” for FC is far more advanced than it is for iSCSI and will remain so until the vendors of SAN management applications, business continuance applications, data archiving applications, SAN merchant silicon, SAN gateways, storage grade switches, etc., switch over to prioritizing support for iSCSI environments over FC.  That will only occur when iSCSI becomes the bigger opportunity, which it can’t become unless the vendor community of related products prioritizes it.  That creates a Catch 22, which in my mind is a signature of “game over.”

IP has a more glorious fight record than Ethernet, in a sense, because it has fought more fights and has more TKOs.  IP can count Novell IPX/SPX, Decnet, Dec LAT, Appletalk, Arcnet, SNA Multidrop and Bisync, a multitude of other proprietary protocols, and OSI as competition fought and squashed.  In the cases of all but OSI, these were protocols that at some point had more widespread adoption than IP, but IP still won out because it was (a) good enough, and (b) represented the only real open, standard solution that the market craved when this battle was fought in the mid-to-late 1980s.  The only competing standard-based solution was OSI, but it was so standards-based it took forever for huge committees to define its complex services.  By the time it even started to become available the game was long over – IP had achieved critical market mass.  An example of a fairer fight might be one within the IP community itself – IPv4 vs. IPv6.   IPv6 has been talked about and worked on for many years, but it has achieved little market penetration to date because most of the promise of IPv6 was achieved over time via incremental innovation in and around IPv4.  It’s just real hard to displace a technology that has achieved critical market mass – even if you are cheaper or more easily managed, or whatever.  In this case iSCSI and Ethernet-based SANs are the Johnny-come-lately and Fibre Channel is the established technology.  Can IP do to FC what IPv6 hasn’t been able to do to IPv4?

I suppose these historical lessons can be interpreted differently depending on your view of where we are in the ballgame.  We know the present score - Fibre Channel roughly two million, iSCSI roughly zero.  That is clear.  What is less clear is what inning we are in.

Personally, I think we’re in the later innings.  During the past 6-9 months I have interviewed 80 CIOs, IT Managers, and Data Center managers from large companies.  These represent a cross section of those who have implemented SANs to date.  In all cases, no one expressed any interest in iSCSI.  All had spent a lot of time and energy getting their Fibre Channel SANs in place and their plan for the next few years was to build out these SANs and to roll out new SANs in a cookie cutter fashion across more of their Data Centers across the world.  It seems to me that this represents a good chunk of the projected SAN growth – people who have already implemented some SANs doing more of the same.

A lot of people argue with me on this, taking the position that most people haven’t implemented SANs yet and that when small-to-medium business implements SANs, they will prefer Ethernet-based solutions.  But I don’t buy this simply because I don’t think there will be a lot of SAN deployment within that community.  SANs are for the elite.  If SAN were a sport, it would be polo, deployed by a small community with big Data Centers and big issues of scale for which the implementation of a SAN provides an adequate ROI to justify the activation cost.  In general, the mantra within IT organizations is “NAS when you can, SAN when you must.”  Today you can buy a very cheap Dell or HPQ server with three 76GB RAID 5 SCSI drives (and 120 GB just around the corner), which will meet the needs of a large number of businesses and applications.  NAS is more like the national pastimes of baseball, beer drinking, and belching.  It may be boring, it may be ugly, but everyone does it.  Furthermore, if a mid-sized organization is forced to go to a SAN it is more likely to replicate on a smaller scale what the polo players are doing rather than take a risk and head out to less-chartered waters.

Of course, as we’ve learned from history, it’s up to the incumbent players to close the doors to those trying to encroach on their turf.  Ethernet staved off FDDI and ATM partly due to the continual enhancement of Ethernet solutions.  IPv4 has deferred the widespread adoption of IPv6 by continually stretching itself to meet the promise of IPv6.   The Fibre Channel community is not playing in a huge market sandbox – SANs aren’t going to be that big.  To make sure the sandbox is at least entirely captured by them, the FC community has the option of squashing iSCSI.  If they don’t, they leave the door open to potentially snatching defeat from the jaws of victory.  If I were the FC community (meaning Brocade), here are three things I would do to create a fatal situation for iSCSI:

1. To once and for all extinguish the mistaken belief that Ethernet is fundamentally cheaper than Fibre Channel, I’d take some of my last generation 1 Gbps fixed configuration products, perhaps put copper interfaces on them, and offer them to the world for less cost per port than an equivalent Ethernet switch.  Nobody may buy these devices, but it will show that FC can be cheap and dirty if it needs to be.  But if you really want to build a SAN, look what else I’ve got for you…

There is one real cost of Fibre Channel compared to Ethernet, which is the cost of linking multiple FC switches together to create a larger scale network.  Right now the speeds of the links that interconnect FC switches tend to be the same as those that connect to end devices.  To avoid frequent congestion between switches, you have to either aggregate a bunch of links together to create virtual inter-switch links (which greatly increases your effective cost per usable port) or buy an expensive high density switch, leveraging a high speed internal fabric rather than creating a lot of inter-switch links.  Neither of these options is cost effective as SANs get even modestly bigger.  It’s not the cost of the boxes, it’s the system cost of building a network that requires more than two or three boxes.  Ethernet doesn’t have this problem because Ethernet has leveraged the use of “fat pipes” – Gigabit Ethernet now and 10 Gigabit Ethernet in the future – as inter-switch links to allow different types of Ethernet switches to take on different roles in the network – edge switch, switch-of-switches – so the hub-and-spoke and easily expandable architectures can be built in such a way that each port you add costs less than the ports installed before it.  So my next bit of advice for the FC community is:

2. Steal some good ideas from Ethernet.  Develop edge switches and switch-of-switches with appropriate “fat pipe” ISL capability.  The obvious choice for a fat pipe option is 10 Gbps FC, and I think the FC community has to get more aggressive on this to eliminate the perception that 10 Gbps Ethernet is way ahead.  But the FC community has a less obvious fat pipe alternative which, as far as I can tell, it has tried to bury as an unwelcome distraction up to now – and that is 4 Gbps FC.  There is no such equivalent in Ethernet.  The FC community should aggressively promote 4 Gbps technology, initially and perhaps forever as an interswitch link “fat pipe” option.  Dual 4 Gbps interfaces could be far cheaper and less disruptive on infrastructure than a single 10 Gbps link.  If true, or even if only true in a marketing sense, this gives the FC community an unfair advantage over the Ethernet community, who are more locked into 10x advances in technology, no matter where the cost/performance curves really have significant discontinuities.
    

3.  For a networking technology to achieve critical market mass, the cost-of-entry for vendors to build products to attach to and manage that technology needs to be low, and the technology cannot be viewed as mysterious and cosseted.  One symptom of critical market mass for a networking technology is usually the emergence of a merchant silicon industry to reduce the need for systems vendors to execute internal development programs for interface technology they are not familiar with or perhaps don’t view as strategic.  It is very important for the FC community to nurture a healthy merchant silicon community.  Right now, Qlogic provides some needed capability, and Vixel and Gadzoox are making some good moves to position themselves in that space.  But Vixel and Gadzoox and any other prospective entrant in this space need to drive some product volume in order to become stable and trusted suppliers.  While it’s probably not intuitively obvious within the FC systems community, one of the best things they could do for themselves would be to give design wins to selected merchant silicon suppliers for high volume sockets in some future FC products.  The resultant healthy merchant silicon community could then bring down the cost-of-entry and COGS of FC-related products.  This would lead to a richer mix of lower cost choices for customers, which would increase the overall deployment of FC, making the market bigger for the systems vendors.

Unfortunately, I don’t think the FC community will do most of these things, simply because they can’t afford to.  Most of the FC vendors are public companies and don’t have enough R&D budget to aggressively develop new technologies.  And most of them view their internal teams as quite capable at basic L2 FC chip development and therefore probably aren’t amenable to sourcing such technology from merchant silicon vendors.

But even if the FC community leaves the door open for the iSCSI market to develop, I’d still be surprised if it even reaches a peak of implementation equal to what Fibre Channel was at the end of 2001 today (roughly one million installed devices).

If you believe what I’ve written so far, why even as much as one million ports?  There are some applications in which iSCSI could take a role and create a niche:

• IP SAN WANs:  It’s pretty expensive to interconnect two geographically disparate FC SANs natively at the block-level with good performance.  It generally requires deploying a dark fiber link or a big chunk of a SONET circuit.  The concept of using lower cost IP-based WANs for SAN interconnection has some real merit.  There are a variety of protocols under development for this application (iSCSI, FCIP, iFCP, and some presently proprietary techniques using modified UDP), all of which require more development and field experience before the market can sort it out.  I see an opportunity for systems level products in this space (and in fact there are a boatload out there and more in the development pipeline) but anyone who makes a pure bet on iSCSI as the one and only protocol choice is probably heading off a steep cliff and anyone who approaches this as a semiconductor opportunity probably won’t achieve the quantity numbers needed to make that profitable.  Instead, the winner will be the systems vendors(s) that provide the right overall solution above and beyond the basic protocols, and support a variety of protocols to provide the greatest possible flexibility and market opportunity for their SAN WAN interconnect products.
   

•  NAS dis-aggregation: There is a lot of talk about NAS/SAN convergence, but I see the opposite actually happening.  Up until now, most NAS products have been self-contained; the storage devices are pre-packaged along with all of the file system, management, and networking intelligence required to perform the NAS function.  The upside of this is simplicity, which is why I think NAS is king, but the downside is the end user is restricted in terms of the maximum capacity of the storage and the selection of the storage components.  To deal with this, in some cases, the NAS function is being shrunk to a NAS Head that contains the client-side network interface, the NAS file system and management functionality, and a high speed SAN interface for connection to external storage devices.  A small external network is then created to enable end users or VARs to provision all the required storage, which can be added to over time as needed.  This external network may be stand-alone, meaning the only device that may have access to it is the NAS Head or a cluster of NAS Heads acting in concert.  In other words, the storage components hanging off the NAS Head may not be part of pooled storage with centralized data management operations like a cross-platform Data Center SAN.   But the technology used to create this external network may be the same as is used in Data Center SANs, so let’s call them SANlets.

Right now most of the NAS Heads I am familiar with are offering FC interfaces or are integrating small FC switches in them in order to create these SANlets.  I could easily see the vendors of these devices supporting an alternative approach – embedding an Ethernet switch to enable connection to native-Ethernet storage devices (when such exist).   The vendors of NAS Heads could work with specific native-Ethernet storage vendors and certify the use of a small list of options for customers for whom the performance and reliability would be sanctioned by the NAS Head vendor, and the NAS vendors value-added software would be guaranteed to work.  This eliminates several of the key hurdles Ethernet faces in penetrating the Data Center SAN space and creates a “grass roots” way for iSCSI to seep into enterprise companies, possibly unknown to the people running the centralized Data Centers.

Whenever I make such predictions about the future, I like to ask myself, where could I be wrong?  In this case I can see a couple of possibilities:

• If NAS SANlet approach discussed above takes off with Ethernet as the preferred storage-side protocol.  This could happen because people are really buying NAS solutions, not SAN, which I think will continue to dominate.  If enough of these NAS-inspired SANlets get deployed, sooner or later someone will notice that and decide that things would be more efficient if all the SANlets were interconnected and managed as a common pool of storage.  All of a sudden you’ve got a large Ethernet-based SAN.  This grass roots way of technology gaining a foothold and then spreading is quite common.  But this won’t happen here if the FC people are smart and capture this application right out of the chute.
   

• If SANs get converged with general purpose LANs; with SCSI traffic acting as just another payload along with e-mail, HTTP, and VoIP.  I rather doubt this will happen, but if it did, it certainly would work in favor of Ethernet-based SANs.

Continuing with the “how Hoover could be wrong” theme, I should mention that a lot of people disagree with me on my iSCSI projections.  Many of these people are faceless to me.  But one person that I have a lot of respect for that disagrees with many of my arguments is Glen Anderson.  You can view some of his thoughts at http://www.stackabledesign.com/white-iscsi.htm.

Next month, I am going to lay out my thoughts on the TCP Offload Engine (TOE) market, which is sometimes confused as the same market as iSCSI but which I think is a decidedly different market.

(volume 5, number 9)

Send email to info@acuitive.com with questions or comments about this web site.
Copyright ©1997-2002 Acuitive, Inc.  All Rights Reserved