The Ripple Effect: Part 4 (And Final)

This is the last in my sequence of Musings projecting the possible implications (or Ripple Effect) of servers that can actually run at 1 or 2 Gbps, potentially as enabled by technology recently announced and soon to be delivered by Precision I/O.  This Musing focuses on impacts in the networking arena.  

Wither Infiniband?

A few years ago, Infiniband was the “Great White Hope” for a standards-based technology that would alleviate some of the known I/O inefficiency issues of servers, with server clustering, Blade server backplanes, and server-to-storage as the primary assumed applications.  But Infiniband took awhile to develop and came out at a bad time –when customers were looking to buy nothing, certainly not the brightest shiniest new thing.  As a result, Infiniband has not achieved anything close to critical mass.  One after another, vendors who were formerly bullish on Infiniband, at least publicly, have backed off bit by bit.  In the meantime, the concepts of RDMA and RNICs have achieved some public airing, giving end users just another reason to delay and see how things play out.

Although I think Infiniband was already kind of dead-on-arrival, and would surely have had whatever life it still had in it expunged by the introduction of RDMA, the advent of the Precision I/O technology – sooner and better than RDMA, just accelerates the dying process for Infiniband.   (Unfortunately for Precision I/O, the impact on Infiniband was the main press/analyst topic as a reaction to their initial announcement. But this was focusing on the fleas on the tail of the dog rather than the big picture.  Precision I/O can accelerate all servers with significant I/O loads.  Those that may have otherwise used Infiniband represents only a very small percentage of such servers.)

It is possible that Infiniband may hang on to an ever diminishing niche is in support of tightly coupled applications that move off of expensive Symmetrical Multiprocessing Platforms (SMPs) to commodity computing hardware using Infiniband as a low latency interconnect.  The low latency of Infiniband comes from three sources – elimination of inefficient multiple memory copies to get stuff from application space to the network, a low latency hardware-based underlying guaranteed delivery protocol (which obviates the need for TCP), and low latency switching achieved via cut-through mechanisms.    Precision I/O provides pretty much the same latency characteristics in an Ethernet/IP/TCP environment as the first two items in the Infiniband list above. In the past, these two latency sources dominated the overall end-to-end latency. So right there latency is greatly reduced.  What’s left now is the switch latency, especially in large clusters where the server-to-server path might traverse multiple switches and inter-switch links.   Historically, Ethernet switch vendors have not focused too much on latency, preferring to let that slide a bit to ensure high throughput.  But even so, some switches (albeit store-and forward) are pretty low in latency and clever engineers could make them even lower if  tasked to do so (possibly even supporting cut-through between same-speed ports, which would be useful at 1 Gbps but less so at 10 Gbps).  Therefore the Infiniband niche gets smaller and smaller. When you have to use a microscope to see a niche, is it really a niche or is it a nick?

Ripple Effect Winners

·         High Performance Computing (HPC) users

·         Distributed database users

·         Ethernet switch vendors with low latency

Ripple Effect Losers

·         Infiniband systems and components vendors

·         Customers who prematurely adopted Infiniband

Storage Network Implications

This one is a tough call.  The tempting thing is to say that a solution that Precision I/O brings to the table will catalyze the growth of iSCSI.  To a degree I think that is true.  But there are so many other considerations and forces at work in the Fiber Channel vs. iSCSI battle that I don’t think this one element will change the end result much.  With or without Precision I/O, iSCSI will slowly seep into the enterprise landscape and probably over time become dominant as a transport for block transfers over networks.  But with or without Precision I/O, this will occur over an extended period of time.  I don’t think Precision I/O will impact the timelines much, even though Precision I/O will probably be a useful part of the overall iSCSI solution, edging out others who thought their hardware would be necessary.

 Ripple Effect Winners

·         Analysts and press, who have many years to talk about (in sequence) the death of Fiber Channel, Why iSCSI Didn’t Happen, and then Why iSCSI Killed Fiber Channel

Ripple Effect Losers

·         TOE and iSCSI HBA vendors who built hardware oriented 1 Gbps solutions. 

Front-End Network Infrastructure Impacts

As servers get smaller, Data Centers are building out denser server racks.  Racks that used to house 3-6 servers now house 48 or 96 or more.  Independent of network speed, this changes the optimum physical network topology from one where servers are home-run connected to centralized switches in the Data Center to one where edge switches are deployed right in the server rack, potentially pre-wired before the rack is shipped to the end user site.  This simplifies cabling between the servers and the network, providing a small number of external cables for connection to the Data Center or backbone network.  If you want to get really cheap, the switches in the racks can be Fast Ethernet switches with a couple of Gigabit Ethernet uplinks, because most servers struggle to run at even Fast Ethernet speeds today.  The alternative today, which is growing in popularity as 100/1000 switches get cheaper and cheaper and as Gigabit Ethernet comes for free on the servers, is to put Gigabit Ethernet switches in the racks (generally 48 port devices) but still only come out of the rack with a small number of Gigabit Ethernet fiber uplink trunks, allowing a large amount of oversubscription (44 GigE server ports playing into dual 2 GigE link aggregated trunks, for example).  The oversubscription rarely presents a problem because the servers today run at speeds so much less than 1 Gbps and usually servers aren’t all running at their top speed simultaneously.   In either case, the switch/routers that interconnect the switches that the servers are connected to are usually high density GigE devices. 

 But all this needs to change if the servers become truly Gig-capable.  Fast Ethernet connections to the servers will no longer cut it.  Massive over-subscription with-in the server rack will no longer be viable as well. To avoid re-creating the cabling mess we had before switches made their way into the server racks, 10 GigE will be needed out of the rack.  A typical rack switch will then be 48 1 GigE ports for server connections with dual 10 GigE trunks for uplinks.  This configuration is easily supported by the Broadcom Ethernet Switch chip set, which is rapidly becoming the industry de-facto standard for non-modular Ethernet switches. 

 Precision I/O represents the “killer application” that will drive 10 GigE adoption.  This is by far the biggest Ripple Effect that will be created by the introduction of the Precision I/O technology and has some very important repercussions. 

 If 10 GigE is coming out of the server racks, then the aggregation switches need to be 10 GigE capable at a high density.  If the Data Center aggregation switches are 10 GigE capable, then the backbone switches they connect to need to be 10 GigE capable and then the switches feeding the uplinks to wiring closets probably need to be 10 GigE capable.  This doesn’t necessarily lead to logic that says 1 GigE to the desktop, but it might if that becomes the bottleneck for various applications now that the servers have been unblocked.  More likely, the 10 Gig downlinks from the wiring closet switches would be supporting aggregate traffic from lots of Fast Ethernet endpoints and new higher speed 802.11 Access Points that would combine to over-saturate GigE downlinks. 

 So what needs changing here from a networking standpoint?

 1.      The introduction of switches into server racks (or replacement with 1/10 GigE switches if they already exist).

2.      The deployment of very high speed (500-1000 Gbps) aggregation switches that support a high density of 10 GigE ports with-in the Data Center to connect the rack server switches to one another, to the campus backbone, and to the WAN (directly or indirectly). 

3.      Upgrade of campus backbone switches to 10 GigE trunk ports facing the Data Center aggregation switches.

4.      (Possibly) upgrade of IDF aggregation switches to 10 GigE trunk ports.

5.      (Possibly) upgrade of wiring closet switches to 10 GigE downlinks facing the IDF aggregation switches.

6.      (Possibly) massive WAN bandwidth build out.  

 In other words – what doesn’t change?  While it was inevitable that networks would evolve in this manner, it could have taken a decade to occur without the stimulus that Precision I/O brings to the table.  Server I/O capability has been holding us back, retarding the need to provision more bandwidth throughout the rest of the system. Eliminate that restriction and everything changes.  Hmm.  Maybe we don’t want to eliminate that bottleneck!

 The diagram below shows the historical uptake of switched Fast Ethernet and switched GigE as a percentage of the total number of switched Ethernet ports shipped in a year, time normalized to the first effective shipping year for each technology.  Although you can barely see it, the line at the bottom shows the history of 10 GigE in its first couple of years of life, as well as industry projections for the future.

 This shows that each generation of Ethernet has taken longer to penetrate the marketplace than the previous one.  One of the reasons for this, however, is that each new generation of Ethernet helps stimulate deployment of the previous generation, because it provides a necessary “fat pipe” to supplement the previous generation in network designs.

 The diagram below shows the historical actual numbers for shipped ports (with projections for 10 GigE past the first couple of years).

 Looking at this diagram, you can see the acceleration effect that switched GigE provided for switched Fast Ethernet.  The same will happen with GigE and 10 GigE, only faster than projected now, because of the Precision I/O effect. The diagram below shows my adjusted projected switched 1 GigE and 10 GigE ports due to the Precision I/O effect.

This is either very good for Cisco, or, since almost everything needs to change (very little installed base equipment can be effectively upgraded to meet these requirements), very good for a Cisco competitor who has been waiting for a disruption to create an opportunity to go displace Cisco.

 Given Cisco’s software, service, and political muscle with-in the enterprise, I wouldn’t bet against them.  Don’t sell your Cisco stock.  

 This will also be very good for the vendors who supply critical components into Cisco high end switch/routers – ASIC vendors, optical module vendors (XFPs),  power supply vendors, memory vendors, etc.  Especially blessed will be anyone that helps Cisco increase the non-blocking 10 GigE density on their line cards. 

 However the component vendor that will benefit the most might be the one that cracks the challenge of running 10 GigE over multimode building fiber economically and at reasonable distances (300-500 meters).  That issue is the main issue holding back implementation of 1 Gbps to the desktop to be deployed broadly. Who knows if it will ever be needed?  But if the incremental cost of providing it is small, why not?  Gbps comes for free on the desktop PCs.  Right now, 1 Gbps wiring closet switches are cheap and are getting more so every month. But they lack a cost effective “fat pipe”.  That “fat pipe” needs to be 10 Gbps, but it needs to run over the existing vertical fiber, and the cost of the 10 Gbps ports on either end needs to be cheap enough to only add a small amount to the overall system cost.  If someone can enable such a cost point, combined with removing servers as a system bottleneck, GigE to the desktop supported by 10 GigE trunks will become commonplace, and all of the adoption curves above will prove to be far too conservative for both technologies. 

 With-in this new and updated architecture, it’s hard to figure out where the slew of products fit that want to sit in front of the servers (especially web servers) like caches, server load balancers, Port 80 security gateways, compression/acceleration devices, etc. will fit.  If server-to-network edge connections move into the server racks,  the natural point at which to interdict traffic headed to the servers either goes away or is replaced by a much higher speed link.  Most of these devices have benefited from the facts that in the past servers were slow, and most of the client of the servers were separated by a WAN from the servers. Those two facts combined to allow vendors of such equipment to focus on features more than speed.  In fact, some start-ups that focused on developing complex features deployed at high speed, like Nexsi and Desana, are no longer with us. If they could have waited a couple of years, they might have found that the market rapidly shifted towards them, because server farm front-end devices that only run at 100-500 Mbps may not cut it any more.    Multi-gig capability and pretty soon, 10 Gig capability will be required.  Since it may be hard to perform very complex functions at those speeds, such functions, many of which are intended to off-load servers, will just move back onto the servers since they will now be much more efficient.  In that case, what will be needed is software to simplify the consistent deployment of security and traffic management functions across many servers in a consistent manner.

 Ripple Effect Winners

·         Cisco

·         Suppliers to Cisco for high speed devices

·         The optoelectronics supplier who can crack the 10 GigE over multimode fiber problem cost effectively. 

·         Anyone who can come up with a significantly better 10 Gbps total data center/campus network solution than Cisco

·         Server cluster management vendors that enable simplified deployment of common functions and policies across a set of servers.

 Ripple Effect Losers

·         Application Front-End vendors who either don’t keep up speed-wise or don’t figure out how to deploy their capability on the servers (perhaps becoming a server themselves). 

 That ends my series of Musings on the potential Ripple Effect due to the impact of Precision I/O technology.  As a look back on my prognostications, I see that overall what Precision I/O will bring is an acceleration effect to a lot of trends that are already in motion.  But that very acceleration will increase the dynamic range of the impacts on vendors, with those who benefit benefiting a lot more and those who don’t feeling a much more searing pain.   It should be fun to watch.   Especially if you can get on the right side of some of the Ripple Effects.

All historical market numbers used in this Musing are taken from published reports by Del Oro and IDC.  All future projections are my own.