The 2010 Supercomputer conference and trade show (SC10), in New Orleans held November 13–19, focused on high-performance computing (HPC) served up interesting developments for those interested in high-speed interconnects, especially considering technology viewed here often trickles down to volume commercial use in both datacom and telecom.
With over 10,000 attendees and 350 exhibitors, SC10 has become the largest computer event in the world. It showcases everything from HPCs at national labs and large engineering and science companies to desktop superPCs filled with three NVidia graphics cards used as parallel processing engines. This event has more high-speed optical and copper interconnect component and semiconductor companies and server and switching and router companies on display than any other event LightCounting analysts have seen. It has clearly become the computing-centric “high-speed everything show.”
On that note, the big news was the Chinese claiming the high-water mark for the fastest supercomputer in the world. China also now claims three of the top five slots and is poised to dominate the TOP 10 by 2011. Called Tianhe-1A, China’s supercomputer combines off-the-shelf Intel microprocessors and Nvidia graphic processors used as parallel processors with graphics engines and Emcore’s active optical cables to construct computing clusters. Tianhe-1A uses a dual-QDR active optical cable configuration that is channel bonded to create an 80 Gbps parallel link connected to custom silicon switch chips. Called Galaxy, these chips convert the bonded link to conventional 5 Gbps links running over the PCI express 2.0 bus. LightCounting believes that that a single 8-bit word is “stripped” across the eight 10 Gbps channels enabling a single 8-bit word to transfer in one clock cycle.
While Tianhe-1A is only 49% faster than the dethroned Cray Jaguar, this event is likely to spark a competition between the United States and China and serves as a wake-up call to the U.S. government. Rumors report 10 or more systems on the way from various provinces and governmental agencies in China, but no purchase orders have been confirmed. Europe and Japan still have a weak showing with only 1 HPC each in the top 10.
Global HPC revenues grew to approximately $20 billion in 2009 during one of the worst economic years on record. Interestingly, the high-end HPC segment grew 65% in revenues! That figure is not surprising, as the average selling price of a high-end system approaches $100 million. While the perception is that HPCs are used for exotic scientific applications, over 60% of the applications are in traditional science and engineering, manufacturing, oil and gas exploration, finance, and bioscience. New superPCs are enabling propagating supercomputing to the masses by making it affordable to even small engineering organizations. SuperPC will be used in conjunction with HPCs that run large tasks. What is truly amazing is, riding off the business model created by PC Gamers, the technology to create a SuperPC is available from consumer electronic stores and ten years ago the equivalent computing performance would have cost $10 million and cover the floor space of a medium sized house!
One-Quarter Terabit Network - Constructed for a Trade Show?
SC10 constructed a 270 Gbps high-speed optical network to demonstrate and interconnect the systems on the show floor. Valued at over $40 million, it consists of approximately 200 miles of optical fiber, sixteen 10 Gbps wide-area links and one 100GbE link. Boasting bandwidth greater than many countries and capable of transferring an amount data equivalent to the Library of Congress collection in less than 1 minute, it represents one of the world’s fastest networks—but amazingly, it’s been created for a week long trade show and assembled by volunteers! This illustrates how fast a high-speed optical network can be constructed in a data center environment.
Vendor Announcements and Floor Action
Oracle/Sun Introduced Sparc-based SuperCluster powered by the Sparc T3 16-core microprocessor, putting an end to the rumors and debate that Oracle would exit the InfiniBand HPC and storage business. This fits well with their Exadata storage/database systems which are becoming very popular.
Mellanox introduced new 8- and 18-port InfiniBand switches.
10GBASE-T: InfiniBand is used in 43% of the Top 500 HPC clusters and implemented about 25% with AOCs and 75% with Direct Attach copper cabling. Although, roughly 50% of the HPCs clusters use Cat-5, 1G Ethernet, the presence of 10GbE at SC10 was underwhelming! With high heat the main culprit in HPC system, the current power consumption of 10GBASE-T chips makes it unviable at the current time for use in a computing cluster with thousands of links. Only a few vendors showed 10GBASE-T Ethernet on the floor. Stay tuned next year and the story may be very different. There was more said about Direct Attach, 14G FDR and 25G EDR than 10GBASE-T copper and RJ-45. The shift to 10G Ethernet in HPC will happen only once it is on every server motherboard as is the case with 1Gb Ethernet. LightCounting is preparing a focus report on 10GBASE-T due out in January.
InfiniBand FDR and Board-Level Optics: While opinions vary about the future adoption of FDR because of the small data rate jump from 10G to 14G, what many do not realize is the InfiniBand Trade Association changed the error encoding from 8b/10b to 64b/66b. 8b/10b used in QDR has a nearly 25% bandwidth tax, whereas 64b/66b does not impact bandwidth much at all, meaning that the effective bandwidth almost doubles from QDR (10G minus 25% = 7.5G) to FDR at 14G.
It is clear now that 14G FDR is the next step for HPCs clusters, because the implementation of 25Gbps–based EDR still faces many technical challenges in all aspects of the transceiver. Implementing EDR is proving to be a lot harder than expected to implement in both the transmit and receive sides; then too, it requires the MSA to change to the “Z” versions of QSFP and CXP and new switch chips to be designed. Tyco and Molex discussed the new zQSFP and zCXP supporting EDR 25Gbps signals.
In other news, Finisar showed working 14G FDR active optical cables and its new embedded board-level optical engines. Avago announced an optical 10G mini-SFP+ enabling higher front panel density and its 12x10G miniPOD optical engine for high-density board-level optics, which is destined to replace the aging SNAP12. Unlike the ceramic-based microPOD, the miniPOD is suited for use with traditional FR4 PCBs. Avago also introduced 4x10G and 12x10G CXP parallel optic transceivers. While LightCounting could not confirm it, the 12x10G CXP transceiver is likely to consist of the same optical engine as the miniPOD.
Luxtera discussed its up-and-coming 25 Gbps silicon photonics offering and announced 25 Gbps receiver technology complementing its previously announced transmitters. Tyco showed a 1x10G SFP+ and 12x10G CXP active optical cables.
More vendors continue to stream into the optical transceiver, AOC and Direct Attach space. Foxcon, Santec announced their entrance into 4x10G AOCs and Meritec in both AOCs and the Direct Attach business. AlpenIO, whose roots date back to the Intel LightLogic acquisition, introduced a line of QSFP and PCI Express AOCs using miniature optical engines supporting InfiniBand QDR/FDR, 40GbE, and PCI Express Gen3. Lastly, Hitachi Cable showed but did not announce yet a new high-speed coax cable for Direct Attach with the capability to extend the data rate and reach of Direct Attach cables and challenge both Cat6A and AOCs. While Hitachi is still characterizing the cable, it might be possible to reach 10+ meters at 25 Gbps EDR rates – with copper! Someone once said, “Never underestimate the incumbent technology (copper)”.
LightCounting is preparing a focus report update on Active Optical Cables due out in December.
Mellanox to Acquire Voltaire
Right after the SC10 show, on November 29, Mellanox announced that it was acquiring Voltaire for $218 million. Both companies are based in Israel. Net of cash, the deal is for $176 million. With the acquisition, Mellanox acquires its largest distributor whereby Voltaire buys InfiniBand switching and adapter chips and resells them along with its management software to end users. At this point, InfiniBand is a two company market: Mellanox and QLogic, with Mellanox the much larger player. With the acquisition, Mellanox acquires its distributor, allowing it to recoup the markup that Voltaire was charging, the management software, an end user sales force, and incidentally, keeps Voltaire from possibly acquiring QLogic’s InfiniBand business. The third point is key because with QLogic’s InfiniBand business, Voltaire could have been a major thorn in Mellanox’s side, as Voltaire is only focused on InfiniBand while QLogic has Fibre Channel adapters and switches, Ethernet adapters and switches, and its InfiniBand business that it is not very focused on. Mellanox can now go after the market with the full cadre of tools: chips, software, brand (it will be interesting to see if they drop the Voltaire brand), and sales and service to end users. Mellanox has been developing InfiniBand to Ethernet switching for years, because it sees the limitation of the pure InfiniBand market almost entirely in the HPC space. Obviously, the market can only carry two players, and we will see if it drops to one anytime soon. With the gain in market share within the InfiniBand market, the economies of scale should allow them to dominate. Look out QLogic.
