SC11 Wrap-up: Japan Takes the HPC Lead; FDR Takes the Spotlight

The Supercomputing Conference 2011 (SC11) was held in Seattle, Washington the week of November 12–18. This is the event for high-speed everything, from servers and switches to transceivers and interconnects. Besides seeing what is in store for the next-generation, exotic high-performance computers (HPCs) and interconnect schemes, SC11 also shows what is in store for the mainstream datacenters only a few years from now. Transceiver, active optical cable (AOC), and cabling vendors’ new product announcements and technology demonstrations were in full swing, as many companies used SC11 as a launch platform for products. March 2012, at OFC 2012 in Los Angeles, will be the next product launch window. At SC11, key announcements were

Japan Takes Pole Position in Top500 HPCs
At four times faster than last year’s winner China’s Tianhe-1A system, Fujitsu built the K computer with 88,128 SPARC64, eight-core microprocessors for a total of 705,024 cores—equivalent to over 1 million desktop PCs linked together. LightCounting could not confirm the interconnect scheme, but each CPU appears to be linked via ten 5Gbps links (most likely PCI Express Gen2) enabling the use of copper, instead of optical, links. We believe the K computer does not use any AOC or optical interconnects at all, opting for copper to connect 864 racks in a six-dimension Mesh/Torus configuration and consuming 10MWatts of power.

Fujitsu’s Kcomputer



100G and SCinet
The SCinet network demonstration presented one of the most sophisticated high-speed networks in the world; boasting over 100 miles of fiber cable and bandwidth of 450Gbps; it links national labs and trade show booth systems. Last year, 10G links were all the rage, and we saw only one 100G transceiver in the SCinet networking demonstration. This year, it was all about 100G. We counted three wide-area, 100G networks and approximately 20 100G transceivers in various live demonstrations, as well as a 100G network linking Oak Ridge National Laboratory to Juniper Network’s booth.

FDR Is Here
Last year, SC10 proclaimed, “QDR is here and FDR is coming.” This year, the SC11 show asserts, “FDR is here and EDR is coming.” QDR (quad data rate) is the 40G InfiniBand format using 4´10Gbps links, which became very popular for HPC's using passive copper cables and active optical cables. SC11 kicks off the upgrade cycle with the next-generation systems and connectivity infrastructure centered around 56G FDR (fourteen data rate) or 4´14Gbps. As with QDR, QSFP MSA is the preferred format, with CXP hardly seen anywhere except for a few vendors in the Ethernet space.

While the jump from 10Gbps to 14Gbps seems small and maybe not worth the upgrade cost, the real story is that that error encoding scheme also changed from 8b/10b, consuming about 20% in performance penalty, to 64b/66b, which consumes almost no performance penalty. So the increase in bandwidth is really from 8G to 14G—about a 75% increase.

InfiniBand is used in 42% of the TOP500 HPCs for low latency, with Ethernet and custom protocols accounting for the remaining percentage, where low cost is preferred over high performance. FDR is primarily an InfiniBand concern, with little interest from the Ethernet community because it is off their speed roadmap of 10G, 40G, and 100G line rates. However, the 14G data rate does line up with the Fibre Channel 16´ data rates. The Fibre Channel community often refers to the speeds as 8G and 16G, but the correct terminology is “16´,” or 16 times the original Fibre Channel data rate. The actual line rate is 14Gbps. The 4´14G active optical cables (AOCs) fit nicely with the 16´ Fibre Channel data rates, and AOCs may finally gain some traction in Fibre Channel, in which adoption has lagged to date. At EDR, or 4´25G data rates, the InfiniBand and Ethernet roadmaps will converge once again, but this is not likely to occur until late 2013 at the earliest.

A major InfiniBand system supplier noted that FDR direct attach cables (DAC) were having trouble reaching 5 meters and maintaining signal quality. At 10 Gbps, direct attach cables could reach 7 meters passive and about 15 meters with active signal processing chips embedded in the cable ends (called active direct attach). At 25Gbps, direct attach will likely only reach 1–2 meters, and if active ends can be built at all at 25Gbps, the reach might be 3–4 meters at best. This will bring optical transceivers and AOCs closer and closer to the server and likely to become the choice of interconnect at these high speeds. Starting in HPCs, this is a precursor as to what will eventually ripple down to the commercial datacenter.

Key Announcements

Finisar’s Embedded Module; Avago/Altera FPGA PCIe Optical Line Card; 3M Ribbon DA Cable



Finally, HPCs are being used to accelerate Hadoop processing in specialized clusters starting the low-latency interest in Web 2.0 applications along with flash trading for financial applications and continuing the spread of HPCs outside traditional scientific computing. Hadoop was inspired by Google's MapReduce and Google File System (GFS) to support data-intensive distributed applications. It enables applications to work with thousands of nodes and petabytes of data—which sounds “HPC-ish”!

AOCs and DA Cables
There seems to be no end in sight to the continued stream of companies entering the active optical cable and direct attach cable markets—all gearing up for the next generation of servers and switches. LightCounting also picked up rumors of a significant number of new HPCs planned to be built that will use AOCs. 3M introduced its foray into direct attach cables and active optical cables. The coaxial ribbons enable a flat ribbon of coaxial cables that can be bent approximately 90° and soldered as a unit in a single step, promising further reductions in manufacturing costs. 3M is one of the largest manufacturers of optical fiber connectors systems that eliminates the time-consuming glass/ceramic polishing steps. 3M has applied this technique to building AOCs and eliminated the costly lensing assemblies to mate fibers to VCSELS and detectors. This promises to further reduce AOC manufacturing costs and market prices. At this point, the OEM price for a 10-meter, 4´10G AOC using QSFP is less than the price of a single 4´10G QSFP transceiver!

LightCounting is publishing its updated AOC report in December titled AOCS Supercomputers to SmartPhones that includes not only the new FDR products but also more extensive coverage on consumer optics such as HDMI, USB, Thunderbolt, and MIPI optical interconnects for use inside future smart phones and tablets.

Brad Smith, VP & Chief Analyst,