System Interoperability's Impact on Hyperscale Data Centers

Madhumita Sanyal, Priyank Shukla

Jan 18, 2022 / 5 min read

When seemingly disparate things can come together, the results are often that much better. Like interlocking puzzle pieces that reveal a pretty picture. Or a sports team that draws players from different backgrounds—some groundwork, good coaching, and a healthy culture can bring out the best from each individual to form a winning squad. In the chip design world, we can find this kind of symbiosis in interoperable devices.

The importance of interoperability in an electronics ecosystem cannot be overstated. As a designer, when you’ve got different pieces that work together seamlessly, you can be assured of a smoother, faster, more efficient design process. You don’t have to worry about testing various scenarios to make sure that you’ll get the desired results. That work will have already been done.

In this blog post, we’ll center our discussion around IP and interconnects to show how their interoperability is benefiting demanding applications like high-performance computing (HPC) and artificial intelligence (AI). Read on for more insights.

Person walking in front of data center

Data, Data Everywhere

HPC and AI rely on data centers—in many cases, hyperscale data centers—to process and compute massive volumes of data quickly. Indeed, the world of data center compute, networking, storage, and optics is being impacted by two very powerful underlying forces: more data and complex data. The increasing volume of data is outlined by the table below, based on information from the April 2020 IEEE 802.3 Industry Connections Bandwidth Assessment.

IEEE Bandwidth Assessment Report | Synopsys

The emergence of hyperscale data centers—which boast the technical infrastructure to manage petabytes or more of data—has spurred growth in machine-to-machine (M2M) traffic. Inside the box are a variety of compute resources, like CPUs, GPUs, accelerators, and smart network interface cards (NICs) that communicate with each other primarily through the PCI Express® (PCIe®) or CXL protocols. On the box-to-box side, switches and routers based on Ethernet standards handle networking functions from 100G to 800G to 1.6Tb data rates. In addition, key to driving warehouse-scale efficiency in HPC are interconnects.

Providing the electrical connectivity between circuit elements, interconnects play a critical role in how well a chip performs. A large IC can contain around 30 miles of interconnect wires to transport electrons and connect transistors. The interconnect’s speed in moving signals and power through these wires affects chip performance. With compute-intense applications that rely on real-time processing, interconnects demand interface IP that supports high throughput, low latency, and power efficiency. Interface IP aligns with the high-speed protocols that dictate data transfer speed on the interconnect, such as PCIe 5.0/6.0 and 400G/800G Ethernet.

Innovating Ahead of the Standards

Over the last several years, the pace of innovation has truly been breathtaking, surpassing what the standards have defined. It’s no longer uncommon to find chipmakers rolling out systems before the next generation of a given standard has been ratified. How do you ensure that all of your parts will work together—especially in chips for HPC and AI, where large, complex, hyper-convergent designs are becoming increasingly common? This is where interoperability between the IP and interconnects makes a world of difference.

Without proven interoperability, you’ll need to test each component under a variety of scenarios to ensure that they’ll work well together. A complete, interoperable solution, on the other hand, provides a credible proof-of-concept that can give you a head start on your design. Then, once the standard has been approved, you’ll be that much ahead of the game.

Synopsys is engaging with ecosystem partners such as Amphenol, Keysight, and Samtec to demonstrate successful interoperability for PCIe 6.0 and 112G PAM-4 systems from IP to interconnect, targeting HPC and AI designs. One example of interoperability success comes from the long-standing collaboration between Synopsys and Samtec, a global manufacturer of a broad line of high-performance interconnect solutions. In addition to the teamwork between the two, both companies also work closely with standards bodies and are aligned with protocol roadmaps.

Among the fruits of the collaboration? Demos that show how Samtec’s interconnects work with Synopsys’ interface IP under a variety of real-world conditions. Mutual customers would have to do their own system-level testing but otherwise gain a platform that simplifies and shortens their design process and frees them up to focus on their core competencies.

Showcasing PCIe 6.0 in a Hyperscale Data Center

At last year's AI Hardware Summit and SC21, the two companies demonstrated a comprehensive PCIe 6.0 and 112G Ethernet solution, featuring the IP, ICs, PCBs, connectors, and cable assemblies. As a high-speed interface that is getting faster with each generation, PCIe and Ethernet are well suited to HPC applications. Deployed in a hyperscale data center with state-of-the-art channels, the demos featured:

PCIe 6.0:

  • 2e-8 raw PHY pre-forward error correction (FEC) bit error rate (BER) versus 1e-6 BER target of the PCIe 6.0 specification
  • All four lanes operating at 64 GT/s across channels with ~36 dB loss
  • Package, PCB and connector loss, reflection, and discontinuities mimicking the PCIe 6.0 channel

112G Ethernet:

  • 1e-8 BER
  • 37 dB of channel loss
  • Maximum performance in a transmitter and receiver loopback mode

While the full version of PCIe 6.0 could come in early 2022, many chip designers have already been developing designs to take advantage of the faster data transfer rate (64 GT/s per pin). In fact, some are looking ahead to the even faster bandwidth that the next generation of the standard is expected to bring. This is also true for 112G Ethernet: while standards organizations are defining the standard for 100G, 400G, and 800G Ethernet, Synopsys is already demonstrating successful interoperability with our ecosystem partners. So, demos such as the one from Synopsys and Samtec provide mutual customers with the ability to see how their design might work in their unique use cases.

The benefits of interoperability are clear for designers. For the companies engaged in collaborations, their working relationship also gives them a head start in developing their solutions for emerging standards. For example, say Synopsys is developing new SerDes IP. By obtaining, in advance, the channels from Samtec, Synopsys engineers could design the IP based on the latest specs, while compensating for real-world impairments before silicon validation. In other words, Synopsys, too, can do some early groundwork to ensure that the IP will work smoothly with the connectors, modules, and cables.

Summary

Chip designers commonly use components from multiple vendors, but these components aren't always designed from the start to work together. When the components are interoperable, however, designers can shorten their time-to-market and simplify their design process, while also getting a jump on the next interface standard. For data-intensive applications like HPC and AI, which have spawned new SoC and system architectures, interoperable system-level IP and interconnects provide a productive path toward meeting their highly demanding requirements.

Continue Reading