opa-cardQuestion of the month (QoM for February is: Will Intel Omni-Path (Architecture, OPA) GA in scale out enterprise storage by February 2016?

In this forecast enterprise storage are the major and startup vendors supplying storage to data center customers.

What is OPA?

OPA is Intel's replacement for InfiniBand and starts out at 100Gbps. It's intended more for high performance computing (HPC), to be used as an inter-cluster server interconnect or next generation fabric. Intel says it "will maintain consistency and compatibility with existing Intel True Scale Fabric and InfiniBand APIs by working through the open source OpenFabrics Alliance (OFA) software stack on leading Linux* distribution releases". Seems like Intel is making it as easy as possible for vendors to adopt the technology.

There are already implementations on a few HPC supercomputing data centers that currently using OPA (see Intel's OPA overview page for more info).

Why would any storage vendor want to use it?

In enterprise storage we find InfiniBand used primarily in scale out architectures such as EMC Isilon, VMAX and XtremIO. There are quite a few scale-out storage systems that use Ethernet for their inter cluster interconnect, e.g., NetApp,  IBM SAN Volume Controller, and HDS HNAS from established vendors and (NetApp) SolidFire, Coho Data, and Qumulo from startups.  And several that use their own IP for cluster interconnect HPE 3PAR and HDS VSP G1000.

InfiniBand 4 link, FDR supports ~54Gbps, so a single link OPA would almost double this bandwidth.

Moreover, in the HPC space we already know of DDN using OPA in their SFA14K hybrid storage systems.

Not sure whether Intel's OPA can compete with a company's internal IP for cluster/mesh interconnect but any vendor using InfiniBand or Ethernet should see a significant bandwidth advantage in adopting OPA.

Most of Ethernet interconnected, scale-out storage do so because of the economical advantage of Ethernet and its availability across a wide spectrum of off the shelf servers. Whether the speed advantages of OPA would be enough to convince these vendors to adopt is an open debate. Further, it might be more pertinent for all flash arrays than hybrid storage but a case can be made for both.

Any storage vendor looking or currently using InfiniBand would probably be interested in going OPA over time, if the price was right.

Is the hardware available?

Intel released switches and PCIe add-on cards for OPA last year. Aside from Intel,, SuperMicro has also released a Omni-Path 100G 48-port top of rack networking switch and an Omni-Path PCIe add-on card for their (storage) servers that can be used to connect to an OPA fabric. So yes, the hardware is available from multiple vendors today.

Pricing of OPA componentry

Comparing a SuperMicros 48-port OPA switch to a 36-port Mellanox FDR Infiniband switch, current pricing for OPA ports is about 3.2X the price of InfiniBand FDR ports (see pricing at 3D-systems). Although I didn't check on the PCIe cards for the two, I assume the pricing differential for PCIe cards is similar.

The pricing differential for ethernet components is a bit cheaper than InfiniBand so, OPA is about 5.0X more expensive than Ethernet on a port basis. Again, I didn't check NICs vs. OPA PCIe cards but believe the differential would be similar.

Summary:

We have already estimated (see QoW15-001 post) that there are likely to be 13 new hardware systems over the next 12 months. And of that, probably a third are scale-out systems, so maybe, ~4.3 new hardware scale-out storage systems will be introduced over the next 12 months.

The InfiniBand users are probably most likely to adopt it, 2nd most would be internal proprietary bus users and finally, the ethernet crowd. So, for the ~4.3 new scale-out systems: ~1.6 will be new versions of InfiniBand scale out storage; ~1.1 will be new versions of internal IP cluster interconnect storage; and another ~1.6 will be new versions of Ethernet storage.

Given the pricing differential, I believe that 15% of InfiniBand systems are likely to adopt OPA, 10% of internal IP systems are likely to adopt OPA and 5% of ethernet systems are likely to adopt OPA in the next 12 months.

So given those assumptions: ~0.2 InfiniBand system adopters, ~0.1 Internal IP adopters, and ~0.1 Ethernet system adopters or a high likelihood of ~0.4 new hardware storage systems will adopt OPA over the 12 months.

So given the current pricing environment I would forecast OPA adoption in enterprise storage over the next 12 months at NO with a 0.91 probability. 

Forecasting contest updates:

QoW15-002 Will 3D TLC NAND GA in major enterprise storage systems by YE 2016 is now closed.

EMC announced 2/29/16 that new VMAX All Flash will ship with 3D TLC NAND SSDs.

Also Dell has been shipping 3D TLC NAND SSDs since last June in their Compellent Storage systems SC8000, SC4000 and SCv2000 storage systems. I had not considered Dell to be a "major enterprise storage" vendor before but I must confess a number of analysts have a different opinion.

My latest QoW15-002 forecast was No with a 0.53 probability, I was wrong (see my original forecast on QoW15-002). I will post something later in March about what went wrong when I figure it out.

Two other forecasters had it correct: one with a forecast YES at 1.0 probability and the other had it as a YES at 0.95 probability.

So far I am 0 for 1, and they are both 1 for 1....

Other updates: there were no other updates to the current outstanding questions of the week/month.

Photo Credit(s): TechReport News website