Corporate, Customer, and Academic Open Source Communities for Next Generation Software

I submitted a paper for the NSF's Cyberinfrastructure Software Sustainability & Reusability Workshop.  I discuss some of the innovative open source models we've used to develop Red Hat Enterprise MRG:

The NSF is looking both at models of how to build sustainable cyberinfrastructure software as well as specific software that will benefit its goals like providing communities with access to a “world class high performance computing (HPC) environment.” Red Hat Enterprise MRG, a high performance distributed computing platform which integrates Messaging, Realtime, and Grid capabilities, provides both an open source model of how academic researchers, customers, and corporations can collaborate as well as powerful software infrastructure which can help the NSF meet its next-generation cyberinfrastructure goals.

For example, I discuss the partnership we have with the University of Wisconsin around Condor for Grid scheduling:

This partnership between UW and Red Hat is adding many innovative capabilities to Condor and also expanding significantly Condor's reach from research environments to enterprises. For example, Red Hat has focused on adding many capabilities which enterprises require for deployment but which are not paramount for academia. These enhancements range from new graphical management tools to enterprise maintainability to concurrency limits on scarce resources like software licenses. Furthermore, Red Hat has also focused on advancing Condor towards utility and cloud models of computing by adding capabilities like libvirt virtualization support and Amazon EC2 integration. Many enterprises are now looking at MRG and Condor for building private clouds and moving to cloud computing.

The paper also discusses how we're feeding back developments from the enterprise into academia, how we've collaborated with customers and users around AMQP and messaging, how the technologies in MRG can help the NSF meet its software goals around sustainable HPC, and so on.

You can see the submission at the NSF's site for papers.

You can also directly download a pdf of the paper from the NSF's site: Corporate, Customer, and Academic Open Source Communities for Next Generation Software.

Software is Low Latency's Weakest Link...Unless You're Running Red Hat Enterprise MRG

There's an article in today's Wall St and Technology, Software is Low Latency's Weakest Link.  The article states:

The weakest link in the low-latency value chain is older software or poorly written code, not market data feeds, lack of ultra-fast processors or older networks, according to experts at Wall Street & Technology's Accelerating Wall Street 2009 conference.

Furthermore,

"Within the applications, we see the greatest opportunity to improve latency," said Rob Wallos, global head of market data architecture at Citi during the session The Complete Low Latency Value Chain. "Hardware can give you a generic 20 percent improvement in performance, but there is only so far you can go with hardware. "

This is certainly true in general.  We have found that even moving up to exotic hardware like infiniband only provides incremental gains in performance unless you optimize your application to take advantage of that hardware.  That's why we've been working with hardware manufacturers and optimizing Red Hat Enterprise MRG for the latest hardware as well as for Linux.


For example, MRG includes RDMA drivers to achieve extremely low latency on infiniband.  Compare the results of MRG Messaging running on infiniband with standard TCP versus with our RDMA drivers:

This chart illustrates latency using 1K messages at a sustained throughput of 50,000 messages/second using MRG Messaging.  This is for fully reliable application-level latency between three peers: a producer is sending a message through a broker to a client, and that message is acknowledged back.  So, there are four network hops in this case.  As you can see with standard TCP (the red line), the latency is nothing ground-breaking at around 1 millsecond, even though we're using infiniband.  In other words, moving the application to infiniband doesn't automatically improve latency significantly.

With our RDMA drivers , however, MRG can achieve latencies an order of magnitude better--about 70 microseconds on this same particular hardware.  And, remember, this is for 1K message sizes and 4 network hops!  So, yes, it is true that in general, low latency bottlenecks are in software.  However, we have been developing MRG to take full advantage of modern hardware and also Linux, so we've pushed the bottlenecks back to the hardware domain in many cases.

I should also note that our next release of MRG will include a cross-memory driver for co-locating our messaging broker with an application client.  This would remove one of the network links for latency and should allow us to halve our latency performance.

QCon Presentation Slides For Download

I presented Next-Generation AMQP Messaging Performance, Architectures, and Ecosystems with Red Hat Enterprise MRG last week at QCon London, along with Carl Trieloff.  The talk included a detailed use case of work we've been doing with customers to build financial trading systems and stock exchanges with MRG.  Messaging deployments don't get any higher end than this, so it's an interesting study. 

In particular, we've added capabilities in MRG like LVQ, Ring Queue, and TTL so that companies can build market caches or provide updated streaming quotes directly from the messaging broker rather than build these capabilities themselves.  We've also done a lot of work to get some pretty impressive performance (e.g. fully reliable latency across three peers in around 60 microseconds).  And, of course, we support capabilities like active/active clustering for high availability and federation for disaster recovery.

Beyond the case study, we also talked about the new ecosystems that are forming around AMQP and MRG as well as new types of messaging-oriented applications people are building now.

You can download our slides from the talk here.

QCon London

I'll be at QCon London next week.  Red Hat is sponsoring a booth, and we'll be giving two talks:

• AMQP in Action
• Next-Generation AMQP Messaging Performance, Architectures, and Ecosystems with Red Hat Enterprise MRG

We've got a good customer case study and some pretty interesting features, use cases, and performance data that we'll be presenting around messaging.  Come to our talks and stop by our booth if you're attending the show.

QPid is a Top-Level Apache Project

The Apache Software Foundation announced today that it has elevated QPid to a top-level project, based on the accomplishments it has made in developing a community and software.  We did the initial submission of QPid to Apache, so it's gratifying to see that it has come such a long way now and has developed such a large community and following--including some surprising members.

"On the heels of its recent graduation, Qpid has also reached the completion of the major Qpid M4 release. We're thrilled to have our project's growth and maturity recognized by the Apache Software Foundation," said Carl Trieloff, Chair of the Apache Qpid Project Management Committee (PMC) and Senior Consulting Software Engineer at Red Hat. "With the promotion to an Apache Top-Level Project, Qpid is recognized for outstanding development based on our vibrant, rapidly expanding community, infrastructure, and for collaborative development."

John O'Hara, Chairman of the AMQP Working Group and Executive Director at JPMorgan said, "I am delighted that the Apache Software Foundation has graduated the Qpid project. AMQP is an open infrastructure for business messaging over the Internet. Apache Qpid developers have been active participants in the AMQP Working Group working in partnership with other AMQP solution developers and end-users. The ASF's provision of Qpid as its AMQP implementation adds to the range of AMQP solutions businesses can choose from to improve their efficiency." 

Read more in the full press release from Apache.

Check out QPid's new home: http://qpid.apache.org