syntatic

The Closet JRubyists

with 15 comments

For too long we’ve let the JRuby core contributors be the only voice for JRuby. I for one am guilty of taking and taking and taking from the tireless and thankless work the JRuby team has done. Charles, Ola, Tom, Nick, Vladimir and many others need to be thanked.

Almost all of the JRuby projects I’ve been aware of or a part of are nowhere to be found on blogs, twitter or any other techno-coder communication flavor of the month. These projects aren’t going to become popular or have the codista spotlight on them. Most JRuby work is done in the deep inner workings of the corporate bureaucratic sinkhole that is enterprise IT. JRuby work is hidden behind non-disclosure agreements and kept secret because of the technological edge secrecy provides. The great stories haven’t been told and Charles is only able to hint at them because they really aren’t his to tell.

This is one such story and I hope that this post encourages other JRubyists to speak up and at least share parts of their JRuby experience. You owe it to the JRuby team and the Ruby community in general.

I’ll start out by being blunt and if you want to dismiss the rest of the post due to the next sentence then go ahead and move along because this post is not for you. JRuby is fantastic. The rest of this post will hopefully explain why that statement is true.

I joined a project that started out using MRI to wrap a C library, which built into a gem. The C library is a financial analytics package used to price instruments and extract contract specifications. Working in C with MRI was easy. The Ruby C API methods are simple and you almost get the sense you’re working in Ruby. Everything was lollipops and gumdrops just as working with Ruby should be. A rails application was built to display data provided by the gem. As the rails community moved through new deployment strategies so did we moving from webrick to mongrel with lighttpd, etc.

Then some of the business specifications required pricing to be done on hundreds of thousands of instruments at a time. An order of magnitude change in usage made speed and memory usage become very important. Pricing that many instruments in 3 months is not helpful. It needed to be done in parallel.

With this many instruments needing to be priced my team and I created a simple system for distributing the data and processing it in parallel using DRb similar in many ways to Hadoop. Had we been using JRuby at the time Hadoop would be perfect to wrap, but MRI didn’t give us that option.

Right around this point in time MRI became a huge bottleneck. MRI wasn’t going to handle the over 6 million objects we needed to push through DRb and even if it could get data to the workers on a grid of machines it couldn’t fully utilize all the cores on each machine. A combination of running out of memory and MRI failing to fully utilize all the cores of 64-core servers ground the project to a halt.

JRuby 1.0 had just been released and was starting to gain some traction. A 1.0 project? Certainly that can’t handle these problems. With nowhere else to go we took part of the C Ruby API and moved to a C, JNI, Java and JRuby stack. The new stack of tools wasn’t lollipops and gumdrops, but if it worked then who cares? Not me. I enjoyed the polygot work passing ruby objects into C callbacks and unit testing C from JRuby. Mind streaching stuff.

Turns out JRuby had no problems managing 8GB of memory and 6 million plus objects being passed around over DRb. Having the JVM do memory management for your Ruby objects isn’t that bad. I didn’t have to care about it anymore and not caring about the JVM is light years ahead of caring about MRI memory management. Yes, there is real value to using the JVM.

Additionally, JRuby fit into the rest of our MRI system because we weren’t having any problems with MRI talking to JRuby over DRb. I ran into a few problems with IO and Socket, but Charles and Ola were available via IRC and the problems were fixed in a matter of days. The availability of the JRuby team is something I haven’t found in any other community. Charles always put my questions before his other tasks and if you know anything about the man, he is busy. I don’t know how many talks he’s done recently, but his twitter messages list so many cities I’m not sure where he lives anymore.

The initial pricing times came in at around an hour and fifteen minutes. Not bad considering the client was ok with 2 days. JRuby FTW!

Now the story could end here and I’d consider the transition to JRuby a success, but the story goes on.

Tweaking the JVM options allowed us to move the time to about 45 minutes once we upgraded to JRuby 1.1.1. I added some of my findings to the performance tuning wiki page, which you can find here. When was the last time you heard of someone passing some options to MRI’s garbage collector and see performance increases?

Exporting this much data turned out to be a problem as well. Excel’s 256 column limit wasn’t to happy about my 9,000+ column files and the standard ruby spreadsheet gem had trouble handling anything more than 7MB. Fortunately, the Apache POI (Java) project could handle some these problems as well as other features like auto-sizing columns and freeze panes, which no other MRI compatible gem could provide (Yes, there are Ruby POI bindings). I never thought I’d enjoy working with POI/Excel’s API, but JRuby plus POI libraries had me smiling. Excel with a ruby feel rocks. Using JRuby to wrap pre-existing Java solutions is a great way to sleep at night.

Next we moved our Rails apps over to JRuby by deploying them as wars in JBoss.  Managing the mongel problem was gone and JBoss turned out to be much faster anyway provided you give it enough memory. Nick Seiger has done some great work with warbler and the process was a breeze.  Unfortunately, with the number of apps we moved over the DBAs were starting to get upset about the 60+ database connections we used. Rails 2.2 wasn’t around yet so connection pooling inside of Rails wasn’t an option, but using JNDI inside of JBoss worked perfectly. Using prexisting Java tool’s with an adapter written by the JRuby team made my job a lot easier again.

Meanwhile, JRuby was still releasing new versions. Using 1.1.3 moved our pricing time to about 15 minutes. Yes, from 1 hour and 15 to 15. There were some other tweaks we made along the way, but the most significant improvements came from JRuby itself. In it’s current state the C/Java/JRuby API is now exposed through Merb (http/json), DRb (druby), Rails (http/xml, http/html) and an Excel plugin and more opportunities are ahead.

We’re able to upgrade to a new version of JRuby within a day of release. Yes, it is that stable and easy to switch. Yes, 1.1.5 is currently in our production environment. Upgrading to new versions of MRI was usually a nightmare for me so I welcome the stability. JRuby being a jar has some wonderful benefits.

I won’t go into detail about the other libraries we’ve wrapped with JRuby including JFreeChart and QuicKFIX/J. I won’t go into detail about using JRuby with CORBA or RMI and the many other tools that become available to you with the use of JRuby.

Currently, MRI isn’t even installed on our production servers and I don’t see it being installed in the future. Most if not all the ways that the data is available or usable is due to JRuby. JRuby made my job much easier and many of the features I’ve implemented possible. Give it a try.

About these ads

Written by syntatic

November 25, 2008 at 11:11 am

Posted in programming, rails

Tagged with

15 Responses

Subscribe to comments with RSS.

  1. I’ve had a similar experience, albeit on a much, much smaller scale. Ruby is the best “glue” that I’ve come across… it makes modeling the domain very, very easy, and with JRuby, you have easy access to very mature libraries. JVM / CLR hosted languages are going to continue to grow in importance. JRuby is better than new scripting languages because it already has a wealth of libraries to support its role as glue code, blog posts and mindshare.

    Aaron Blohowiak

    November 25, 2008 at 8:54 pm

  2. FYI, if you’re willing to upgrade, Excel 2007 supports 16k columns: http://blogs.msdn.com/excel/archive/2005/09/26/474258.aspx

    Wayne Kao

    November 25, 2008 at 11:09 pm

  3. Great Blog entry. Have you looked at the recent work Bob did on further integrating JRuby with JBoss? You should look at his various log entries here: http://oddthesis.org/posts/2008-11-mailing-lists-for-jboss-rails-and-jboss-cloud

    If you have any further integration you’ve done, feel free to contribute them back!

    cheers,

    sacha

    Sacha

    November 26, 2008 at 3:53 am

  4. [...] αρκετά ενδιαφέρουσα εξέλιξη, μπορείτε να διαβάσετε εδώ. Κατα την γνώμη μου μία τέτοια προσπάθεια φέρνει την [...]

  5. Excellent post!

    Thank you

    cvasilak

    November 26, 2008 at 5:52 am

  6. Thanks for the great article about JRuby. I think because a lot of rubyists moved from Java there is a tendency to dismiss it as the bad old days, but the MRI team just isn’t pushing out enough improvements rapidly enough for Ruby to start getting enough traction with it. I’m saying this even though I write almost nothing but Ruby these days.

    I love Ruby and I’m glad to be done writing Java, but it looks JRuby is worth a try. The only thing I’ve heard that is really holding it back is lack of support for native gems.

    Jeremy Gailor

    November 26, 2008 at 8:57 am

  7. Thanks for posting this. I’ve wondered for a long time where all the JRuby bloggers were.

    We’ve been using JRuby for a few months at work, and ever since the gem memory issues were fixed, it’s been working quite nicely. Now if they could only fix the stack trace problems…

    Phil

    November 26, 2008 at 11:09 am

  8. Great article, we are also using JRuby (+Rails) but have some trouble to deploy our war file to IBM WAS 6.1.

    phx

    November 27, 2008 at 12:53 am

  9. This is great. I was looking for some info on grand scale resource hogs running on JRuby and it was hard to find.

    It is particularly exciting to see how in a multi-server environment you were able to upgrade only the servers that really needed JRuby and that this was possible through DRb messaging. I expect that in the long run we will be doing something similar.

    Shimon Amit

    November 27, 2008 at 1:24 am

  10. [...] The Closet JRubyists For too long we’ve let the JRuby core contributors be the only voice for JRuby. I for one am guilty of taking and [...] [...]

    Top Posts « WordPress.com

    November 27, 2008 at 6:07 pm

  11. It is reassuring to know that JRuby can achieve such a high performance.

    I was hoping to use jruby for a project I am doing right now but my performance results discouraged me to go that route considering the urgency and performance requirements.

    Here is an entry on experience with JRuby:

    http://saaridev.wordpress.com/2008/11/15/jruby-stochastic-simulation-with-ssj/

    imsaar

    December 3, 2008 at 10:44 pm

  12. Great post. I recently went through the same. Now have a Merb app running on Glassfish with JRuby (more info on http://www.rubytutorials.net/2008/12/03/glassfish-v3-with-merb-and-warbler/). Very much an improvement! Also the core team on IRC is wonderful.
    Go JRuby!

    Harm

    December 5, 2008 at 8:21 am

  13. [...] holes and tell the world about their experience with JRuby. I believe the call was inspired by this blog post by Brian Tatnall. So, I figured I’d throw in my two [...]

  14. great post, actually learned something for once :D

    closet organization systems

    December 1, 2009 at 2:24 am

  15. I love java and now i`m still writing it, but this JRuby looks fun i`ll give it a try.

    YUI

    March 21, 2010 at 7:17 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: