I started an evaluation of Java web services performance based on continuing problems in this area I'd noticed personally, along with reports on some of the framework mailing lists that made it clear my experiences weren't unique. My first cut at this was relatively modest, with results shown in one of the slides from my Cleaning up SOAP Web Services presentation (worth a look even beyond the performance discussion). The performance tests have become more comprehensive since I delivered the initial version of that presentation in late 2003, so I've added this more detailed discussion of the current tests and results as an addendum to the presentation material.

In order to run a reasonable comparison of web services framework performance I wanted several particular characteristics in my target service:

  • It should work with real data, rather than just repeated values or other artificial patterns
  • It should use efficient XML representations (in the case of doc/lit implementations - for rpc/enc there's no control over this)
  • It should have low processing overhead (to keep the focus on the framework performance)
  • It should allow scaling of the size of responses, so that it's easy to test the relative performance with different data volumes

After looking into several alternatives, I settled on a service that would run queries against a database of earthquake information. I was able to find several public sources of raw earthquake event information online with no redistribution restrictions, which was one of the factors in favor of this choice. The version of the data used in this test has been preprocessed for fast loading into memory as part of the service initialization, so that all queries can be handled without the need for any disk operations.

The actual application code on both client and server side is kept as close as possible across implementations. The only differences are in the actual Java classes used for the earthquake information. Most of the SOAP frameworks (with JibxSoap the only exception) require the use of generated classes for representing data to be communicated between the client and the server. In order to avoid added overhead in converting between formats, I modified the client and server code for each implementation to work with the preferred format for that implementation.

The test clients all use common code to generate a pseudo-random sequence of queries based on command line arguments, sending each query to the corresponding server, which then responds with the sets of matching quakes. Since the generation code is the same for each client, the actual sequence of queries generated by a particular set of arguments will be the same, and since the servers use identical earthquake databases the results returned by each query will also be the same. The test clients by default list query and response information to the console, allowing easy comparisons between versions to verify that everything is working properly.

If you want to experiment with the tests yourself, you'll find download links at the bottoms of the pages discussing the implementation(s) using each framework (JAX-RPC RI, Apache Axis, JibxSoap, and RMI - the RMI version is not a web service, but is included to give a baseline for the performance comparisons). You can also see the doc/lit format for a request and response, and, of course, the actual performance test results. All these pages are also linked in the navigation bar at the left of each page.