The Middleware Company has released a J2EE and .NET Performance case study, the latest study (an MDA productivity study was released a few weeks ago) based on their Application Server Baseline Spec. Except for the web services test, the two platforms came out mostly equal in performance.

How did TMC get to the results? Starting in February, TMC invited experts to define a spec that case studies would implement. TMC then published the spec and opened it for public review in May.

Last week, the Productivity Study Results were unveiled and now the Performance Study Results have been published.

TMC invited all of the vendors whose products are in the study to be involved in the tests. Some of them accepted. Some did not.

The study goes into a lot of detail on what was tuned and how it was done. This information is invaluable as it shows how to tune this application in J2EE (and .NET).

The following sections have the most relevant details related to tuning J2EE:

- Tuning the Java Virtual Machine
- Tuning the app server's runtime settings
- Tuning the application's deployment characteristics
- Experiences with EJB Container Managed Persistence 2.0

The study also explains why appservers and .NET can behave differently in the PetStore application scenario. For more details, see the sections: "J2EE appserver X much better than J2EE appserver Y - an anomaly?" and, "Technical theories on why app server X out performed app server Y"

The case study is divided into three testing areas:

Web Application Test: This tested performance hosting a typical Web application with steadily increasing user loads. This test used two different databases: Oracle 9i and Microsoft SQL Server 2000.

24 Hour Reliability Test: This tested the sustainable performance and reliability of platforms over a 24-hour period as transaction-heavy clients are run against the Web application under a constant, peak-throughout user load. This test used different databases for each code base based upon performance data of the application server. The J2EE code bases used Oracle 9i. The .NET code base used Microsoft SQL Server.

Web Services Test: This tested the performance of the application server hosting a Web service accessible over SOAP 1.1.

Results:

View the results of the study paper:
http://www.middleware-company.com/casestudy/tmc-performance-study.shtml

View "Why this study is different to the one in October 2002"
http://www.middleware-company.com/casestudy/comparing-oct-2002-and-jul-2003.pdf

View the Case Study home page:
http://www.middleware-company.com/casestudy

View the Application Server Platform Baseline Specification.
http://www.middleware-company.com/casestudy/specification.pdf

View interviews with TMC with:
- Salil Deshpande the baseline spec, the expert group, and the concept of 'case studies' as opposed to benchmarks.
- Will Edwards discussing the tech choices in the various tiers, comparing the TMC mPetstore to the JPetstore, and talking about why EJBs were used.

View coverage and complaints from the original PetStore:
http://www.middleware-company.com/casestudy/coverage.shtml