On the hunt for some more benchmarks ..

Sourcing independant benchmarks or comparisons is always a difficult exercise. To continue on my recent raster benchmark quest, here are some other similar raster performance studies i’ve found round the interweb. If you know of any others, please throw them in the comments!

ESRI Image Server vs LizardTech Express Server

http://geoinfo1.lib.uidaho.edu/loadtest/ (June 2009)

Not much background info on the test setup in particular the input data … but the Pylot results are available and when compared produce a clear winner in LizardTech at all tests. The following are the throughput achieved at 50 user load. The average response times showed Express Server with ~0.5 sec average compared with ~2.5 seconds with Image Server.


Pursuit of the Perfect Digital Ortho File Format

http://www.igic.org/training/pres/conf/2009/perfectortho.pdf (February 2009)

Although again lacking specific reasoning on how they ran the tests., this one is more centered on desktop performance reading a variety of raster formats. There’s a couple of flaws with the support table, as ECW is in fact supported through Microsoft Office and likewise ESRI Image Server can also read the format after purchasing the required license from ERDAS. I would have liked Larry to list the resulting file sizes but his general consensus of running with 2 formats … Geotiff + JPEG compressed with ECW seems like a common conclusion. People seem to forget however that as soon as you introduce enterprise image serving capabilities, the output formats caters for the common request to be able to open in MS Paint or MS Word. I would love to see someone try to open a 700mb JPEG Compressed GeoTIFF in Paint without waiting a very, very long time :)

Overall though, ECW achieved very good performance in many of the tests with Geotiff winning some large scale tests, presumably when it was requesting 1 reasonably sized geotiff tile


Benchmarks updated

Small update,

  1. Added a new ECW 256px tile test
  2. Added a new ECW reprojection test
  3. Added some more info on my mapguide config problems

I’m still keen for someone to verify my numbers. It would be nice to know whether I’m alone in the universe or not even if the setup is different ..

Raster Image Serving Benchmarks

I am pleased to announce my own performance metrics continuuing on from the FOSS4G 2009 WMS Raster tests. So, whats new?

  1. Tests extended from just TIF and ECW to TIF, ECW, JP2, MrSID, TIF Tiled, TIF Internal Pyramid, TIF External Pyramid
  2. Platform changed from RHEL to Windows Server 2008 x64
  3. Increased the threads from 1,10,20,40 to 1,10,20,40,80,150.
  4. Hardware increased from a 4 core to 8 core server
  5. Analysed throughput not only by input format, but by output WMS Format as well. 8bit PNG vs 24 bit PNG vs JPEG vs GIF
  6. Added ERDAS Apollo to the mix along with Mapserver and Geoserver (Deegree and Mapguide was with very limited success … I’ll add these later)

I will endeavour to update the page with new results as there is no question further tuning could be applied. I am not going to comment specifically on the results, as I want to leave the interpretation up to you.


Look out for more performance tests in the coming days..


PostgreSQL driving you nuts?

I’ve long had issues with PostgreSQL 8.3 and now 8.4 refusing to start on my laptop which is really a pain for live demonstrations of ERDAS Apollo. I’m not sure if anyone else has similar problems, but I believe (or rather have a hunch) that it is something to do with “ungraceful” shutdowns of the db when you may lose power or consistently go to sleep. Its only ever occured on my laptop, so luckily postgres on a server is still stable as hell.

If you are getting something like the following in postgresql.log on Windows or are just tearing your hair out wondering why the service that was working before is no longer starting …

2009-12-04 09:05:37 WSTLOG:  database system is ready to accept connections
FATAL:  could not reattach to shared memory (key=240, addr=02690000): 487
2009-12-04 09:05:38 WSTLOG:  autovacuum launcher started
2009-12-04 09:05:38 WSTLOG:  background writer process (PID 9712) exited with exit code 1
2009-12-04 09:05:38 WSTLOG:  terminating any other active server processes
2009-12-04 09:05:38 WSTLOG:  all server processes terminated; reinitializing
2009-12-04 09:05:48 WSTFATAL:  pre-existing shared memory block is still in use
2009-12-04 09:05:48 WSTHINT:  Check if there are any old server processes still running, and terminate them.

To reliably fix things for me anyway,

  1. Delete postmaster.opts or postmaster.pid (if they exist) in your postgres data dir
  2. Kill the pg_ctl.exe process (if running)
  3. Start the postgres service



Fun and games

Ok. I am kinda pre-empting the fallout of the FOSS4G Benchmarking list here. But for everyone interested, this is a cross post from the Tilecache list where Bruce Foster has, I believe, drawn a very long bow.

This offer also extends to anyone else at Foss4g who may draw similar conclusions for ERDAS not participating in the event. I’m easy to spot in a crowd :)

I have never been fond of this mythical Opensource vs Commercial divide in the spatial industry; even when I was exclusively using opensource software. Mapserver is no different from a commercially competitive point of view than ESRI products.


Lets set the record straight here. We were invited, but declined. There is nothing more to it.

I will be at FOSS4G on Thursday and Friday on the CCIP booth. Swing by and I will sit down and show you whatever you seem to think we are hiding?

The FOSS4G benchmarks are a great starting point and I would like to think we can be involved next year. I really hope customers do much more by way of due diligence in selecting enterprise software than just some benchmark results.

Lets not forget that we are not the only ones not competing in the event. Mapguide, Deegree, Mapnik, Cubewerx, Mapinfo, Oracle, Cadcorp, Manifold  … the list goes on (notice, there is no delineation between OS vs Commercial here). There is no conspiracy here mate and I’d be happy to show you.

Look forward to seeing you next week.


Chris Tweedie
ERDAS Australia

Now, where did that unicorn go and my car with that rocket booster …