10 utils to make opensource gis that little bit easier

In the same vain as my other ArcIMS thread, here are my top 10 utilities (ok so its not really ten) for making my life with OGC apps that little bit easier. If i have missed any which you recommend, please let me know. I have tried to cut the list down to what i think are useful, as there are unfortunately a lot of applications which are not worth the trouble or are incredibly outdated (sorry freegis.org, but the list needs culling guys).

My first and most important tip when getting into opensource GIS …. do not give up. Yes, the documentation is poor most of the time, finding information or help is often frustrating but you will get there. Just the other day i found a solution to a problem i have been having in mapserver in a 2 year old newsgroup post. The information is there, you just gotta dig a little. Carn devs, lets move to a more web based forum instead of increpid old mailinglist applications please.

Desktop Clients
GAIA A free .NET based application capable of consuming WFS / WMS / GML (variety of versions). Suffers from some apparent memory leaks, but otherwise is a great app to debug services.
CADCORP SIS An excellent WFS / WMS / WMC / GML desktop client with all the bells and whistles you could want, short of doing it yourself of course :)

Web Clients
Mapbuilder
Mapbender A PHP/JS frontend quite similar to mapbuilder, but in my eyes, a lot easier to play with / setup. Recommended for customisability and quick setup
Chameleon A highly customizable and adaptable environment for deploying and managing Web mapping applications
Ka-map A frontend that utilises XMLHTTPRequest and mapservers tile generation technology to create a responsive interface much like other AJAX apps such as Google maps or Yahoo maps

Server Applications
Mapserver A CGI (compiled c) application that leverages off existing opensource utilities such as GDAL / OGR and the PROJ4 library. Very quick, well supported.
Geoserver A J2EE application that supports WFS / WFS-T and WMS. Leverages off another opensource package, GeoTools. Excellent web-based configuration frontend, although suffers from lack of WMS features such as raster and SLD support. Highly recommended for WFS and Transactional WFS (WFS-T), WMS support gaining momentum at every release
Deegree Another Java based app, with support for nearly all the major OGC standards. I have only had the pleasure of using the deegree WMS server, but their iGeoPortal product definately looks interesting.
PostGIS Adds support for geographic objects to PostgreSQL. Its a free, opensource alternative to Oracle Spatial / ESRI ArcSDE. Unfortunately i have had little background in using PostGIS, but it comes highly recommended if you do not have the $$ or would like an alternative spatial rdbms, get this.

Server Application Utilities
CITE A test suite is designed to exercise WFS and WMS servers against the requirements in the technical specifications. Its quite funny (or sad) how difficult it is to find the developers CITE results. They might say they are compliant, but trying to find the evidence is another matter.
Maplab Another product in the maptools.org line, maplab assists in the creation, editing and deployment of mapserver mapfiles. The web frontend for creating map files is great for newbies, or for checking changes as you go. The deployment app. (using ROSA) isnt too bad for a quick job either.
Open Geospatial Where would my list be if i didnt list the consortium that gave birth to these standards. Word of warning, the server applications whilst maybe being OGC compliant, often have their own methods of implementing them and or, can also include a lot of extra features. Remember, there is more than one way to skin a cat :)

Conversion utilities
ESRI ArcMap -> UMN Mapserver Similar to GIX, but can convert MXD to mapserver map files instead. Again, be wary. Converting can sometimes be time saving but never assume its going to work straight out the box. Some editing will always be required.
ESRI Arcview -> UMN Mapserver GIX is a great tool for converting arcview projects into a variety of formats (including .map files). Unfortunately the project is not updated as frequently as the mapserver application so it does convert redundant or incorrect syntax. (Still quicker to edit, than to create it from scratch). It also creates referenced reference images and retains any symbology or class definitions. Great tool to have in the belt

… and most important of all, a decent text editor. I recommend ultraedit or textpad. Do not skimp on this, as this is your holy grail. A poor text editor will surely give you headaches in the road ahead :)

If you are having any troubles with anything OGC please get in touch. If i cant solve it, i will hopefully be able to point you in the right direction.

As for those who are interested in the status of Mapserver vs. Geoserver performance benchmarks, they are proceeding nicely. Hopefully we will have some publically available documents for everyone to have a look at soon. The results are not nearly as close as i was expecting ..

Useful ArcIMS tools

In my travels there have been a few nifty tools i have picked up along the way to make my life in ArcIMS a little bit easier.

ArcXML DTD. Is extremely useful for validating those painful AXL files. I dont know how often I have left a field open, or missed a quotation mark. This will validate the structure and report any errors found. Very useful for those doing a lot of editing of AXL’s by hand. I recommend XMLStar to do the DTD and general well-formedness checking, but there are lots of others out there.

Shapefile UPPERCASE.exe. Arcims has an uncanny habit of dropping or simply not adding services with shapefile layers having a combination of lower and upper case field names. This program will open the DBF file and automatically convert them to all uppercase. Sure beats doing it manually.

PHP Logwatch. This is a multipurpose PHP script that monitors any number of log files running on your system and outputs that last X lines (eg. 50). It is particularly useful for monitoring the 400 arcims log file locations and any other system log of any importance (eg. IIS/Apache logs). Oh, and it also has error syntax highlighting. The script detects the latest log file in a given directory, making it even more relevant for ArcIMS where it creates a new log at every restart (not neccessarily each day). The script can be monitored from a remote location with only the PHP script needing direct access to the log files.

Webtail. Webtail is similar to PHP Logwatch in that it tails log files, however the main difference is that Webtail is able to tail any log file visible on the web so it does not need to reside on the same machine. Extremely useful if your sysadmins have given you read access only to a web readable directory, but you dont actually have access to the machine.
Example: Read access only to a website with the logs (eg. http://log.com/arcimslogs/). All we need to do is call webtail via “java -jar WebTail.jar http://log.com/arcimslogs/AppServer_1123623189000_0.log 10″ and at 10 second intervals, it will look for any updates to the file and output the difference. It only needs to retrieve the entire file once, and so lessening the impact of downloading a large log file every 10 seconds. HIGHLY RECOMMENDED

MXD to AXL converter. If you search at ESRI you’ll hear that there is no way to convert between the two, but there is, and there has been since 2003. Jeroen Ticheler’s converter on Arcscripts has been helping me for well over the year doing the arduous task of converting MXD’s to AXL’s. It isnt going to solve all our problems (as you will still need to manually do some of it), but it sure as hell speeds up the process.

Arcexplorer installed on the same ArcIMS machine. Nothing annoys me more than the conflicting installs of ArcIMS / ArcMap and ArcExplorer. “Sorry, we detected X on your system, install cannot proceed.” There is thank god, an easy way to get round this and this is to just use ArcExplorer version 4 (unfortunately i dont have a link handy). Version 9 will throw the error, but v4 runs happily on the same machine. This workaround has saved me 20 times over when we have needed to debug our production ArcIMS server and this was the easiest way to grab a service. Very useful when there is no other access to the machine.

Thats all that comes to mind at the moment, but im sure theres a heap more.

Please post a comment if you have something else to recommend. Anything to make administering the nightmare that is ArcIMS, a little less painful :)

Its google … it must be right.

The following article has been superceded and the issue has been resolved. There is no issue (as far as i am aware) with the state of Google Earth’s data.
After comparing some common overlays from both Mapserver and Arcims (using the PHP code i posted earlier), it was obvious something was up,

Projection descrepencies

  • either my data being projected was wrong
  • or googles base data was out

Upon closer inspection of the cadastre and road overlays, the general accuracy was roughly 35m to the south east.

Projection descrepencies

Makes you wonder doesnt it (yoohoo, metadata would be lovely guys). I have double checked the projection using both Arcims and Mapserver output and it appears at the same level. A collegue in Darwin has had no problems overlaying his own data … has anyone else noticed this? Or is the Perth Digitalglobe data just poorly processed?

Of course, it could just be me :)

Nevertheless, the sooner people stop asking me ‘HAVE YOU SEEN GOOGLE EARTH@!#!$?’, my life would be just *that* much better.

Im sure many other GIS’ers agree that it has limited use or flexibility in the marketplace and goes against pretty much all the open standards that have been put in place over the years.

Yes, you can fly through the grand canyon
Yes, you can “view” anything on the globe
Yes, i now deem myself as GIS professional extraordinare because i can the read the API and draw my name using JS. HEHE
No, i dont know how old, or what accuracy this image is .. who cares! I can see Pam Andersons house!
What do you mean projection? I just use what google gives me, google is always right

If you sense angst, probably because their is. It has its place guys (ie. visualisation), but the sooner everyone realises this the happier i will be :)

Clarification on adding WMS to Google Earth

Ive noticed a few other blogs mention the need for WMS server operators to run my PHP code i posted earlier. This is untrue, so to clarify … read on.

1. If you havent already done so, grab the code here

2. This code will require at least 2 small changes for it to work for you … first off, go to lines 26 to 35. This block contains 2 arrays, $WMS (the full getmap WMS request to the server) and $name (a pretty server/layer descriptor).

3. Each array entry corresponds to a new layer inside google earth. Obviously, depending on how you want to display your map, you may wish to only have 1 layer. I will assume this is what most will like to do, so remove all the other array entries so you are only left with a single layer. eg

[0] = "Cadastre";  $WMS[0] = "http://link.to/your/wms?ServiceName=earth&VERSION=1.1.0&REQUEST=GetMap&SRS=EPSG:4326&WIDTH=".$width."&HEIGHT=".$height."&LAYERS=Land Parcels&STYLES=&BGCOLOR=0xCEFFFF&FORMAT=image/gif&TRANSPARENT=FALSE&WRAPDATELINE=TRUE";  ?>

4. Now, i will assume you understand how to submit a WMS getmap request. Many of the public WMS servers already have documentation on how to do this. The syntax should be similar to the above, but REMEMBER to set the EPSG code to 4326 (WGS84) otherwise your data will not fit googles projection. If the WMS server has the option, its always a good idea to set the background to transparent also.

5. Now assuming you have added your first WMS entry into the PHP code, all you have to do is modify the IP range which you are allowing to access your WMS-KML reflector script. Go to line 46

if ($match[1] != '151.130' && $match[1] != '202.72' && $match[1] != '115.205') {  ?>

Edit the partial IP addresses to reflect that of your own organisation / home computer. I have only done the pregmatch on the first 2 numbers, but feel free to modify it on line 42 if you wish to do a full ip comparison.

6. Ok, time to try adding it to google earth ..

7. Load it up and click Add -> Network Link. Enter an appropriate title for the entry and in the ‘location’ textbox, enter the full URL path to your PHP code eg. ‘http://url/kml.php’. In the dropdown for image request, change it to ‘When camera stops’ and ensure the value is set to 4 seconds.

8. Zoom to an area of interest (one which is covered by your WMS) and stop the camera from rotating. Wait for 4 seconds and a red ‘Loading..’ image should be shown over your extents. This is the standard image google earth places when it is attempting to retrieve the network link (ie. your WMS image). If all goes well, you should have your image overlaid very nicely. Otherwise, if you recieve a nasty error popup check that your WMS server request was entered correctly.

Thats pretty much all there is to it. You dont need to run your own WMS server, you simply need to point to an available public server. Of course, running your own you get added benefits of being able to completely customise the look and feel … but if you’re feeling really lucky, try playing around with some SLD documents :)

As always, any Q’s flick me an email or post a comment. Hope i have cleared any confused.

DMSolutions done it again

Paul Spencer at DMSolutions has posted their latest demonstration of KA-Map which uses AJAX-esque technology to transform mapserver output into a tiled format similar to maps@google but with a lot more features.

Check out the demo site here. It is without a doubt the best demo i have seen showcasing Ka-Map and mapserver’s new WMS support for a time attribute. Excellent stuff … just make sure you press play !

ps. In a somewhat related line, anyone know of any projects working on an opensource routing solution? Let me know