OK. I’m not a GeoHipster. I’m a GIS Worker. This is what I do.

But worry not, GIS worker: Spatial might no longer be special, but projections, datums, and legacy file formats will continue to be very, very special.
      – Brian Timoney, MapBrief

Okay… GIS is not sexy like your iPhone locator app.  But it will always be special.  In the same way that your plumbing and electrical and HVAC systems are special.  Next time they stop working, ask yourself just how special they are.  Next time your iPhone map is missing Main Street, ask yourself how special GIS is.

Consumer and business geo apps and services keep evolving – cutting edge today and obsolete tomorrow.  BigData today – legacy file format tomorrow.  But projections, datums, and georeferencing will always be there to provide ground truth. 

Esri and the ArcGIS platform have been around for 30 years, defining and documenting GIS work.  So when we needed to understand what GIS workers are doing across a variety of industries, we looked at the list of tools in ArcGIS Toolbox as a compendium of GIS workflow and process skills.  

We needed the ArcToolbox list itself as a data set — not a PDF poster or searchable web site.  I asked Esri for an ArcToolbox list as Excel table or other word processing format, but they could not provide.  So my friend at Esri, Nick Toscano, and I compiled the table ourselves.   That’s almost 900 “GIS tools” in an Excel data table.  Easy to sort, filter, annotate, categorize in ways that meet our needs.  

Here it is.

Some of the tools are pretty arcane.  But most are plain markers for the nuts-and-bolts work of GIS professionals in towns, cities, and counties who have to deal with projections, datums, and legacy file formats every day.  Not the next wave or next-big-thing.  Just the actual data and task at hand.

How will we use this?

  • Survey and assess the experience of our GIS workers.
  • Plan GIS skills training and investment.
  • Design new workflows to make common tasks more efficient.
  • As a quicker reference for building geoprocessing services. 

I need my proprietary GIS for this hard-core geospatial analysis, right?

Bill Dollins blogged recently that he doesn’t see much difference in capability between open source and proprietary geospatial tools.   

I think Web/GIS developers would agree.  But I’m not so sure that geospatial analysts would agree.

I agree, if we’re talking about building and using map apps — displaying points, lines, and polygons, and routine map functions like routing, thematic mapping, and interactive display of the map and underlying data.

But what about hard-core geospatial analysis?  Case at hand:

I’m working on an emergency management application that estimates the age-group populations that may be affected by an emergency event.   The only population and demographics data I have is census block groups.  I create a buffer around the event, then clip the census block groups that intersect the buffer.

The problem is that for many of the included block groups, only a small portion lie within the event zone.  So, only a fraction of the population in each of those groups should be included.

I was ready to write the function for my Python script that would calculate the percentage of the area of  each block group that got clipped.  If only 15% of the area got clipped, I would grab only 15% of the population counts for that block group to add to my population total.

Then I saw Esri’s announcement that ArcGIS 10.1 now includes an has areal interpolation tool.

Seems like the perfect fit for what I need to do.  I can create a grid of 100 x100 meter polygons and re-assign portions of the population counts to these much smaller and regularly shaped grid polygons.  And save this data layer for use any time a need a more precise population estimate.

Can any open source geospatial tools do this?  I admit that I don’t know.

I think it would be good to do something like what Tobin Bradley did to evaluate the importance of different elements of the Google Maps API.  Maybe do the same with Esri’s ArcToolbox.  How much of this tool set is present in open source geospatial software?  How important are the missing tools?

Has anyone besides Esri done an assessment like this?

Metadata Lite: Publishing metadata cheap (or not at all) through ArcGIS Server

You can use ArcGIS Desktop to author metadata for data layers in your geodatabase, and to publish your data as web services on ArcGIS Server.  But you can’t publish your geospatial metadata with those web services.

Esri will say, Not true.  You can publish your metadata to the Web through Esri’s free Geoportal Server.  Then point your metadata on the Geoportal Server to your map services on your ArcGIS Server, like this:
Right.  But that means you have to do the same task twice — send your metadata to your geodatabase and send it again to your geoportal.  Besides maintaining two applications servers for this purpose.
This redundancy sits atop the already cumbersome and expensive process of authoring, publishing, and maintaining FGDC/ISO-compliant geospatial metadata.
Are there less expensive options?  Even Esri’s FGDC/ISO-compliant metadata guru seems to think so.  Here is a slide from Marten Hogeweg’s geoportal workshop at the 2011 Dev Summit:
(The title/question is mine, not Marten’s.)

Marten was talking about different metadata for different audiences.  The public does not want or need FGDC/ISO compliant metadata.  So, how can we meet the public demand, as well as the needs of our professional partners in the GIS community?  I suggest a middle path – Metadata Lite – that’s already suggested in Marten’s diagram.
Yes, “Verbose Metadata is Desired” for the GIS Specialist Community.  But today more GIS professionals are concerned with pushing data out to the public.  Knowing that, it seems to me that Esri (unconsciously?) moved away from fully-compliant metadata  when they “forgot” to support metadata publishing through ArcGIS Server 9.3 and 10.  And now we see mere “tagging” promoted in Marten’s metadata/geoportal presentations.    
Take a look at ArcGIS.com.  At the Esri MUG last week, Clint Brown told us that millions of maps are added to ArcGIS.com each month.  But where is the geospatial metadata that enables search in a geoportal?  There is none at ArcGIS.com.  You can’t search through others’ FGDC/ISO-compliant metadata, and you can’t publish your own there.  
There is only Metadata Lite — tagging to give a minor boost to the ArcGIS.com search function:

If you want to publish descriptive info about your map data, there are a couple of free/low-cost options.  First, you can publish your data in map apps at ArcGIS.com.  (See my other posts about using ArcGIS.com for your organization’s geodata portal.)  Then simply author the descriptive information in the details page.  San Mateo County GIS has done it this way, using a text template to ensure that all the legal info is included, like in this example:

Even better, I think, is to publish your geospatial metadata through your ArcGIS Server REST catalog.  When it’s in the REST catalog, it doesn’t intrude on the public attention span, yet GIS professionals can find it when they need it.

This is what Metadata Lite looks like from San Mateo County, published through ArcGIS Server to the county’s REST catalog:

But 90 percent of the time, if you open up a REST service page like this, all the fields are blank, except a few that are machine-generated,  like Extent and Spatial Reference.  Why can’t we simply author metadata that shows up in the REST catalog?  Because …

You need a wiring diagram.  Because some of the REST catalog fields are populated from the ArcMap Document Properties tab.  Others from the ArcMap Data Frame Properties tab.  Others from the ArcGIS Server Manager Properties tab.  Like this:
Besides that, the field names are different for the same information found in the source (e.g. ArcMap) and the destination (REST page).  Like this:
So, here are a couple of wiring diagrams to help you populate the REST catalog fields:
2.  Metadata Lite live map service – hosted at University of Maryland, shows where the REST catalog fields values originated.
(From a presentation I gave at the 2011 Esri MUG)