OK. I’m not a GeoHipster. I’m a GIS Worker. This is what I do.

But worry not, GIS worker: Spatial might no longer be special, but projections, datums, and legacy file formats will continue to be very, very special.
      – Brian Timoney, MapBrief

Okay… GIS is not sexy like your iPhone locator app.  But it will always be special.  In the same way that your plumbing and electrical and HVAC systems are special.  Next time they stop working, ask yourself just how special they are.  Next time your iPhone map is missing Main Street, ask yourself how special GIS is.

Consumer and business geo apps and services keep evolving – cutting edge today and obsolete tomorrow.  BigData today – legacy file format tomorrow.  But projections, datums, and georeferencing will always be there to provide ground truth. 

Esri and the ArcGIS platform have been around for 30 years, defining and documenting GIS work.  So when we needed to understand what GIS workers are doing across a variety of industries, we looked at the list of tools in ArcGIS Toolbox as a compendium of GIS workflow and process skills.  

We needed the ArcToolbox list itself as a data set — not a PDF poster or searchable web site.  I asked Esri for an ArcToolbox list as Excel table or other word processing format, but they could not provide.  So my friend at Esri, Nick Toscano, and I compiled the table ourselves.   That’s almost 900 “GIS tools” in an Excel data table.  Easy to sort, filter, annotate, categorize in ways that meet our needs.  

Here it is.

Some of the tools are pretty arcane.  But most are plain markers for the nuts-and-bolts work of GIS professionals in towns, cities, and counties who have to deal with projections, datums, and legacy file formats every day.  Not the next wave or next-big-thing.  Just the actual data and task at hand.

How will we use this?

  • Survey and assess the experience of our GIS workers.
  • Plan GIS skills training and investment.
  • Design new workflows to make common tasks more efficient.
  • As a quicker reference for building geoprocessing services. 


Explain it to Your Boss: The Map is There but Your Data is Still Here

Some business managers don’t want “our” data sitting on “their” map server.   So they’re reluctant to share sensitive business data in maps that are authored and published at public map portals.

They may think that when we publish our map data in the “cloud”, we give up ownership and the security oversight of our data.   That’s not necessarily so.  You can explain it to your boss like this, using an example from the ArcGIS Online mapping portal —  

You can sit behind your firewall and publish your sensitive data in a web map.  Here’s what it looks like:

Your boss might not understand that although the map is at ArcGIS Online, your map data does not have to be.  Web page and map components can come from many public web servers, while the map data itself remains secure.  The map data comes from your secure GIS server:

It gets delivered to the web map only if the browser user has been granted access to our GIS server; that is, if the user’s computer is also sitting behind the same firewall with the GIS server, or if the user has login access to your GIS server from outside.   

If no access is granted to your GIS server, the points will not appear on the web map.   Only the background (basemap) will appear.

The business data (map points) are all we need to own and control.  At the same time, we can build on free, publicly-available code and data platforms for map programming and publishing.   For example, the background map could come from public servers at Esri, Google, or OSM:

We don’t have to build the map widgets ourselves.  In this example, they’re sent to the browser from Esri’s Javascript API server.  But they could have come from Google Maps or another server:

The ”Dojo-Javascript”  layout of the web page – header, banner panel, navigation panel – could be from code that sits on the Google or Yandex CDN servers:

Code from other Web servers could also be embedded in the map or the web page.  For example, we might overlay our data layer with weather data coming from NOAA or tweets from Twitter.  So, in an example like this, we may rely on 4 or more different public web servers, all supporting the publication and sharing of our secure geospatial data.

So, your boss and trusted colleagues can sit behind our firewall and benefit from web mapping in the public cloud.  Their web browser reaches out to the public Internet to bring in elements of the map and page that surround your map data.  But the map data itself stays inside a secure channel from our GIS server to the web browser and map.

I know you understand this.  But your boss might not.

::: ——- :::

Not so simple?

My scenario above won’t address the concerns of all business managers.  For example, I worked with a county health department whose management was reluctant to serve up water quality monitoring maps to the public unless they appeared on the county’s web site, rather than at ArcGIS.com.  It was an ownership and branding issue. 

Esri has tried to address this concern with its Public MapsGallery template.  The Gallery is a clever approach that lets an organization author, publish, and host maps at ArcGIS Online, but deliver them in web pages that come from the organization’s own web server, with the organization’s own style and labeling.  No “ArcGIS Online” labels anywhere.  Too clever?  Your boss will have to decide.

Valid security issues — not just branding — are raised if you actually upload some or all of your data to the public map portal.  Or if you go to the public portal to author and publish information related to the map, such as authorship, map description, and geospatial metadata.  In that case, you need to understand the hosting services security policy and implementation.  You can read about Esri’s for ArcGIS Online, here .

Server geoprocessing for 911/CAD. Got some?

I’m working with the metro police dept. to build and automate geoprocesses that support 911 and computer-aided dispatch (CAD).  The goal is simple:  Within minutes of a 911 call, we want to provide an “investigative package” of map layers, tables, and fact sheets for the area surrounding the location of the event.  Info like this:

  • 911 calls of the same or different types in past 24 hours
  • Active warrants and prison releases
  • Weapons seized and ownership information
  • Population and demographics
  • Schools and care facilities and their populations
  • Land use and cover, building footprints
  • Police dept. assets, fixed and mobile

I was fortunate to get on the phone with Esri’s Public Safety Team to discuss this.  They’re a great bunch – knowledgeable and helpful.  I asked the Esri team:

Has anyone in the ArcGIS community already built  911/CAD geoprocessing models they can share?

I had already searched Esri’s Public Safety Resource Center, the Public Safety Forum, ArcGIS Online, and AGO’s Public Safety Group.  And googled around, of course.  I didn’t find anything.

The Esri team acknowledged that a published set of standard tools and scripts to support public safety generally, and 911/CAD in particular, is still needed.  They’re interested in working with us over the next few months to help get that started. 

Who has customized ArcGIS Server to support 911 or other public safety operations?

It looks like there’s not much out there yet. But Esri could point to one example:

New York City ‘s Office of Emergency Management is using ArcGIS Server to automatically generate an “incident response packet” about each 911 incident scene. The packet includes maps of

  • area
  • aerial 
  • neighborhood maps

and reports of

  • administrative boundary
  • nearest critical facilities
  • demographic
  • land use
That’s close to our “investigative package”.

Application development at NYC OEM called for database design, ArcGIS Server 9.3, and programming with ArcObjects, ASP.NET, C#, and Ajax.  (Does that mean Esri Javascript API and Dojo?  Not sure.)  I’d like to see the geoprocessing scripts.

Is anyone using ArcGIS Server with Sharepoint for 911 or other public safety?

The obvious answer is that Esri has partnered with Microsoft to integrate ArcGIS Server with Sharepoint into what’s they call the Fusion Core Solution.  Surprisingly, if you google it, you won’t find a lot out there except Powerpoint by Esri and MS. But thanks to stoptimeculp at YouTube, we have a simple video demo.  

After the YouTube demo spends time on “punching in” and tracking personnel hours, it finally shows what you would expect for web mapping and Sharepoint:  Select layers to add to a map, and map the new incidents that you intake using web forms. and then looked at them together.  So far, I’ve seen no geospatial analysis — automated or otherwise — to provide additional location-based intelligence like what’s listed above.

By the way … Is there a national or standard data model for 911/CAD ?

I was thinking of standard categories for 911 events, and standard or recommended fields and formatting for 911 call processing. The Esri team discussed generic industry-level information exchange models. But we couldn’t identify a standard specifically for 911.

Got ArcGIS Server geoprocessing models for 911/CAD you want to discuss and share?

Metadata Lite: Publishing metadata cheap (or not at all) through ArcGIS Server

You can use ArcGIS Desktop to author metadata for data layers in your geodatabase, and to publish your data as web services on ArcGIS Server.  But you can’t publish your geospatial metadata with those web services.

Esri will say, Not true.  You can publish your metadata to the Web through Esri’s free Geoportal Server.  Then point your metadata on the Geoportal Server to your map services on your ArcGIS Server, like this:
Right.  But that means you have to do the same task twice — send your metadata to your geodatabase and send it again to your geoportal.  Besides maintaining two applications servers for this purpose.
This redundancy sits atop the already cumbersome and expensive process of authoring, publishing, and maintaining FGDC/ISO-compliant geospatial metadata.
Are there less expensive options?  Even Esri’s FGDC/ISO-compliant metadata guru seems to think so.  Here is a slide from Marten Hogeweg’s geoportal workshop at the 2011 Dev Summit:
(The title/question is mine, not Marten’s.)

Marten was talking about different metadata for different audiences.  The public does not want or need FGDC/ISO compliant metadata.  So, how can we meet the public demand, as well as the needs of our professional partners in the GIS community?  I suggest a middle path – Metadata Lite – that’s already suggested in Marten’s diagram.
Yes, “Verbose Metadata is Desired” for the GIS Specialist Community.  But today more GIS professionals are concerned with pushing data out to the public.  Knowing that, it seems to me that Esri (unconsciously?) moved away from fully-compliant metadata  when they “forgot” to support metadata publishing through ArcGIS Server 9.3 and 10.  And now we see mere “tagging” promoted in Marten’s metadata/geoportal presentations.    
Take a look at ArcGIS.com.  At the Esri MUG last week, Clint Brown told us that millions of maps are added to ArcGIS.com each month.  But where is the geospatial metadata that enables search in a geoportal?  There is none at ArcGIS.com.  You can’t search through others’ FGDC/ISO-compliant metadata, and you can’t publish your own there.  
There is only Metadata Lite — tagging to give a minor boost to the ArcGIS.com search function:

If you want to publish descriptive info about your map data, there are a couple of free/low-cost options.  First, you can publish your data in map apps at ArcGIS.com.  (See my other posts about using ArcGIS.com for your organization’s geodata portal.)  Then simply author the descriptive information in the details page.  San Mateo County GIS has done it this way, using a text template to ensure that all the legal info is included, like in this example:

Even better, I think, is to publish your geospatial metadata through your ArcGIS Server REST catalog.  When it’s in the REST catalog, it doesn’t intrude on the public attention span, yet GIS professionals can find it when they need it.

This is what Metadata Lite looks like from San Mateo County, published through ArcGIS Server to the county’s REST catalog:

But 90 percent of the time, if you open up a REST service page like this, all the fields are blank, except a few that are machine-generated,  like Extent and Spatial Reference.  Why can’t we simply author metadata that shows up in the REST catalog?  Because …

You need a wiring diagram.  Because some of the REST catalog fields are populated from the ArcMap Document Properties tab.  Others from the ArcMap Data Frame Properties tab.  Others from the ArcGIS Server Manager Properties tab.  Like this:
Besides that, the field names are different for the same information found in the source (e.g. ArcMap) and the destination (REST page).  Like this:
So, here are a couple of wiring diagrams to help you populate the REST catalog fields:
2.  Metadata Lite live map service – hosted at University of Maryland, shows where the REST catalog fields values originated.
(From a presentation I gave at the 2011 Esri MUG)

ISO19115 to REST – You can’t go there with metadata

It took me awhile to figure out that it’s not easy to author geospatial metadata that will show up in your ArcGIS Server REST catalog for a map service.
The Excel table “Geospatial Metadata Authoring for ArcGIS Server REST and ArcGIS Online Display Throughput”  is a plumbing diagram.  It shows the information flow (and barriers) from ISO19115 metadata authoring in ArcGIS Desktop to display in the ArcGIS Server REST catalog and ArcGIS Online.
I wanted to understand how to author metadata so that it shows up in the ArcGIS Server REST directory and in map descriptions at ArcGIS Online.  And to find opportunities to author metadata once and use it in many places.  I used the ArcGIS desktop metadata editor (ISO19115 NAP format) to author metadata for two map layers (feature classes) and one map document (.mxd).  I published the map service then examined the REST directory.
In the table, metadata input columns are on the left:
§  The ISO19115 NAP metadata fields required by San Mateo County GIS
§  ArcMap Map Document (.mxd), Group Layer, and Layer Properties fields
§  ArcGIS Server Manager configuration fields
§  ArcGIS Online Map Details fields

Web outputs are on the right:
§  ArcGIS Server REST Directory
§  ArcGIS Online Web Map Details page

I was surprised to find that:
  •  For Layers in an ArcMap Document, no text from any field in the metadata editor appears in the ArcGIS Server REST directory for the resulting map service.
  • These REST directory fields are populated from these Layer Properties fields in ArcMap (not the metadata editor):
    • Description  (Description in mxd)
    • Copyright Text (Credits in mxd)
Tags added in the ArcMap Document Properties page appear as “Tags for Searching” in the metadata editor.
Confused yet?  If you want to document your map service, you need a metadata roadmap.

Why doesn’t Esri support metadata publishing through ArcGIS Server and ArcGIS Online?

Part of my task at San Mateo County GIS (Apr-Oct 2011) was to set up a process for publishing geospatial metadata along with the County’s map services.  We looked at Esri’s free Geoportal Server.  But we decided that was overkill for the County’s data sets.  And we were also having trouble with metadata authoring and publishing with ArcGIS v10.  We needed to publish metadata through ArcGIS Server and the REST API, or through ArcGIS Online where the public would access our map services.  But we couldn’t do either one.
Why doesn’t Esri support metadata publishing through ArcGIS Server and ArcGIS Online?
This is what I’ve concluded from my conversations with several Esri customer reps and product managers, including  Sud Menon:
Back in the late 90s the federal government wanted to save money by sharing data so it didn’t have to pay to collect the same data twice.  The Federal Geographic  Data Committee (FGDC) thought it was a good idea to develop a metadata standard and taxonomy to enable fast searching on tags, keywords, and technical data descriptions.   FGDC figured that vendors like Esri should support this with geosptial metadata creators and search capability.  Esri agreed, probably because the USG is their biggest customer. 
So Esri built the first geoportal that provided the search and share capability for Dept of Interior’s geodata.gov site, based on FGDC standard metadata.  Esri (and everyone else in big government) thought that government organizations could mandate that their GIS units would make the effort to document all their geodata in the FGDC metadata standard.   But it didn’t happen, because it’s too big an investment for most organizations.
When Esri developed ArcGIS Online, they opted not to support full FGDC metadata either.  And when they developed ArcGIS Server v9 and v10, they again gave it little attention.  Esri focused instead on simple key words and minimal tag information — less metadata for more audience – to keep the hurdle low for sharing geodata. 
Esri says that ArcGIS Server will support metadata publishing through the geodatabase in the future.   In Q&ADiscussion before the 2011 UC, Esri announced:
“ArcGIS Server 10.1 will automatically capture and store basic metadata about the GIS services you create and allow you to enhance metadata documentation with descriptions, summaries, tags and other information. Any client accessing these services as well as anyone using ArcGIS.com and the Portal for ArcGIS will be able to leverage this information….  you can create and update your GIS service metadata using the tools that are built-in to ArcGIS. This metadata will be available via a simple URL.”
Lots of organizations went to the trouble of creating structured metadata.  But San Mateo County has only a loosely-structured html-based metadata catalog.  The best approach for the County now seems to be a blend of FGDC-like metadata documentation for geodata that is complex, that must be accurate and precise, and/or is a component of a geodata model that is shared among GIS experts.  And to author and publish a more compact set of simpler (but still standardized) tags and keywords for most geodata shared with partners and the public.
Standard tags is what we used for “metadata lite” with the San Mateo County Geodata Catalog.

Metadata really does matter … to me, right now

Tonight I worked on assembling Chicago area map layers to inform a real estate analysis task.  I pulled together map services into Chicago Area Land Use Map.

 I found some usable map services by googling for ArcGIS Server REST catalogs like this:

url:rest AND url:arcgis AND chicago

I found interesting map layers from cmap.illinois.gov and Loyola Univ.  But guess what… not a single word of descriptive metadata was published with any of these map services.  The Loyola server is obviously used by students, so I can understand why there’s no documentation.  But the .gov GIS server is public-facing and, I would assume, authoritative.  But I can’t really tell.  Can’t tell anything about any of these services.

Maybe the map service authors couldn’t figure out how to get information into the REST catalog fields for these services.  It’s not straightforward when publishing to ArcGIS Server.