A Window on Your Public Geodatabase — Really Fast

At San Mateo County (Calif.) GIS, we adapted Esri’s Public Map Gallery template into a County Geodata Catalog.  The purpose of the Catalog was to get county geodatabase layers out to the public as quickly as possible.  (SMC GIS published a separate Public Map Gallery to share published maps authored for specific audiences.)


The Geodata Catalog put a lot of data into the hands of more experienced geodata users out in the public domain.  To quickly publish the Catalog, we took advantage of the county’s ArcGIS Server and some less well-known features at ArcGIS Online.


Here’s how county geodatabase layers went public in a matter of hours:


1.  Serve up some Metadata Lite – We authored map services each that contained one to a dozen geodatabase layers.  We used a standard template for “metadata lite” — descriptive information that appears in the map service REST catalog.  (See my post about deciding how many map layers to combine into one service.)




2.  Create an ArcGIS Online Group – Once the service were published on the county’s ArcGIS Server, we created a San Mateo County Public Geodata Catalog group at ArcGIS Online.




3.  “Register” Map Services – With the group in place, we rapidly added maps to the group by using the “Add Item” button

 
 
When you paste in the URL for your ArcGIS map servicie hit “tab”, the Title and Tags fields are auto-filled with the fielded content from your “metadata lite” in the REST catalog page for this service:
 
 
Shining a light on County geodata
At San Mateo County (Calif.) GIS last summer, we wanted to get as many of the county’s geodatabase layers as possible into the hands of the public and of developers of location-based services.  To  do this quickly and at very low cost, we combined the Esri Map Gallery template with some less-known functions at ArcGIS Online to create a San Mateo County Geodata Catalog.
Most of the county’s geodata had never been seen by the public.  Although many of the data sets were available through FTP or by email request , you had to be a GIS expert to find and understand the data description in the County’s online metadata catalog, then obtain, import, and view the data  in desktop GIS software.  The Geodata Gallery put each data layer into a simple web map for the firs time, so that the data was easy to visualize geographically, to evaluate for a given purpose, and to combine with other data.
Same Map Gallery but different
The San Mateo County Geodata Catalog looks like the Esri map gallery.  But there are key differences in its implementation and use.  First, we made sure that our geodatabase layers were documented in each ArcGIS map document properties fields that appear in the published ArcGIS Server REST catalog.  (See my post about publishing “Metadata Lite”.)  Once the map services were published, we “registered” each map service at ArcGIS Online.   This allowed us to rapidly produce dozens of web maps for all of the county’s map services literally in minutes. 
“Registering” map services – What you do  you don’t
Although the default published web maps are not 
it’s really designed for the County’s more GIS-savvy partners and stakeholders, rather than for the general public.
Authoring the Map Service
The Geodata Catalog points the user to all map services published by the County.  Each map service  contains one or more layers from the County geodatabase.  GIS server resources were limited, so we reduced the number of map services by combining geodatabase layers that are topically similar, or that contain polygons that don’t overlap each other within a single map service. 
Why be careful about overlapping polygons?   Because San Mateo County GIS has centered its web map publishing at ArcGIS Online.  AGO’s web map authoring tools allow layer manipulation (e.g. transparency, position above/below other layers) only for the entire map service, but not for individual layers in the map service.  So all polygons in a map service end up with the same transparency and other settings.  We keep large polygons in separate map services to avoid interference.
Keywords and Tags
To publish each map service for the Geodata Catalog, we started with an ArcMap document (.mxd) template that has properties fields pre-filled with SMC GIS standard text.  We paid close attention to location and topic keywords in the map document fields, because these keywords get imported as-is when you “register”  the map service with ArcGIS Online.  Good tags give better results when you search for layers to add to ArcGIS Online maps. 
Registering map services at ArcGIS Online
 “Registering the map service” is a specific process that Esri has set up to ensure that ArcGIS Server map services are easy to find and add to maps at ArcGIS Online.  Here are the steps register each service so that it also populates the SMC Geodata Catalog (map picker):
Cont here
When logged in to AGO at  My Content, navigate to the Geodata Catalog folder (Public or Intranet).
 Click Add Item > ArcGIS Server web service > [REST URL] ; paste in the REST URL for the SMC map service.
Tab through the other fields, and AGO autofills the tags and description that already appear in the published map service.
For this project, we have changed the auth-filled title to a standard title = “[map service name without hyphens ] – San Mateo County”.  (We appended “(Intranet)” while we were still publishing these to the dev server.)
Once the map service app is saved at AGO, click the “Share”button to share with “Everyone” plus only one AGO group (Public or Intranet Geodata Catalog)
You can edit the thumbnail by snapping a screenshot of the zoomed in map service displayed in the map.    Optimum file size is 200×133 pixels.
Update the SMC Map Service Publishing Log, if it is to be used to manage map service workflows and/or the Excel file will be used to generate the optional SMC Map Services Dashboard.

"Slashmaps for Mapbox" Drupal Module is cool. And I can do that with ArcGIS Online.


This post shows how I created map galleries for my Drupal web site like the galleries you get from the “Slashmaps for Mapbox” Drupal module.  

I worked on this because my maps are already in ArcGIS Online.  And because I’m a Drupal novice and found the new mapping module intimidating. 

Here is a quick-ref list of all the Drupal map galleries that are described and compared below:

1.  FCC.gov/maps built on the Slashmaps for Mapbox Drupal module.

2.  My ArcGIS Online map galleries based on the tag:
     a.  Drupal Map Gallery – Demo 1:  grid view 
     b.  Drupal Map Gallery – Demo 2:  list view
     c.  Drupal Map Gallery – Demo 3:  carousel
     d.  Drupal Map Gallery – Demo 4:  gallery hosted elsewhere and embedded here (gridview)

3.  Marten Hogeweg’s Esri_MapGallery Drupal module prototype

Are my ArcGIS Online map galleries as good as the Slashmap for Mapbox module?  You can look for yourself at the links above.  I’ll do a capabilities comparison in a future post.

The FCC Announcement:  Slashmap for Mapbox Module for Drupal

I’m in the process of migrating my Choptank River Heritage web site, which already includes a map gallery, to Drupal.  So I was interested in the FCC announcement back in March about their new map gallery.  And their collaboration with the Drupal community to build the Drupal module called Slashmaps for Mapbox.

I took a look at fcc.gov/maps:

And I was suprised at how similar it looks to my own map galleries built from ArcGIS Online at
maps.choptankeriverheritage.org and stories.choptankeriverheritage.org :
  


Since my Choptank River Heritage maps are already published in ArcGIS Online, I looked for a Drupal module that connects to ArcGIS Online.

Is There a Drupal module for ArcGIS Online?

I googled and found Marten Hogeweg’s Drupal sandbox for ArcGIS Online.  It was still rough.  So I contacted Marten and asked if Esri will develop and support a Drupal module for ArcGIS Online.  He said there’s interest.  And we plan to meet up at the Esri UC to talk more about this.  (Contact Marten or me if you want to join in.)

Meantime, Marten did some quick work over the weekend and committed a new Esri MapGallery module to Drupal git for me to try out.   Here is the very rough result so far.  Marten has already made improvements (such as removing the huge title banner) that I will incorporate in the next few days.

Do I really need a Drupal module for a map gallery?  How about a simple  tag ?

I also wondered if I couldn’t do a simpler Drupal integration using the tag.  Same as we see already for embedding ArcGIS Online maps into web pages:

And for embedding Google Maps into web pages:




My four Map Galleries for Drupal

Esri’s first “map gallery” was a simple display of maps that belong to an ArcGIS Online user group.  Nothing else to do.  Last summer (2011), Esri released its Javascript template that can be published in an organization’s own web site.  This required simple Javascript configuration and publishing the html, css, and js files to your web server.  This is still an option.  Since then, Esri has created other ways to publish map galleries in your own web site.  All of them run off of the user groups at ArcGIS Online.  

Here is the list of demo map galleries in my Drupal web site that leverage the tag to embed map group contents from ArcGIS Online.  Each of my Drupal pages gives a short explanation of the configuration and subtle differences between them:

1.  Drupal Map Gallery – Demo 1:  grid view is simple, usable as-is with no changes to sizing.

2.  Drupal Map Gallery – Demo 2:  list view, size has to be tweeked to hide annoying side panel, and the gawdy title bar is not configurable.

3.  Drupal Map Carousel – Demo 3:  map carousel, requires sizing but otherwise easy, and pretty cool.

4.  Drupal Map Carousel – Demo 4:   of the Esri gallery template already published on my non-Drupal web site.  Gives the most configuration control but requires some knowledge of HTML and CSS.


Let me know of your experience with the Mapbox module or these ArcGIS approaches.

Why do I still use both Chrome and Firefox? For bookmarks on desktop, laptop, and mobile.

Web browser bookmarks are important to me.  I use them every day for my professional GIS work.  I need fast access to ArcGIS and Python resources on the web.  I need to get back to training and tutorials.  I need to flip back and forth between web sites, Gmail, and calendar.

I put time and effort into managing and updating web browser bookmarks.  And I need to access them on my laptop, my office workstation, and my Android phone. 

Chrome gives me web bookmarks on my laptop and workstation — effortlessly.

Firefox puts my bookmarks on my phone.

Neither browser does desktop, laptop, and mobile.  
Why can’t Chrome or Firefox get the whole thing right?  I don’t know.
Chrome — Log in and get bookmarks everywhere … except on my phone.

Some of my co-workers resist the Chrome login idea.  They don’t want Google tracking their every web move.

But Chrome with login gives me updated bookmarks on both my laptop and workstation – or any other computer where I choose to log in to my Google account.  But not on my Google/Android phone.

The Android browser is crappy.  No bookmarks management at all.  There is no Chrome-like browser for Android mobile.  At least not the OS running on my DroidX —  Andoid 2.4.3.  There is hope at Android 4.0, which offers Chrome-to-Phone Beta and other Chrome toys.  (If I’m wrong, please leave a comment.)

Firefox — Manually import Chrome bookmarks to get them onto my phone.

The Firefox mobile browser is pretty nice, with tabs, good handling of Javascript and HTML5.  Best of all, it replicates my bookmarks in the same way I organize them on my desktop.


Not without a cost.  I have to a weekly export-import from Chrome to Firefox to keep all this up to date.


Why don’t I use IE?

Because IE mixes together web bookmarks and desktop “favorites”.  And that screws up everything.

Is there a better way?

ArcGIS Online pricing – what is a Service Credit worth?

Confused about AGOL pricing?  So was I.

I got on the phone with Esri today.   Here is is our Q&A:

ArcGIS Online for Organizations, Level 1 purchase gets “2500 Service Credits”.  But what does that mean?

Service credits are expended when ArcGIS Online functionality is deployed.  With the final release, we are providing a dashboard for the administrator, so that they may review how the service credits are being utilized.

Here are some examples:

1)      Service Credits are not used when you upload data or services to your instance of AGOL
2)      Service Credits are not consumed when using a esri basemap service in your application
3)      Service Credits are used when ‘mash’ up a shapefile, map service, table your company/agency uses in a service
4)      Service Credits are used when you create and store a feature service
5)      Service Credits are used when you create or store a tile or geospatial data service(layer, map package)
6)      Service Credits are used when you use a geoprocessing service (i.e. …batch geocodes), others to be added
7)      Service Credits are used when you do data transfer

This info would be more useful if it read “XX [Number of] service credits are used when…”

Here are examples of AGO Credit Consumption

  • Data Transfer – Data transferred out as hosted services or downloading data files
    • 6 credits / 1 GB data
  • Geocodes –
    • 12.5 geocodes / credit
    • 80 credits / 1,000 geocodes
    • 100,000 geocodes / 8000 credits
  • Tile & Data Storage – 1.2 credits per GB of storage per month, 14.4 credits/yr  per GB of data
  • Feature service –
    • 2.4 credits per 10 MB of storage per month, 28.8 credits/yr  per 10 MB
    • 2880 credits/yr  for 1GB of storage
  • Tile generation – 1 credit per 1000 tiles generated

Is there a more complete table online somewhere — to help me with service budgeting and making a decision about how much to purchase?

No.  But you might want to download the trial and give it a try. There is a dashboard that shows you how you are using your credits.

I need my proprietary GIS for this hard-core geospatial analysis, right?

Bill Dollins blogged recently that he doesn’t see much difference in capability between open source and proprietary geospatial tools.   


I think Web/GIS developers would agree.  But I’m not so sure that geospatial analysts would agree.


I agree, if we’re talking about building and using map apps — displaying points, lines, and polygons, and routine map functions like routing, thematic mapping, and interactive display of the map and underlying data.


But what about hard-core geospatial analysis?  Case at hand:


I’m working on an emergency management application that estimates the age-group populations that may be affected by an emergency event.   The only population and demographics data I have is census block groups.  I create a buffer around the event, then clip the census block groups that intersect the buffer.



The problem is that for many of the included block groups, only a small portion lie within the event zone.  So, only a fraction of the population in each of those groups should be included.


I was ready to write the function for my Python script that would calculate the percentage of the area of  each block group that got clipped.  If only 15% of the area got clipped, I would grab only 15% of the population counts for that block group to add to my population total.


Then I saw Esri’s announcement that ArcGIS 10.1 now includes an has areal interpolation tool.



Seems like the perfect fit for what I need to do.  I can create a grid of 100 x100 meter polygons and re-assign portions of the population counts to these much smaller and regularly shaped grid polygons.  And save this data layer for use any time a need a more precise population estimate.


Can any open source geospatial tools do this?  I admit that I don’t know.


I think it would be good to do something like what Tobin Bradley did to evaluate the importance of different elements of the Google Maps API.  Maybe do the same with Esri’s ArcToolbox.  How much of this tool set is present in open source geospatial software?  How important are the missing tools?


Has anyone besides Esri done an assessment like this?



Am I projecting or transforming that coordinate system? Or both?

We received a shipment of data from Esri for Philadelphia business locations.  We needed it transformed from geographic coordinate system (lat/long) to Penna State Plane South (3702) coordinate system (in feet).  Esri told us we have to do that ourselves.
Projections, coordinate systems, and transformations are always a little confusing for those of us who don’t mess with them a lot.  Esri’s tool labeling doesn’t help any.
We had a team huddle and decided we were not projecting a coordinate system — we were transforming from one coordinate system to another.  I looked in ArcToolbox under  Data Management > Projections & Transformations.  There is no Transform tool.  So I tried the only tool in there for Features:  Project.
I opened the Project tool and ignored the options first time around.  Got an error for not choosing the option:
Read the Help.  The option is required if you are changing coordinate systems.  Which is what we came here for in the first place.
How to choose?


Not sure.  I saw no hints in the input or output coordinate system descriptions.  So I used the first option on the list shown .  Looks like to worked fine.  This tells me where in State Plane:

Questions:

1.  Were we transforming or projecting?  Or both?  

2.  Is “re-projecting” a necessary step in transforming a data set’s coordinate system from one to another?

3.  How do we decide which Geographic Transformation to apply?

I appreciate your comments.


Geospatial One-Stop >> Geo.Data.gov >> GeoPlatform.gov

Here are details about GeoPlatform evolution, provided by Esri’s Geoportal Server lead, Marten Hogeweg, and published here with Marten’s permission:


Geoplatform.gov is actually built on ArcGIS Online which is running on-premises in the GSA hosting environment. Geoplatform.gov currently has a small set of publishers who ‘curate’ the content that is visible, with a focus on web services.

When Geospatial One-Stop retired, it was integrated into the Data.gov website as http://geo.data.govGeo.data.gov IS built on the open source Geoportal Server (http://esriurl.com/geoportalserver).

There were close to 650,000 items in Geospatial One-Stop, many of which were from state/local government or from academia and do not meet the criteria (http://www.data.gov/datapolicy) to be made discoverable through Geo.data.gov.

To provide access to that full set (by now grown to about 950,000 geospatial resources), the search from the Geoplatform.gov site was included.

There are really two programs (Geoplatform and Data.gov) that are looking at how to best implement their objectives/mandates: serve nationally significant geospatial data assets from and to the geospatial community at large) vs increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government (http://www.data.gov/about).

What you see now is (hopefully) a transition from the Geospatial One-Stop period into the new open government data period where these two objectives/mandates unite…