What’s eating up my ArcGIS Online credits?

My presence on ArcGIS Online is small:  Choptank River Heritage.

It runs on the Esri Home License – ArcGIS Pro and an Online site with 100 credits – for $100 per year.  I use it to improve my ArcGIS skills while publishing historic maps to support my Choptank River Heritage volunteer blog about Caroline County, Maryland, history.

I was surprised to find that 7 percent of my annual credits were used up in the past 14 days, mostly for storage:


Storage of what?  You have to drill down deep to find that.  The help doc is here.  The gist is:

Click View Status > Credits > Storage > [Storage Usage Chart] Feature Storage > [Feature Storage Usage Report] Database Storage.  Then download the CSV.
 
Now I see it.  I can certainly get rid of this stuff:
 

 
 
And pay out another $100 to keep my historic maps and GIS skills development going.
 

OK. I’m not a GeoHipster. I’m a GIS Worker. This is what I do.


But worry not, GIS worker: Spatial might no longer be special, but projections, datums, and legacy file formats will continue to be very, very special.
      – Brian Timoney, MapBrief

Okay… GIS is not sexy like your iPhone locator app.  But it will always be special.  In the same way that your plumbing and electrical and HVAC systems are special.  Next time they stop working, ask yourself just how special they are.  Next time your iPhone map is missing Main Street, ask yourself how special GIS is.

Consumer and business geo apps and services keep evolving – cutting edge today and obsolete tomorrow.  BigData today – legacy file format tomorrow.  But projections, datums, and georeferencing will always be there to provide ground truth. 

Esri and the ArcGIS platform have been around for 30 years, defining and documenting GIS work.  So when we needed to understand what GIS workers are doing across a variety of industries, we looked at the list of tools in ArcGIS Toolbox as a compendium of GIS workflow and process skills.  

We needed the ArcToolbox list itself as a data set — not a PDF poster or searchable web site.  I asked Esri for an ArcToolbox list as Excel table or other word processing format, but they could not provide.  So my friend at Esri, Nick Toscano, and I compiled the table ourselves.   That’s almost 900 “GIS tools” in an Excel data table.  Easy to sort, filter, annotate, categorize in ways that meet our needs.  

Here it is.

Some of the tools are pretty arcane.  But most are plain markers for the nuts-and-bolts work of GIS professionals in towns, cities, and counties who have to deal with projections, datums, and legacy file formats every day.  Not the next wave or next-big-thing.  Just the actual data and task at hand.

How will we use this?

  • Survey and assess the experience of our GIS workers.
  • Plan GIS skills training and investment.
  • Design new workflows to make common tasks more efficient.
  • As a quicker reference for building geoprocessing services. 



"Slashmaps for Mapbox" Drupal Module is cool. And I can do that with ArcGIS Online.


This post shows how I created map galleries for my Drupal web site like the galleries you get from the “Slashmaps for Mapbox” Drupal module.  

I worked on this because my maps are already in ArcGIS Online.  And because I’m a Drupal novice and found the new mapping module intimidating. 

Here is a quick-ref list of all the Drupal map galleries that are described and compared below:

1.  FCC.gov/maps built on the Slashmaps for Mapbox Drupal module.

2.  My ArcGIS Online map galleries based on the tag:
     a.  Drupal Map Gallery – Demo 1:  grid view 
     b.  Drupal Map Gallery – Demo 2:  list view
     c.  Drupal Map Gallery – Demo 3:  carousel
     d.  Drupal Map Gallery – Demo 4:  gallery hosted elsewhere and embedded here (gridview)

3.  Marten Hogeweg’s Esri_MapGallery Drupal module prototype

Are my ArcGIS Online map galleries as good as the Slashmap for Mapbox module?  You can look for yourself at the links above.  I’ll do a capabilities comparison in a future post.

The FCC Announcement:  Slashmap for Mapbox Module for Drupal

I’m in the process of migrating my Choptank River Heritage web site, which already includes a map gallery, to Drupal.  So I was interested in the FCC announcement back in March about their new map gallery.  And their collaboration with the Drupal community to build the Drupal module called Slashmaps for Mapbox.

I took a look at fcc.gov/maps:

And I was suprised at how similar it looks to my own map galleries built from ArcGIS Online at
maps.choptankeriverheritage.org and stories.choptankeriverheritage.org :
  


Since my Choptank River Heritage maps are already published in ArcGIS Online, I looked for a Drupal module that connects to ArcGIS Online.

Is There a Drupal module for ArcGIS Online?

I googled and found Marten Hogeweg’s Drupal sandbox for ArcGIS Online.  It was still rough.  So I contacted Marten and asked if Esri will develop and support a Drupal module for ArcGIS Online.  He said there’s interest.  And we plan to meet up at the Esri UC to talk more about this.  (Contact Marten or me if you want to join in.)

Meantime, Marten did some quick work over the weekend and committed a new Esri MapGallery module to Drupal git for me to try out.   Here is the very rough result so far.  Marten has already made improvements (such as removing the huge title banner) that I will incorporate in the next few days.

Do I really need a Drupal module for a map gallery?  How about a simple  tag ?

I also wondered if I couldn’t do a simpler Drupal integration using the tag.  Same as we see already for embedding ArcGIS Online maps into web pages:

And for embedding Google Maps into web pages:




My four Map Galleries for Drupal

Esri’s first “map gallery” was a simple display of maps that belong to an ArcGIS Online user group.  Nothing else to do.  Last summer (2011), Esri released its Javascript template that can be published in an organization’s own web site.  This required simple Javascript configuration and publishing the html, css, and js files to your web server.  This is still an option.  Since then, Esri has created other ways to publish map galleries in your own web site.  All of them run off of the user groups at ArcGIS Online.  

Here is the list of demo map galleries in my Drupal web site that leverage the tag to embed map group contents from ArcGIS Online.  Each of my Drupal pages gives a short explanation of the configuration and subtle differences between them:

1.  Drupal Map Gallery – Demo 1:  grid view is simple, usable as-is with no changes to sizing.

2.  Drupal Map Gallery – Demo 2:  list view, size has to be tweeked to hide annoying side panel, and the gawdy title bar is not configurable.

3.  Drupal Map Carousel – Demo 3:  map carousel, requires sizing but otherwise easy, and pretty cool.

4.  Drupal Map Carousel – Demo 4:   of the Esri gallery template already published on my non-Drupal web site.  Gives the most configuration control but requires some knowledge of HTML and CSS.


Let me know of your experience with the Mapbox module or these ArcGIS approaches.

ArcGIS Online pricing – what is a Service Credit worth?

Confused about AGOL pricing?  So was I.

I got on the phone with Esri today.   Here is is our Q&A:

ArcGIS Online for Organizations, Level 1 purchase gets “2500 Service Credits”.  But what does that mean?

Service credits are expended when ArcGIS Online functionality is deployed.  With the final release, we are providing a dashboard for the administrator, so that they may review how the service credits are being utilized.

Here are some examples:

1)      Service Credits are not used when you upload data or services to your instance of AGOL
2)      Service Credits are not consumed when using a esri basemap service in your application
3)      Service Credits are used when ‘mash’ up a shapefile, map service, table your company/agency uses in a service
4)      Service Credits are used when you create and store a feature service
5)      Service Credits are used when you create or store a tile or geospatial data service(layer, map package)
6)      Service Credits are used when you use a geoprocessing service (i.e. …batch geocodes), others to be added
7)      Service Credits are used when you do data transfer

This info would be more useful if it read “XX [Number of] service credits are used when…”

Here are examples of AGO Credit Consumption

  • Data Transfer – Data transferred out as hosted services or downloading data files
    • 6 credits / 1 GB data
  • Geocodes –
    • 12.5 geocodes / credit
    • 80 credits / 1,000 geocodes
    • 100,000 geocodes / 8000 credits
  • Tile & Data Storage – 1.2 credits per GB of storage per month, 14.4 credits/yr  per GB of data
  • Feature service –
    • 2.4 credits per 10 MB of storage per month, 28.8 credits/yr  per 10 MB
    • 2880 credits/yr  for 1GB of storage
  • Tile generation – 1 credit per 1000 tiles generated

Is there a more complete table online somewhere — to help me with service budgeting and making a decision about how much to purchase?

No.  But you might want to download the trial and give it a try. There is a dashboard that shows you how you are using your credits.

Explain it to Your Boss: The Map is There but Your Data is Still Here

Some business managers don’t want “our” data sitting on “their” map server.   So they’re reluctant to share sensitive business data in maps that are authored and published at public map portals.


They may think that when we publish our map data in the “cloud”, we give up ownership and the security oversight of our data.   That’s not necessarily so.  You can explain it to your boss like this, using an example from the ArcGIS Online mapping portal —  

You can sit behind your firewall and publish your sensitive data in a web map.  Here’s what it looks like:



Your boss might not understand that although the map is at ArcGIS Online, your map data does not have to be.  Web page and map components can come from many public web servers, while the map data itself remains secure.  The map data comes from your secure GIS server:



It gets delivered to the web map only if the browser user has been granted access to our GIS server; that is, if the user’s computer is also sitting behind the same firewall with the GIS server, or if the user has login access to your GIS server from outside.   

If no access is granted to your GIS server, the points will not appear on the web map.   Only the background (basemap) will appear.

The business data (map points) are all we need to own and control.  At the same time, we can build on free, publicly-available code and data platforms for map programming and publishing.   For example, the background map could come from public servers at Esri, Google, or OSM:



We don’t have to build the map widgets ourselves.  In this example, they’re sent to the browser from Esri’s Javascript API server.  But they could have come from Google Maps or another server:

The ”Dojo-Javascript”  layout of the web page – header, banner panel, navigation panel – could be from code that sits on the Google or Yandex CDN servers:



Code from other Web servers could also be embedded in the map or the web page.  For example, we might overlay our data layer with weather data coming from NOAA or tweets from Twitter.  So, in an example like this, we may rely on 4 or more different public web servers, all supporting the publication and sharing of our secure geospatial data.

So, your boss and trusted colleagues can sit behind our firewall and benefit from web mapping in the public cloud.  Their web browser reaches out to the public Internet to bring in elements of the map and page that surround your map data.  But the map data itself stays inside a secure channel from our GIS server to the web browser and map.

I know you understand this.  But your boss might not.


::: ——- :::

Not so simple?

My scenario above won’t address the concerns of all business managers.  For example, I worked with a county health department whose management was reluctant to serve up water quality monitoring maps to the public unless they appeared on the county’s web site, rather than at ArcGIS.com.  It was an ownership and branding issue. 

Esri has tried to address this concern with its Public MapsGallery template.  The Gallery is a clever approach that lets an organization author, publish, and host maps at ArcGIS Online, but deliver them in web pages that come from the organization’s own web server, with the organization’s own style and labeling.  No “ArcGIS Online” labels anywhere.  Too clever?  Your boss will have to decide.

Valid security issues — not just branding — are raised if you actually upload some or all of your data to the public map portal.  Or if you go to the public portal to author and publish information related to the map, such as authorship, map description, and geospatial metadata.  In that case, you need to understand the hosting services security policy and implementation.  You can read about Esri’s for ArcGIS Online, here .

Okay, ArcGIS Online is not an online geospatial CMS

I read Jim Fee’s blog to get a GIS reality check.  He challenges assumptions and points out the color of the Kool-Aid.


So, he’s right:  Esri’s ArcGIS Online is not a geospatial content management system.  Why not?  He says:


– It doesn’t support open standards let alone other formats.
– There is no geneology of data.
– There is no lifecycle to the product.


By that definition, it’s not an online CMS.  


But the ArcGIS platform certainly is a geospatial content management system, by any valid definition.  and there are good reasons why GIS professionals (I mean, people trained in geospatial content creation, management, and presentation) buy into Esri’s proprietary formats and “$10,000 client”.  


It’s because our work is more complex than simply creating Google My Maps.  For more complex geospatial work, GIS product developers and enterprise managers have told me time and again that open source doesn’t always make economic sense for their bottom line.


When Esri has marketed ArcGIS Online before now, they’ve emphasized the web map, not “online geospatial content management”.  You don’t have to fall off a turnip truck to appreciate what Esri has done for us in the web map arena.  GIS professionals (I mean, cartographers, geographers, geo-statistical analysts, crime analysts who produce location-based intelligence, health professionals using geospatial analysis to see trends) can now publish and share our data in maps on the Web, without hiring a programmer.  We couldn’t do that a couple of years ago.  


Our data isn’t in CSV files or Fusion Tables, and we don’t share it by clicking on the web map to set pushpins.  Our data is in our geospatial content management systems made up of geospatial databases, GIS servers, and applications servers.  We can hire Esri to integrate all that into a geospatial content management system.  Or we can hire other systems integrators to do it from scratch, using open source.


So, maybe it’s premature to call ArcGIS Online an online geospatial CMS.  But that, too, is just around the corner for organizations that I’ve worked with.  For good reasons, most of them use the ArcGIS platform to create, manage, and present their geospatial data.  Most of them are now talking seriously about moving their entire geospatial enterprise into the Esri cloud.   Call it ArcGIS Online.



Metadata Lite: Publishing metadata cheap (or not at all) through ArcGIS Server

You can use ArcGIS Desktop to author metadata for data layers in your geodatabase, and to publish your data as web services on ArcGIS Server.  But you can’t publish your geospatial metadata with those web services.

Esri will say, Not true.  You can publish your metadata to the Web through Esri’s free Geoportal Server.  Then point your metadata on the Geoportal Server to your map services on your ArcGIS Server, like this:
Right.  But that means you have to do the same task twice — send your metadata to your geodatabase and send it again to your geoportal.  Besides maintaining two applications servers for this purpose.
This redundancy sits atop the already cumbersome and expensive process of authoring, publishing, and maintaining FGDC/ISO-compliant geospatial metadata.
Are there less expensive options?  Even Esri’s FGDC/ISO-compliant metadata guru seems to think so.  Here is a slide from Marten Hogeweg’s geoportal workshop at the 2011 Dev Summit:
(The title/question is mine, not Marten’s.)

Marten was talking about different metadata for different audiences.  The public does not want or need FGDC/ISO compliant metadata.  So, how can we meet the public demand, as well as the needs of our professional partners in the GIS community?  I suggest a middle path – Metadata Lite – that’s already suggested in Marten’s diagram.
Yes, “Verbose Metadata is Desired” for the GIS Specialist Community.  But today more GIS professionals are concerned with pushing data out to the public.  Knowing that, it seems to me that Esri (unconsciously?) moved away from fully-compliant metadata  when they “forgot” to support metadata publishing through ArcGIS Server 9.3 and 10.  And now we see mere “tagging” promoted in Marten’s metadata/geoportal presentations.    
Take a look at ArcGIS.com.  At the Esri MUG last week, Clint Brown told us that millions of maps are added to ArcGIS.com each month.  But where is the geospatial metadata that enables search in a geoportal?  There is none at ArcGIS.com.  You can’t search through others’ FGDC/ISO-compliant metadata, and you can’t publish your own there.  
There is only Metadata Lite — tagging to give a minor boost to the ArcGIS.com search function:

If you want to publish descriptive info about your map data, there are a couple of free/low-cost options.  First, you can publish your data in map apps at ArcGIS.com.  (See my other posts about using ArcGIS.com for your organization’s geodata portal.)  Then simply author the descriptive information in the details page.  San Mateo County GIS has done it this way, using a text template to ensure that all the legal info is included, like in this example:

Even better, I think, is to publish your geospatial metadata through your ArcGIS Server REST catalog.  When it’s in the REST catalog, it doesn’t intrude on the public attention span, yet GIS professionals can find it when they need it.

This is what Metadata Lite looks like from San Mateo County, published through ArcGIS Server to the county’s REST catalog:

But 90 percent of the time, if you open up a REST service page like this, all the fields are blank, except a few that are machine-generated,  like Extent and Spatial Reference.  Why can’t we simply author metadata that shows up in the REST catalog?  Because …

You need a wiring diagram.  Because some of the REST catalog fields are populated from the ArcMap Document Properties tab.  Others from the ArcMap Data Frame Properties tab.  Others from the ArcGIS Server Manager Properties tab.  Like this:
Besides that, the field names are different for the same information found in the source (e.g. ArcMap) and the destination (REST page).  Like this:
So, here are a couple of wiring diagrams to help you populate the REST catalog fields:
2.  Metadata Lite live map service – hosted at University of Maryland, shows where the REST catalog fields values originated.
(From a presentation I gave at the 2011 Esri MUG)