Category Archives: Data

Tufts Data Rescue

From 1/20-1/23 Tufts University and the Open Geoportal Community conducted a sizeable U.S. federal data harvest. We surveyed Tufts faculty to identify those geospatial data most critical their teaching and research. We then conducted a curated harvest to back up potentially at-risk federal, environmental and social justice geospatial data and associated tabular data.  Currently we are at around 20 TB of data backed up on Tufts Research Data Storage Network.  This is an ongoing effort as we are continuing to harvest data daily. Please contact Patrick.florance@tufts.edu if you would like to participate or learn more.

Here is a summary of the Tufts OGP Data Rescue as of 3/1/2017

Other Data Rescue Efforts Around the Nation

Nationwide Data Rescue at Penn with Calendar of Events
Datarefuge.org – Penn CKAN repository

End of Term project

Environmental Data and Governance Initiative (EDGI)

Climate mirror

Azimuth climate data backup project:
Run by John Baez baez@math.ucr.edu

Crawling Rockland Site

Data from the Rockland County (NY) GIS site is now available at http://www.WorldWideGeoWeb.com.  The following image shows some of the search results after a crawl based ingest of the Rockland data site:

RocklandSearchResults

Even though no web services may be available, these search results can still be previewed. The following image shows the Rockland boundaries for State Senate:

RocklandStateSenate

Here’s a screenshot with two separate bus routes previewed.

RocklandBusPreview

On the Rockland County site, many bus routes are stored in a single zip file.  These are individually searchable and previewable on WorldWideGeoWeb.

Directed crawling of new sites presents new challenges and reveals limitations in the existing code.  Several changes were made to successfully crawl Rockland, NY data at https://geopower.jws.com/rockland/DataPage.jsp .

The Rockland site does not contain links to zip files.  A typical link to a data file is https://geopower.jws.com/rockland/DownloadData.jsp?pck_oid=2464.  The crawl code was changed change to support links to servlets rather than simple zip files.

The Rockland metadata files contain minimal, often cryptic titles; for example,“monsey2” and “TZX”.  These titles are not sufficient.  Fortunately, they can be augmented with information scraped from the crawled web page.  Specifically, the text from the anchor tag linking to DownloadData.jsp servlet is concatenated to the title field in the xml metadata file.  This creates user friendly titles, for example, “monsey2: TOR Bus Routes” and “TZX: Tappan Zee Express Bus Route”.

The Rockland site contains zip files that hold multiple shapefiles.  For example, the file TOR.zip contains 6 separate bus routes, each in a separate shapefile.  Each shapefile is ingested as a separate entity so it can be independently searched and previewed.  The 28 links on the Rockland data page expand into 47 searchable, previewable spatial resources.  Note that the OpenGeoPortal download operation pulls down entire shape files from the Rockland server, not the individual shapefiles.

Since the Rockland site only supports secure connections, the ingest code was enhanced to support https.

Crawling Westchester Site

At the NYC OGP meeting, I demoed my thesis site. Since there isn’t a video of it, here’s a write-up covering the same material. There are also some slides available.

The goal of my Masters thesis is to make all the world’s spatial data accessible. This goal is accomplished by expanding OpenGeoPortal in two significant ways. First, spatial data files are discovered via web crawls and then ingested. Second, the ability to preview and download layers without needing OGC protocols was developed. This expanded version of OpenGeoPortal is on the web at http://WorldWideGeoWeb.com.

Data on WorldWideGeoWeb.com was discovered by crawling the web, relying exclusively on HTTP GET requests. This is the same technique used by Google and other search engines. The WorldWideGeoWeb crawler can be instructed to crawl a specific site. Sites are searched for links to zip files. The ingest code retrieves and unzips these files. If they contain a shapefile, the bounding box is determined using the shp and prj files. Any metadata file is also parsed. Information about each discovered layer is ingested into WorldWideGeoWeb’s Solr instance.

After ingest, OpenGeoPortal’s powerful search interface allows users to quickly and easily find spatial data layers. Preview of shapefiles on the map is based on parsing and rendering shapefiles entirely in JavaScript. It does not use image tiles from GeoServer or ArcGIS Server. When the user selects a layer to preview, the browser sends a request to the server to create a temporary, server-side copy of the zip file. During ingest, the URL of the zip file was stored in the Solr record. This is used and an HTTP GET request is issued to create local copy of the zip file. Then the file unzipped. At this point the browser requests the .shp, .shx, .prj and .dbf elements of the shapefile. They are processed in JavaScript as binary data streams. If the data is not in a suitable projection, it is reprojected on the browser. Then the features in the shapefile are parsed are rendered on OpenGeoPortal’s map. Attributes in the .dbf file are displayed as features are moused-over.

The following screenshot shows WorldWideGeoWeb.com. The search results were discovered by crawling Westchester County’s data web site at http://giswww.westchestergov.com/wcgis/DataWarehouse.htm. The map shows a preview for layer titled ”County Legislative Districts”. The browser debug panel at the bottom of the screenshot shows the network traffic generated by the preview request. The “cacheShapeFile.jsp” ajax call told the server to copy the shapefile from http://giswww.westchestergov.com using an HTTP GET and unzip the results. After the Ajax request completes, the .shx, .shp, .dbf and .prj are requested by the browser and parsed in JavaScript. Transferring this 220 kilobyte layer first to the WorldWideGeoWeb server and then to the browser took just under 2 seconds.

WestchesterCrawl4

The user can add any of these Westchester layers to the cart and download them. The zip files are transferred directly from the Westchester server to the browser. Clipping the data or converting it to another format are not supported.

WorldWideGeoWeb shows it is possible to build a powerful, interactive portal without requiring data holders create web services. Data only available on web sites designed for people can be ingested using a web crawl and previewed using advanced JavaScript techniques that weren’t available when the OGC protocols were created. Since WorldWideGeoWeb is built on OpenGeoPortal, data available with web services can also be supported.

Limitations

There are significant limitations with the current version of the software. Most notable is its inability to deal with large shapefiles. Currently, shapefiles over one megabyte can cause the browser to hang. Search results are color-coded to advise the user. Green layers are small and should preview quickly. Yellow layers are larger but should preview without too much delay. Layers in red represent shapefiles over a megabyte and should not be previewed. Even these large layers can be downloaded easily downloaded, just not previewed.

My thesis code is not production quality.

Future Directions

During a crawl, only spatial resources in shapefiles are discovered and their associated metadata must be in FGDC or ISO19115. It would be trivial to add support for KML and KMZ files. Support for other file formats and metadata standards could also be integrated.

Crawling based OGC protocols such as Get Capabilities and CSW could be added.

The ranking of the search results could be based on the “Page Rank” of the page that linked to the zip file.

Semi-spatial data such as web pages about places could be ingested and searched spatially.

Other Notes

I now work for Voyager Search (http://voyagersearch.com/). We are investigating how some of these ideas could be incorporated into their existing products. The code created for my thesis has been released under the GPL.

Some data an organization provides may be more critical or widely used. This data could be available via an OGC compliant server while other, less critical data is made available only via HTTP GET.

NYC Open Geoportal Meeting – December 13th

LINK TO SUMMARY OF GREATER NYC OPEN GEOPORTAL MEETING

NYC Open Geoportal Meeting
Friday, December 13th
Location: New York University (NYU)
Map of Meeting Location (PDF)

Interactive Map

Morning Sessions (8:30am-12:30pm)
NYU Kimmel Center for University Life – Room 914, 60 Washington Square S, New York

Afternoon Working Meeting (1:30-4:45pm)
NYU Bobst Library – Room 619, 70 Washington Square S, New York, NY

Through the support of the Alfred P. Sloan Foundation and NYU, we are happy to announce the first New York City Open Geoportal Meeting on Friday, December 13th hosted by New York University (NYU).  The goals of the meeting are to:

1. Build relationships between individuals and organizations interested in geospatial data management around the greater New York City area.
2. Introduce the Open Geoportal community, software application and underlying technology
3. Work towards setting up a NYC Open Geoportal environment.

AGENDA
In the morning we will introduce OGP functionality, underlying technology, metadata requirements, and future developments. We will also allow for selected participants to provide an optional five minute ignite talk introducing their interest in greater NYC geospatial data.

After the morning sessions, we will hold a smaller gathering of working meetings to discuss a NYC OGP task force, answer detailed technical questions, and determine key action items to develop a NYC OGP instance. The afternoon is designed for those with an interest in spatial data infrastructure (SDI).

Participant List (PDF)

Agenda (PDF)

DETAILED AGENDA

Time Title Presenter
Morning Conference (8:30am-12:30pm)
NYU Kimmel Center for University Life – Room 914
60 Washington Square S, New York, NY
8:30-9:00 Food & Refreshments
9:00-9:30am Welcome & Introductions Scott Collard, NYU
9:30-10:00am Ignite Sessions 1
Alan Leidner, Booz Allen; NYS GIS Association
Matt Knutzen, New York Public Library
Himanshu Mistry, New York University
Nathan Storey, PediaCities
Selected Presenters
10:00-10:30am Overview of Open Geoportal Patrick Florance, Tufts
10:30-11:00am Overview OGP Information Architecture & Underlying Technology Chris Barnett, Tufts
Steve McDonald
11:00-11:30am Break with Refreshments
11:30-12:00pm OGP Current & Future Developments: OGP 2.0, Harvester, Metadata Toolkit, Hosted Services, OGP Crawler
Chris Barnett & Patrick Florance, Tufts, Steve McDonald
12:00-12:30 Ignite Sessions 2
Steve Romalewski, Center for Urban Research, CUNY Graduate Center
Frank Donnelly, Baruch College–CUNY
Holly Orr, New York University
Eric Glass, Columbia University
Selected Presenters
12:30-1:30pm Lunch
Afternoon Working Meetings (1:30-4:45pm)
NYU Bobst Library – Room 619
70 Washington Square S, New York, NY
1:30-2:15pm Greater NYC Spatial Data Infrastructure (SDI) Participants discuss their SDI
Data holdings, IT infrastructure, staff, metadata, etc.
2:15-3:00pm Metadata Formats & Best Practices Intro: Marc McGee, Harvard
Participants discuss metadata authoring, workflow,  coordination
OpenGeoportal_Metadata
3:00-3:30pm Break
3:30-4:30pm Information Architecture (IA) & Governance Participants discuss IA & Governance needed to establish a greater NYC OGP Instance
4:30-4:45 Summary Patrick Florance, Tufts

REGISTRATION
General registration is now closed.

Participants include Columbia University, NYU, CUNY Graduate Center, Hunter-CUNY, Baruch-CUNY, New York Public Library, Tufts University, Harvard  University, and various local and federal government agencies among others.

QUESTIONS
Please contact patrick.florance@tufts.edu

BACKGROUND
The idea of the NYC OGP originated during the National OGP Summit. The Open Geoportal is a collaboratively developed, open source, federated web application to rapidly discover, preview, and retrieve geospatial data from multiple repositories. It is also a community of best-practice around those interested in geospatial data management and spatial data infrastructure (SDI). It was developed by Tufts, Harvard, and MIT and currently has over 30 partner organizations.

Tufts Instance – just zoom in on the map to New York City, Boston, Afghanistan, etc. to see some relevant holdings
http://geodata.tufts.edu

Open Geoportal Organization
http://opengeoportal.org