GALEON Phase 1 Implementation Goals and Status

GALEON Implementation Goals and Status

Implementation goals and status for individual GALEON client and server implementations. There is also a complementary Client Server Interaction Status Table.

Server implementations

Unidata

THREDDS Data Server: see http://motherlode.ucar.edu:8080/thredds/docs/WCS/index.html, especially "Notes for clients"

U of Florence and IMAA-CNR

We are involved in the implementation of a WCS 1.0 server (WCS-G) capable of serving (subsets of) netCDF-CF datasets, in binary netCDF format as well as in ncML-GML. The server is based on an underlying OPeNDAP server instance and on the ncML-GML Java API, performing the conversion from netCDF-CF to ncML-GML

We are investigating a generic conceptual mapping from netCDF model to WCS model. To this end, we have identified several testbed datasets, with different features and degrees of complexity (summarized in the table below). WCS-G is primarily targeted at accommodating those datasets, compatibly with WCS 1.0 limitations

 

• WCS-G server
  • Notes: WCS-G currently provides access to the following netCDF-CF datasets: simple.nc, sst.nc, sst-2v.nc, trid, striped_can (see below), few other CEOP datasets
  • HTTP binding:
    • getCapabilities URL 
    • describeCoverage URL (sst.nc)
    • getCoverage URL (subsetting of sst.nc in ncML-GML format)
    • getCoverage URL (subsetting of sst.nc in netCDF format)
      (note: because application/x-netcdf mime-type is not supported by web browsers, the file will be saved using the servlet name. Please rename it using .nc extension)

• • SOAP binding
    WSDLs URL: http://athena.pin.unifi.it:8080/galeon/services
  • Roadmap:
    • TBD: support to rangeset queries
    • TBD: support to parametric rangeset

  WCS 1.0 implementation which is able to leverage ncML-GML capabilities in order to:

  • facilitate the generation of necessary semantics for implementing the WCS protocol and data model, starting from netCDF-CF dataset

  • generate and return a netCDF dataset description using both Earth Science and GIS metadata structure and semantics -i.e. ncML and GML grammar, respectively

  • address the granularity difference issue existing between netCDF dataset and WCS coverage structures: to introduce the "netCDF set of coverages" structure which makes explicit the implicit and, potentially, complex structure of netCDF datasets

  • include ASCII data in XML, directly

  • link netCDF binary data

A useful vision is provided by "Extending THREDDS middleware to serve OGC community" presented at EGU '05

• N2G service (ncML-GML API demo)
  • Notes: implementation of a service which extracts a ncML-GML representation of a netCDF-CF dataset, by means of the ncML-GML Java API. Current version is 0.8
  • Service URL: http://athena.pin.unifi.it:8080/ncml-gml/n2g-form.htm
  • Roadmap: N2G versioning depends on the complexity of netCDF data supported by the component, as reported in the following table:

Test Dataset	Coverage domain	Coverage co-domain	CRS	Amount of data	N2G version
simple	2D + T	scalar (single)	Geo	small	ver 0.2
sst	2D + T	scalar (single)	Geo	medium	ver 0.4
sst-2v	2D + T	scalar (array)	Geo	medium	ver 0.6
trid	3D	scalar (single)	Geo	small	ver 0.7
striped_can	2D + T + P	parametric	Geo	large	ver 0.8
ruc	3D + T + P	parametric	Geo + Proj	large	ver 1.0

• ncML-GML specification
  • Version 0.7.3 schema URL: http://zeus.pin.unifi.it/schemas/ncml/netcdf-coverage.xsd
  • Roadmap:
    • TBD: Version 1.0, full GML application

ncML-GML is:

• An Abstract and Content Model reconciliation schema for ES and GIS info realms
• A Mediation Markup Language between ncML (netCDF Markup Language) and GML
• An extension of ncML core schema, based on GML grammar

A slide presentation and a paper on ncML-GML v.0.5 are available at: http://www.unidata.ucar.edu/projects/THREDDS/Nativi/GML-Days/GML_Days_Presentation.pdf

http://my.unidata.ucar.edu/projects/THREDDS/GALEON/ncML-GML%200.5_submitted.pdf

 

International U of Bremen (IUB)

IUB is researching on services for large multidimensional rater data, based on the rasdaman raster server which maintains n-D raster data in standard relational databases and gives access via rasql, a raster-extended SQL.

Achievements

In GALEON, IUB has set up a 4-D/5-D database of climate simulation data using rasdaman and the open source DBMS PostgreSQL served via WCS 1.0.0, using data imported into PostgreSQL from netCDF files. The service, which is publicly accessible under http://mango.eecs.iu-bremen.de:8000/~galeon/, supports GET/POST as well as SOAP based GetCapabilities / GetCoverage requests.

Based on GALEON findings, IUB has prepared a list of WCS change requests. To optimise contribution towards the WCS.RWG team, these are based on the current internal discussion version WCS 1.1. A main concern was that WCS currently does not specify at all the coverage contents ("pixel values") - hence, for example an all-black image of appropriate structure would be a perfectly valid response to any GetCoverage request. In February 2006, WCS.RWG has adopted the text proposal to become Section 10.4 of the forthcoming WCS 1.1.

Further, a proposal has been elaborated to define a Web Coverage Processing Service (WCPS) which, based on the conceptual model of WCS, allows to submit arbitrarily nested requests for evaluation on one or more coverages. This is currently under discussion within OGC.

The outcome is summarised in the final report.

Task overview:

• Tasks completed:

• analyse netCDF data format elements (grid, metadata) -- done

• set up database server (linux; postgresql; rasdaman; tomcat) -- done

• specification of data import facility (netCDF to rasdaman database) -- done

• specification of WCS Java server, including SOAP binding -- done

• implementation of data import facility -- done

• implementation of WCS Java server -- done, supporting both GET/POST and SOAP requests

• populate database with sample data set -- done

• deploy publicly accessible server -- done

• Tasks to be addressed:

• test with GALEON clients -- to be done

• Issues wrt. WCS spec:

• WCS 1.1 change requests [ v2 | v3 | v5 ] resulting from our WCS work, in particular: specification of coverage contents in GetCoverage response.

• Linked work on extending WCS to a WCPS (Web Coverage Processing Service):
  • WCPS version 0.0.1, distributed to WCPS group on 16-dec-2005

RSI

RSI have implemented a netCDF to GeoTiff conversion interface using the Java netCDF libraries. The atmospheric levels (z-levels) of the data are represented by the bands of the GeoTiff data.

Using GeoTiff data allow the team to utilise MapServer and in particular the MapServer WCS support for subsetting on bands and the temporal dimension. Using a naming convention for experiments and parameter names allows the support for filtering on parameters. Although this isn't ideal, it works. The team is however looking to develop a true N-D WCS if funding permits.

• Specific plans
  • Support temporal, z-level, and parameter subsetting on data through WCS
  • Release operational WCS in September 2005

There is a tutorial on http://ms.gis.umn.edu:8081/ms_plone/docs/howto/WCSServerFormatHowTo/howto_view describing how to serve temporal ranges of netCDF data using MapServer WCS.

 

JPL

Oceanographic datasets will be distributed via the POET data server at PO.DAAC

GMU

A NWGISS WCS server which can generate netCDF CF-1 output is now available at:

http://data.laits.gmu.edu/cgi-bin/tsunami/wcs100

Please try it and give us some feedback (to Yang Liu : yliu5@gmu.edu). Following are several examples:

getCapabilities

http://data.laits.gmu.edu/cgi-bin/tsunami/wcs100?version=1.0.0&service=WCS&request=getCapabilities

describeCoverage

http://data.laits.gmu.edu/cgi-bin/tsunami/wcs100?version=1.0.0&service=WCS&request=describeCoverage

getCoverage (9.6M bytes)

http://data.laits.gmu.edu/cgi-bin/tsunami/wcs100?version=1.0.0&service=WCS&request=getCoverage&
coverage=/data4/TSUNAMI/L71131056_05620041229_B10.hdf:Grid:L71131056_05620041229_B10:
L71131056_05620041229_B10&crs=EPSG:32647&bbox=23400,534000,267000,747000&resx=300&
resy=300&format=netCDF

The link above will take a while to generate the output.

If your network is not very fast, you may want to try a smaller example:

getCoverage (2.4M bytes)

http://data.laits.gmu.edu/cgi-bin/tsunami/wcs100?version=1.0.0&service=WCS&request=getCoverage&
coverage=/data4/TSUNAMI/L71131056_05620041229_B10.hdf:Grid:L71131056_05620041229_B10:
L71131056_05620041229_B10&crs=EPSG:32647&bbox=23400,534000,267000,747000&resx=600&
resy=600&format=netCDF

Feel free to play with other parameter values, but please don't try too large width/height or too small resx/resy, which will make the result very huge and our server may crash because of insufficient memory.

NERC (Natural Environment Research Council)

NERC is the UK Natural Environment Research Council, and is undertaking various threads of WCS-related activity.

NERC DataGrid has implemented a basic Mapserver WCS, with NetCDF converted to GMT grid format prior to serving. Tested successfully with RSI and CadCorp's WCS clients. Available at:

http://glue.badc.rl.ac.uk/cgi-bin/mapserv?map=/var/www/html/jiscInterop/nerc.map

In collaboration with the Tasmanian Partnership for Advanced Computing, has developed a standalone WCS server providing arbitrary slicing through four dimensional data volumes. The architecture is intended to allow various back-end data sources to be plugged, with the initial implementation supporting OPeNDAP; thus the prototype server is in effect an OPeNDAP->WCS bridge. The prototype is available at (and includes a static list of sample URLs):

http://glue.badc.rl.ac.uk/cgi-bin/TPAC/WCS

Also developing WCS interfaces to netCDF as part of a project to deliver environmental web services (www.dews.org.uk).

The NERC GALEON report provides greater detail.

Client implementations

Unidata

IDV

U of Florence and IMAA-CNR

Java/Webstart implementation of a WCS 1.0 client (WCS Browser Lite), mainly to test our WCS-G server prototype (see above). WCS Browser Lite features include: SOAP and HTTP bindings support, netCDF and ncML-GML format support.

• WCS Browser Lite
  • Known bugs: exceptions in GUI controls for time and range set subsetting; grid/rectified grid subsetting is not implemented; envelope subsetting on 3rd dimension (and over) is not implemented; sporadic verification errors on Webstart launch
  • URL: http://zeus.pin.unifi.it/projects/wcsClientLite/
  • Roadmap:
    • TBD (M2): netCDF coverage preview

RSI

GDET is a pure IDL Web Coverage Service client, support the operations getCapabilities, describeCoverage, and getCoverage. It supports the subsetting of data via the temporal, band, and parameter axes. In addition the returned data (if temporal) can be animated, and scientific analyses (profiles, interpolation functions) can be performed.

• Specific plans
  • To release a first version of the client by September 2005

RSI Boulder (Louie Genduso) is testing a second RSI WCS client. A draft report of experiences with that client is given at: http://www.unidata.ucar.edu/projects/THREDDS/GALEON/Reports/GALEONReportRSIboulder.htm

RSI UK GALEON Final Report

http://217.37.214.210/GALEON/GALEONReport.doc

Cadcorp

SIS (Spatial Information System) is a WCS client, using GDAL (www.gdal.org) for processing returned netCDF and other raster results.

• July 2005: Client tested with rudimentary WCS netCDF services
• February 2006: Further client testing: CadcorpReport

 

GMU

test client

UAH

test client

Northwestern

MyWorld

Participating initially as an observer, but would like to implement Java MyWorld as WCS client if resources can be found.

ESRI

 

Texas A&M University

Currently "Observer" while waiting for our legal folk to work their way through the paperwork.

Mapserver as test client

We will implement a WCS server using Mapserver shortly, probably serving Level II radar data from a selected group of sites.

THREDDS server with SCOOP data subset (Level II radar, TAMU MM5 products) soon.

 

Washington University (DataFed)

DataFed? ( www.datafed.net) is a web service based infrastructure that is used primarily to share air quality related data. It includes a viewer for
exploring data in spatial and temporal displays. DataFed is participating in GALEON as a WCS client and has been tested with WCS coverages from Unidata, U. Florence, and GMU.

Test results are posted on this Wiki (http://galeon-wcs.jot.com/WikiHome/Client%20Server%20Interaction%20Status%20Table )
and on the DataFed site (http://www.datafed.net/DataLinks/OGC/DataFed_WCSClient.htm).

An example of how the THREDDS WCS for CDM gridded data "look" in DataFed:
http://webapps.datafed.net/dvoy_services/datafed.aspx?dataset_abbr=THREDDS_CDM

The gridded data can be explored within the DataFed Viewer by changing the date/time with the time controls or selecting a different coverage
from a drop-down list.

Please let us know if you have a WCS you would like to test in DataFed or if you have any comments regarding the DataFed client.

Dec 5, 2006, RBH: We have prepared a progress report on the GALEON WCS server testing using the DataFed? federated data system client. http://capita.wustl.edu/capita/researchareas/GALEON/051128_GALEON_IE_Rept.htm

We are very ecouraged but see a long road to full WCS interoperability: full 4-5D data testing; loose coupling of THREDDS-WCS datasets (automatic registration); reprojection services

Second GALEON Progress Report, R. Husar, Feb 7, 2006.

http://capita.wustl.edu/capita/researchareas/GALEON/Reports/060206/060206_GALEON_IE_DataFed.htm

This second progress report contains the findings on interoperability between the GALEON WCS test servers and the DataFed WCS client and the development of a DataFed WCS server. The first DataFed report on the GALEON wiki (Dec 5, 2005) contained the early IE results.

The DataFed WCS client can successfully access the WCS data sets from the U. Florence and from the UNIDATA THREDDS WCS test servers. The spatial queries to both servers yield the expected return as spatial grids. Within DataFed, the received WCS (NetCDF) data from both servers, are easily transformed into data views and used in distributed web applications by chaining appropriate web services.

As an experiment, we have created a WCS test server to serve a wide variety of point, grid an image coverage data. Our goal was to evaluate the WCS protocol for accessing coverages of different types arising from a variety Earth observation and modeling systems. We are most encouraged by the possibilities but recognize that considerable work is to be done on the extending the describeCoverage schema to accommodate these coverage types and the returned data types (e.g. a Table) to accommodate these additional coverage types. The discussion on the possible WCS extensions is given in a separate report to the GALEON IE group. Due to the late completion of the DataFed WCS server, it has not been tested for interoperability by remote clients.

Substantial hurdles are still ahead before we reach a 'snap'-like WCS client-server interoperability including:

- the temporal aspects of the WCS queries needs more work both on server and client side
- automatic (loosely coupled) registration of WCS services needs work (using ncML-GML ?)
- geo-re-projection services (preferably web services) to handle the homogenization of the mired projection used by the different data providers.

We recognize that many of the current inadequacies are being addressed in other interoperability efforts and possibly by the continuation of this IE. We are looking forward enhancing our interaction with the group as well as with individuals in this group while striving toward better interoperability among our data systems.

 

 

Attachment	Size
WCS-result-spec_v2.doc	72.5 KB
WCS-result-spec_v3.doc	65.5 KB
WCS-netCDFformat-1.0.doc	38 KB
Final+Report+_IMAA-UNIFI_+0.3.pdf	248.97 KB
WCS-result-spec_v5.doc	71 KB
galeon-final-iub.pdf	316.54 KB
OGC+specs+analysis+_IMAA-UNIFI_+0.3.pdf	108.93 KB
GALEONReportRSIboulder.pdf	9.42 KB
GALEON-NERC-report.doc	348 KB
DataFed-GALEON_Rep060215.pdf	1.41 MB
GALEONReportRSIdraft.doc	1.62 MB
wcs-2005-11-09.ppt	4.09 MB