While the theme of Safe Software's FME User Conference,
held March 6-7 in Vancouver, was New
Dimensions in Spatial ETL (extract, transform, load), I'd say it
was a GIS conference that was not so much about GIS. The challenges are
geographic, to be sure, but the conference was really about integration
- integration of data, platforms, services, levels of government, etc.
While GIS and related technologies were in the picture, the focus was
the "glue" that enables integration. Below are the key announcements
and takeaways from the two-day event.
FME 2008
As is always true in an update of Feature Manipulation Engine (FME),
Safe Software's workhorse for translating data between formats and
restructuring them to meet specific needs, there's a long list of
enhancements. Among them is a new interface tool that allows drag and
drop of transformers. That tool alone, said co-founders Don Murray and
Dale Lutz, will save users "1000s of clicks." The pair tag teamed each
day's product-focused plenary sessions with an amusing banter that
reminded me of twin brothers trying to entertain the family during the
holidays. (They are a riot.)
Some of the new or enhanced formats supported: PDF (3D only), AutoCAD
Map 3D (with object data), CityGML, Golden Software's Surfer, GeoJSON,
IFC (for BIM), LandXML (from a 1998 request!), SQL Server Spatial 2008
(not yet available) and read/write support for GeoConcept. If you are
keeping score, the number of formats FME supports has grown from nine
in 1997 to 220 in 2008. One subtlety of FME that should be emphasized
and may not be widely understood: FME can be consumed as a Feature Data
Object (FDO), the open source "connection" interface pioneered by
Autodesk. Said another way, if your software supports FDO, you can,
with an FME license, gain access to those 220 formats. Which software
supports FDO? I asked around and confirmed just three products: AutoCAD
Map 3D, MapGuide (Enterprise/Open Source) and 1Spatial's Radius Studio.
I hope to see more in the future; FDO is still a relatively new option
for developers.
New transformers (software calls that transform data) tie back to key
customer demands such as 3D (an extruder), geometry management
(triangulation of complex polygons), simplifying data for systems that
don't understand complex types (a donut bridge builder and constructive
solid geometry "flattener"), attribute maintenance (the ability to
retain attributes of simple geometry when they are combined into more
complex features), and Web portal support (tile creation for Microsoft
Virtual Earth and Google Earth). The flashiest demos of the new
transformers and Web services support tapped into existing Web
services. One demo grabbed local data and put them into KML for
visualization in Google Earth. Clicking on a location on the map
provided a link that called a Web Service to populate a pie chart of
the fraction of conference goers from that location. A second demo
called a pay service from White Star (the oil/gas data providers) that
provides locations of wells, and popped them up on a Web-based map.
FME Desktop should be available for download this week and available in
packaged form a few weeks later.
FME Server
If you take the power of FME Desktop and put it on a server you get
That simple statement doesn't reveal the power of this new product. In
simple terms, you can put a workspace (a saved workflow of transformers
and conversions) up on server for use. How do you do that? You develop
the workspace in FME's desktop graphic development tool, Workbench, and
publish it to the Web with the push of a button. ("That's easy!" was
the running in-joke during that presentation.)
How do you use the saved workspace once it's up there? You can call the
workspace to run (1) via a URL with appropriate parameters, (2) via its
own API (soon to be SOAP enabled), or (3) from another application such
as another ETL solution. The resulting data, the output of the
workspace, can be (1) streamed via a MIME type (a PDF for example), (2)
zipped and delivered, (3) saved out to a file to be used later, or (4)
in a future release, served via OGC's Web Map Service or Web Feature
Service specification servlets.
FME Server is aimed at dynamic data, data that changes and needs to be
served, reformatted or re-checked regularly. FME Server provides the
infrastructure to ensure that those who need the data receive them in
exactly the format needed, without necessarily needing to have, or know
how to run, FME.
Safe offers three key use cases for FME Server:
- Distributing dynamic data - allowing users to get
at data formatted as they need, whenever they need, without a copy of
FME or knowledge of how to use it - Off-loading computationally complex and
time-consuming tasks - multiple servers can help break up and speed up
big jobs - Integration with workflow and other systems - for
example, Informatica or IBM WebSphere or ArcGIS server
One of the most powerful demonstrations of how
FME Server might be used had a user key in a city of interest, use that
to find traffic data from the Web (perhaps in GeoRSS), symbolize the
data, and output them in KML to be visualized in Google Earth. The demo
made me think of a "geosavvy version of Yahoo! Pipes (I
about it last year), Yahoo's tool for non-programmers to link
together Web services for their specific needs.
URL to Pipes Article
That, to me, is the power here. While there have been discussions of
standards enabling all sorts of mashups of geodata, non-programmers
have had limited access to any definitive tool for their creation. Now,
with FME Server to do the hard work of understanding the different
formats, querying, symbolizing and combining data in the appropriate
output form, that all changes. One big plus: all of the various formats
are treated the same.
FME Server should be shipping in late March or early April. Beta 3
should be out this week. FME Server does not include a copy of Desktop,
though most users will want to use it as the preferred "authoring
tool." Further, FME Server users will need to have a Web server and
database up and running to use FME Server. Finally, SpatialDirect,
Safe's "clip, zip and ship" solution for data distribution in many
formats, has been retired as a standalone product. It's now an optional
add-on to FME Server. Existing SpatialDirect users will be upgraded to
that version of Server without extra fees. Several SpatialDirect users
noted their excitement about this change. They will finally be able to
use Workbench as an authoring tool, something not previously possible.
Questions, Questions
To my surprise, at this conference about FME, the majority of the
questions to presenters were not about FME! The Log-In-Project, a joint
effort of several ESRI European offices and con terra, presented by
Christian Heisig of con terra, aims to provide a secure Web service
interface to geospatial data, including monitoring of services' uptime,
and uploading and downloading of data of different types. The questions
were mostly about the security and monitoring parts.
Wesley Hardin and Jamie Katz from Burns and McDonnell described an
in-house developed Geospatial Dashboard for a central Connecticut
transmission line project. It uses FME to standardize data and make
them available to all stakeholders and contractors via Google Earth Pro
(soon Google Earth Enterprise). It costs about $150,000 and took three
people three months to develop. The questions were about data quality
and linking to other key systems (Primavera Expedition among others)
and how to keep the datasets small (some tricks - remove symbology
information from the streamed data and lop off a few unneeded decimal
points).
Frank Orr from CH2M Hill described a Homeland Security-funded data
integration project for the Colorado North Central Homeland Security
Region. The project goes live in May and requires 10 counties to be
able to upload and download eight layers of data for hazard planning
needs. That meant "matching" data models from each county on the way in
to the server and mapping the "other" counties' data to that same model
on the way down. The questions were about metadata, security and data
management issues such as edge matching.
What does this tell me? The technology most of these users interact
with day in and day out is sort of invisible to them. They worry more
about what's around it. I think that's how software is supposed to
work.
FME as the Backend to the App
If there's a complaint about the new visualization-focused Web tools
(Google Earth, OpenLayers, MapServer and others) or even some of the
lightweight desktop GIS visualization packages (QGIS comes to mind),
it's that they need more data crunching and analytical power. Some put
ArcGIS Server or GRASS in the back end to provide that functionality.
Many at the FME User Conference put FME back there. The City of Calgary
did. Staffers from that city told me why: "It's easy and fast." They
had taken one of the three advanced FME training courses the day
before. Jason Birch of Nanaimo, British Colombia explained that when he
joined the city's GIS group, it didn't have ArcGIS. So he had to use
the tool at hand, FME, to do some light analytics and data prep. He and
his colleague Matthew Dunstan ran down a long list of projects where
FME is their "leatherman tool" for GIS. Now with FME Server, I expect
we'll see even more FME use in this vein. I'll offer that from what I
saw in user presentations, FME is used maybe 20 percent of the time as
a data format converter and 80 percent of the time as an analytical
toolbox/data restructuring tool.
FME in All Sorts of Shops
As the "Switzerland" of the GIS world, FME pops up in all kinds of
implementations. CH2M Hill, one of three Google Earth partners, uses it
to enable clients to see their own data on Virtual Earth, Google Earth
and Google Fusion. Specifically, users want to overlay their own large
raster imagery and ArcSDE and Oracle Spatial data on these platforms
and FME paves the way. Jubal Harpster of CH2M Hill brought a few
servers and clients to show how it's done. The City of Surrey, British
Columbia uses ArcIMS and GeoCortex IMF to power its Web mapping. FME
provided tools to set up a "pay for data" portal. While that's a
technical challenge, it seems determining the pricing was at least as
difficult!
Live Demos
After seeing some "video recorded" demos recently, I was very happy to
see lots of Safe's demos run live and, frankly, not work! Sometimes it
was Vista weirdness, sometimes it was human error, sometimes it was
running with the code that was compiled the night before... No one in
the audience seemed to care. In fact, one attendee from Michael Baker
noted these "failures" as one of the best parts of the event. Another
attendee commented to me on the same topic, "They are in this with us;
that gives me great confidence things will get fixed!"
Whatever happened to...
Recall all those diagrams from recent years with a centralized database
in the center and many different thin and thick clients around the
outside? I used to draw them myself to highlight how Oracle Spatial
might be in the center, with AutoCAD, ArcGIS and other clients around
the periphery, for example. I didn't see those diagrams at FME UC. What
I did see were diagrams with a centralized data store (or stores) in
the center, then a layer of FME, and then those same clients. Why? The
pure database vision does not take into account the different data
structures or the different data needs of, say, a planner, an engineer
or the assessor. Those end users want - no, insist - on seeing data in
the way they understand them, on the client with which they are
familiar. While it's possible to do that with many databases and
clients, it's quite a lot of work. FME provides a single environment
that can keep everyone happy. I expect to see FME popping up in the
stack, perhaps on a server, more and more often, even as (perhaps
because?) new spatial databases appear, such as Microsoft SQL Server
2008.
src="http://www.directionsmag.com/images/newsletter/2008/03_week2/fme_1.jpg" border="0" height="233" width="350"> |
The Joy of a 130-Person Conference
When we met in a round room at Simon Fraser University on the opening
morning of the event, I knew it was going to be a good few days. You
could see everyone in the room. When Murray or Lutz alluded to, or
asked a question of, a colleague, they were in the room. If you are
investigating, or even vaguely interested in, a technology that has a
conference with fewer than 500 attendees, spend the money to attend.
You'll learn more than at any larger event and not have to work half as
hard to do so.
Users Attend User Conference
It was great that so many users attended the event. They presented all
of the papers in all the sessions, while Safe staff acted as "doctors
in a Doctors Office and instructors in typically filled-to-overflowing,
hands-on workshops. Some conferences seem overloaded with
vendor/partner/reseller presentations. That was not a problem here. Who
was missing? Some of Safe's software partners. I was pleased to speak
with, and be amused and enlightened by, staff from Microsoft and
MapInfo. I know that ESRI staffers were on hand, too, as was a local
rep standing in for 1Spatial. I was disappointed not to see
representatives from other players.
src="http://www.directionsmag.com/images/newsletter/2008/03_week2/fme_2.jpg" border="0" height="179" width="350"> |
Fun and Games
After the buzz last year about "FME Idol, and not having seen the TV
show on which it was based, I was certainly excited to see it live.
Alas, watching a handful of top coders try to decipher a challenge live
(when you don't know the challenge) is not exciting. I did appreciate
that Safe staffers had put together some fun "audition videos" to fill
the time and that some demos missed in a morning session could be
revisited. The coolest part from my perspective, Safe staffers were
competing alongside the players and several conquered the challenge!
Alas, none of the potential Idols fully completed it within the
allotted time. Don't get me wrong, I'm in awe of users with the guts to
get up there and show off their skills. The closest to completion took
home a topographically correct "bowl" featuring the local geography,
while the runners-up pocketed iPhone Touches. After a festive evening
meal overlooking the water, attendees were treated to an improv show.
The actors had clearly been briefed on what Safe does. (Imagine trying
to explain that to actors!) To my surprise, they did a great job
injecting "extract, transform and load" and other subtleties into their
stories. I'd not laughed so hard about geospatial technology in a long
time!
All materials from the general sessions, hands-on workshops and
breakout sessions are now available at:
www.safe.com/fmeuc/presentations. Video footage of the general
sessions, keynote presentation by Peter Batty, as well as a few of the
breakout sessions will be posted on this site within the next two
weeks.
[Disclosure: Safe Software covered my transportation and lodging for
the conference. I received, as did other attendees, a Safe Software
jacket and "tuque (winter hat).]
No comments:
Post a Comment