Thursday, Sept. 21, 2017

Introduction

10:00 – 10:30: Welcome

Presenter:  Edzer Pebesma – ifgi, Albert Remke – 52°North / Esri Germany / ifgi, Daniela Wingert – Esri Germany

10:30 – 12:00: Big data concepts and technologies – Overview

Presenter: Arne de Wall – 52°North, Marius Appel – ifgi, Thomas Paschke – Esri Germany

This presentation provides a general introduction and an overview of today’s big data concepts and technologies. After a short discussion of the big data landscape, concepts such as stream and batch processing and common big data architectures are presented and specific challenges for geographic data are discussed. The presentation will emphasize widely used technologies including Hadoop, Spark, and array databases and give an outlook on their application on geographic data and how cloud-based solutions make these technologies accessible to users.

Lunch Break

13:30 – 15:00: Technologies & Interfaces

13:30 – 14:00: ArcGIS Real-Time & Big Data Solutions

Presenter:  Thomas Paschke – Esri Germany

This presentation will give an overview of Esri’s Real-Time & Big Data solutions. The currently available products as well as research & development efforts including the respective technology used will be discussed.

14:00 – 14:30: R / Python and Big Data, openEO

Presenter: Edzer Pebesma – ifgi

R itself has a number of extension packages that make it easier to work with large amounts of data, and in particular large amounts of spatial data (images, but also features). We will discuss some of these packages, illustrate their use, and discuss recent and planned developments for the analysis of large geospatial data.

14:30 – 15:00: Scalable Earth observation analytics with SciDB

Presenter: Marius Appel – ifgi

This presentation will introduce how the data management and analytical system SciDB and its data representation as multidimensional arrays can be used to perform complex analytics on Earth observation datasets. The first part will discuss SciDB’s basic concepts such as chunking and sparse storage before some extensions that facilitate the conversionfrom file-based scenes to multidimensional arrays with the geographic data abstraction library (GDAL) will be presented. A case study for land use change analysis based on Landsat 7 imagery will then demonstrate how user-defined functions can be scaled by streaming. The presentation will close with a discussion of the limitations and an outlook for which applications SciDB might provide a reasonable platform.

15:00 – 15:30: Flash Talks

Coffee Break

16:00 – 18:00: Application areas

16:00 – 16:30: Big Spatial Data in Agriculture

Presenter:  T. Steckel, H. Warkentin – CLAAS E-Systems

Spatial data are an important source of information in agriculture and thus not new. Up to now, most Farm Management-Systems incorporate some capabilities for spatial and temporal analytics. Increased availability of sensor data and improved wireless communication allow for deeper analysis of machines and processes. Well known IT-Systems encounter their limits. This presentation provides some insight into new environments with particular focus on GIS-aspects.

16:30 – 17:00: A Query Language for Handling Big Observation Data Sets in the Sensor Web

Presenter: Christian Autermann – 52°North / Alfred Wegener Institute

The Sensor Web provides a framework for the standardized Web-based sharing of environmental observations and sensor metadata. While the issue of varying data formats and protocols is addressed by these standards, the fast growing size of observational data is imposing new challenges for the application of these standards. Most solutions for handling big observational datasets currently focus on remote sensing applications, while big in-situ datasets relying on vector features still lack a solid approach. Conventional Sensor Web technologies may not be adequate, as the sheer size of the data transmitted and the amount of metadata accumulated may render traditional OGC Sensor Observation Services (SOS) unusable. Besides novel approaches to store and process observation data in place, e.g. by harnessing big data technologies from mainstream IT, the access layer has to be amended to utilize and integrate these large observational data archives into applications and to enable analysis.

17:00 – 17:30: ArcGIS Big Data Analytics use cases

PresenterThomas Paschke – Esri Germany

This presentation will highlight some use cases where Esri’s Real-Time & Big Data solutions were used to solve Big Data analysis related problems.

17:30 – 18:00: Processing and analysis of Earth Observation data

PresenterCarsten Brockmann – Brockmann Consult

Satellite remote sensing is producing large amount of Earth Observation (EO) raw data. Only the Sentinels of the European Space Agency are generating more than 6 TB/day new data in 2017, increasing over the coming years. A special adaptation of Hadoop for massive parallel processing of EO data has been implemented in the Calvalus processing system. Calvalus is part of several operational processing systems including the Copernicus Data and Exploitation Platform – Deutschland (CODE-DE). Closely linked with Calvalus is SNAP, a Java based toolbox for processing of EO data. Together they allow in-memory chaining of complex processing graphs and exploiting the map-reduce programming model. The production of Essential Climate Variables is chosen as an example to demonstrate the application of Calvalus on full mission data, the analysis with the python based CATE toolbox, and the combination with other earth system data in the Earth System Datacube.

Dinner (19:30)

Friday, Sept. 22, 2017

9:00 – 13:00: Parallel tutorials & hands-on sessions

ArcGIS & Real Time processing

Presenter:  Thomas Paschke – Esri Germany

In the morning user’s will learn how to set up and configure the current ArcGIS Real-Time & Big Data technology stack and gain familiarity with different aspects of the products. They will learn, how to ingest and analyze data feeds in real-time, store large amounts of this observation data and perform big data analysis on them.

R-ArcGIS-Bridge

Presenter:  Edzer Pebesma – ifgi, Melanie Brandmeier – Esri Germany, Benedikt Graeler – 52°North

ArcGIS offers many spatial analysis tools out of the box. By combining ArcGIS with the full capabilities of the open source programming language R, users are able to work with a rich toolset for geostatistical computing. The R-ArcGIS-bridge allows to integrate R into the toolbox to perform statistical analysis on spatial data. We will showcase the entire workflow from installing the R-ArcGIS-bridge, through the design of R scripts and their integration into a tool box and the execution of these tools. Participants will have the opportunity to follow the showcases in a hands-on manner.

System Requirements: For the practical work with the technology in this hands-on session, please bring your own laptop and make sure that you have

  • administrator rights to install software,
  • access to ArcGIS Pro,
  • a local installation of R including the packages sp, gstat, raster, shiny, rgdal, spatstat, maptools, mclust, and
  • a local installation of RStudio Desktop
Processing Earth observation time series with SciDB, R, and Python

Presenter:  Marius Appel – ifgi

In this session, participants will learn how to use SciDB in combination with R and Python to process time series of Earth observation imagery. Participants will work on a prepared dataset and a provided example script on relatively simple operations such as compositing cloud-free images, deriving vegetation indices. An in-depth explanation of SciDB clients in python and the R packages scidb and scidbst is given at the beginning of the session.

13:00: Closing session

14:00 – 17:00: Hackathon & open space for exploring methods & technologies

We prepare some challenges, data and the infrastructure. You can use the sessions to get some hands-on experience, or try to explore the technologies with your own data.

ArcGIS Real-Time & Big Data analysis

Presenter:  Thomas Paschke – Esri Germany

The experience gained in the morning should provide the foundation and inspire for the Hackathon in the afternoon, where Real-Time and Big Data analysis use cases, based on the user’s own data (bring your own data) or provided sample data, can be explored.

R-ArcGIS-Bridge

Presenter:  Edzer Pebesma – ifgi, Melanie Brandmeier – Esri Germany, Benedikt Graeler – 52°North

We will provide the opportunity to develop and run your R code in ArcGIS along challenges that we propose. Participants are also invited to bring their own data sets and analysis scripts that they wish in integrate into ArcGIS. We will provide step-wise solutions and guidance throughout the afternoon.

System Requirements: For the practical work with the technology in this hands-on session, please bring your own laptop and make sure that you have

  • administrator rights to install software,
  • access to ArcGIS Pro,
  • a local installation of R including the packages sp, gstat, raster, shiny, rgdal, spatstat, maptools, mclust, and
  • a local installation of RStudio Desktop
Processing Earth observation time series with SciDB, R, and Python

Presenter:  Marius Appel – ifgi

Based on the morning session, we will look at further use cases in Earth observation time series processing from the participants and discuss how these can be implemented with SciDB, R, and Python. Following the discussion, we will start with the implementation of one selected use case.

Drinks and open end