![](\"geoextent_extraction_example.png\")
\n",
"\n",
"**Image 1. Example of bounding boxes extracted by geoextent.(Left) Correct extraction, (Center) partially correct extraction and (Right) erroneous extraction. (Classification after human verification)**\n",
"\n",
"As a conclusion, we observe that the extraction of geospatial information from records in a general-purpose research data repository could provide geospatial metadata to aid data discovery. Our approach encountered potential geospatial information in a relevant percentage of repositories of different characteristics and successfully extracted geospatial information from various file types. We propose that including geoextent into the pipeline of data curation, e.g., by proposing a bounding box based on the data during record creation, could help researchers and data repositories to improve the quality of the record metadata, but also in terms of data understandability, e.g., by encouraging non-ambiguous formats.\n",
"\n",
"## Funding\n",
"\n",
"- Award1 = {\"agency\": \"German Research Foundation (DFG)\", \"award_code\": \"PE1632/17-1\", \"award_URL\": 'https://gepris.dfg.de/gepris/projekt/415851837'}\n",
"\n",
"## Keywords\n",
"\n",
"keywords=[\"geospatial\", \"discovery\", \"metadata\", \"repositories\", \"data sharing\"]\n",
"\n",
"## Citation\n",
"\n",
"Garzón, Sebastian and Nüst, Daniel, 2021. Exploring Research Data Repositories with geoextent. Accessed 2021-05-14 at https://github.com/o2r-project/geoextent/tree/master/showcase.\n",
"\n",
"## Suggested next steps\n",
"\n",
"First, the survey of Zenodo records can be extended to include records based on more search terms - or even all records - and to include larger records.\n",
"Second, the record retrieval features of geoextent can be extended to include additional research data repositories.\n",
"These can be general-purpose ones, e.g., [Figshare](https://figshare.com/) or [OSF](https://osf.io/), but also specialised repositories for geospatial data, e.g., [Pangaea](https://www.pangaea.de/), or [GFZ Data Services](https://dataservices.gfz-potsdam.de/). In the case of the specialised repositories, the extracted metadata should be compared with the metadata of the platform.\n",
"Third, the development of geoextent will be continued, e.g., to support more data types, to support more output options for integration into other tools, or to communicate progress to users or including tools.\n",
"Especially the support of more data formats and increased stability of the library could make it possible to integrate it into Open Source data repository software, such as [InvenioRDM](https://inveniosoftware.org/products/rdm/) (the base software of Zenodo), and thereby turn geospatial and temporal metadata into regular record-level metadata which can be validated by authors on the creation of new records, and which can aide interdisciplinary collaboration through novel connections between datasets.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Setup\n",
"\n",
"## Library import\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"# Import geoextent\n",
"import geoextent.lib.extent as geoextent\n",
"import geoextent.lib.extent as geoextent_help\n",
"from geoextent.__init__ import __version__ as geoextent_version\n",
"# Data manipulation\n",
"import requests\n",
"import json\n",
"from shapely import wkt\n",
"import pandas as pd\n",
"import geopandas as gpd\n",
"import numpy as np\n",
"\n",
"# Logging\n",
"import logging\n",
"\n",
"# Measure time and sleep function\n",
"import time\n",
"\n",
"# Visualisation (graphs, maps, tables)\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib as mpl\n",
"import matplotlib.ticker as mtick\n",
"import folium\n",
"from folium import plugins\n",
"from folium import FeatureGroup, LayerControl\n",
"from folium.features import GeoJsonPopup "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Local library import"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"# Include local library paths\n",
"import sys\n",
"\n",
"# Import help functions for zenodo API\n",
"sys.path.append('help_functions') \n",
"# Import local libraries\n",
"import help_functions.request_zenodo_api as zenodo_api"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Parameter definitions"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [],
"source": [
"# Search term for Zenodo API query\n",
"SEARCH_TERM = \"geo&geology\"\n",
"# Maximum size of Zenodo record to extract (MB)\n",
"MAXIMUM_RECORD_SIZE_MB = 500\n",
"# CSV output file for Zenodo API statistics\n",
"OUTPUT_API_STATISTICS = \"zenodo_api_statistics.csv\"\n",
"# CSV output file for search results (Zenodo record metadata)\n",
"OUTPUT_API = \"zenodo_api_extraction_geo.csv\"\n",
"# CSV output file for geoextent extraction results \n",
"OUTPUT_GEOEXTENT = \"geoextent_extraction_geo.gpkg\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Data import\n",
"\n",
"For the introductory usage examples, test data files are included in the library and available in the notebook's repository in the [tests/testdata](https://github.com/o2r-project/geoextent/tree/master/tests/testdata) directory, where sources of test data are documented as well.\n",
"For the main analysis, data retrieval is part of the geoxtent library and therefore included in the below processing.\n",
"The Zenodo record identifiers, which can be used to construct the DOI, are stored in the output dataset published next to this notebook.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data processing and analysis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## geoextent usage\n",
"\n",
"### Supported file types\n",
"geoextent supports a subset of the file formats supported by GDAL and uses text-scraping techniques to extract the geospatial and temporal features of common file types storing geospatial information.\n",
"This subset includes the most commonly used formats and ensures stable handling of edge cases and possible errors.\n",
"The supported file formats include vector (e.g., [GeoPackage](https://en.wikipedia.org/wiki/GeoPackage), [Shapefile](https://en.wikipedia.org/wiki/Shapefile), [GeoJSON](https://geojson.org/)) and raster (e.g., [GeoTIFF](https://en.wikipedia.org/wiki/GeoTIFF), [JPEG 2000](https://en.wikipedia.org/wiki/JPEG_2000#GML_JP2_georeferencing)) file types.\n",
"The user can extract the spatial extent, the so-called bounding box (parameter `bbox`), and/or temporal extent (parameter `tbox`) of a file or set of files.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Individual files\n",
"\n",
"geoextent can be run on a single data file using the Python API, as shown below.\n",
"The output includes the spatial and temporal extents.\n",
"The spatial extent is always provided as in the [WGS84](https://en.wikipedia.org/wiki/World_Geodetic_System) coordinate reference system ([CRS](https://en.wikipedia.org/wiki/Spatial_reference_system)), commonly known through its usage in GPS, which is sufficiently precise for the use case of dataset discovery. If the dataset is provided in a different CRS, the bounding box is reprojected using GDAL."
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"../tests/testdata/shapefile/ifgi_denkpause.shp \n",
" bbox: [7.594978277801928, 51.96852473231792, 7.5957650477781415, 51.969118924937405] \n",
" 4326 EPSG \n",
" tbox: ['2021-01-01', '2021-01-01']\n"
]
}
],
"source": [
"# File of interest\n",
"local_filepath = \"../tests/testdata/shapefile/ifgi_denkpause.shp\"\n",
"# Geoextent extraction\n",
"geoextent_file = geoextent.fromFile(filepath = local_filepath, bbox = True, tbox = True)\n",
"\n",
"# Print output\n",
"print(local_filepath, \"\\n\", \"bbox:\", geoextent_file['bbox'],\"\\n\",geoextent_file['crs'], \"EPSG \\n\",\n",
" \"tbox:\",geoextent_file['tbox'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Multiple files\n",
"\n",
"When provided a directory, geoextent by default returns the union of the spatial and temporal extent of all supported files."
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"../tests/testdata/folders/folder_two_files \n",
" bbox: [2.052333387639205, 41.31703852240476, 7.647256851196289, 51.974624029877454] \n",
" 4326 EPSG \n",
" tbox: ['2018-11-14', '2019-09-11']\n"
]
}
],
"source": [
"# Directory of interest\n",
"local_directory_path = \"../tests/testdata/folders/folder_two_files\"\n",
"# Geoextent extraction\n",
"geoextent_directory = geoextent.fromDirectory(path = local_directory_path, bbox = True, tbox = True, details = True)\n",
"\n",
"# Print output\n",
"print(local_directory_path, \"\\n\", \"bbox:\", geoextent_directory['bbox'],\"\\n\", geoextent_directory['crs'], \"EPSG \\n\",\n",
" \"tbox:\",geoextent_directory['tbox'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The previous result is the combination of two files: **muenster_ring_zeit.geojson** and \n",
"**districtes.geojson**, so geoextent also stores (If parameter `details ` = `True `) the details from the files used to compute the final bbox and tbox. For example:"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'format': 'geojson',\n",
" 'geoextent_handler': 'handleVector',\n",
" 'bbox': [7.6016807556152335,\n",
" 51.94881477206191,\n",
" 7.647256851196289,\n",
" 51.974624029877454],\n",
" 'crs': '4326',\n",
" 'tbox': ['2018-11-14', '2018-11-14']}"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"geoextent_directory['details']['muenster_ring_zeit.geojson']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data repositories\n",
"\n",
"Geoextent aims to extract the bounding box (bbox) and temporal box (tbox) from data repositories. The first data repository implemented is Zenodo, and the extraction is based on either a Zenodo Record URL (e.g., `https://zenodo.org/record/820562`) or the URL of a DOI (e.g., `https://doi.org/10.5281/zenodo.820562`)."
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://doi.org/10.5281/zenodo.820562 \n",
" bbox: [96.21146318274846, 25.558346194400002, 96.35495081696702, 25.632931128800003] \n",
"\n",
"https://zenodo.org/record/820562 \n",
" bbox: [96.21146318274846, 25.558346194400002, 96.35495081696702, 25.632931128800003]\n"
]
}
],
"source": [
"# Zenodo Record of interest\n",
"url_zenodo = \"https://zenodo.org/record/820562\"\n",
"url_doi = \"https://doi.org/10.5281/zenodo.820562\"\n",
"\n",
"# Geoextent extraction\n",
"geoextent_url_doi = geoextent.from_repository(url_doi, bbox = True)\n",
"geoextent_zenodo_record = geoextent.from_repository(url_zenodo, bbox = True)\n",
"\n",
"# Print output\n",
"print(url_doi,\"\\n\", \"bbox:\", geoextent_zenodo_record['bbox'],\"\\n\")\n",
"print(url_zenodo, \"\\n\", \"bbox:\", geoextent_zenodo_record['bbox'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Command-line interface"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Geoextent's API functionalities are also accessible through a command-line interface (CLI) for files, folders, and data repositories. In this scenario, configuration options for the extraction are provided by flags. For example, `-b` indicates a bounding box (bbox) extraction, and `-t` a time box (tbox) extraction. Additional options are available, e.g., `--details` to print the individual results by file for folders and data repositories or `--output` to store the output in a *GeoPackage* file instead of printing to console."
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'format': 'shp', 'geoextent_handler': 'handleVector', 'bbox': [7.594978277801928, 51.96852473231792, 7.5957650477781415, 51.969118924937405], 'crs': '4326', 'tbox': ['2021-01-01', '2021-01-01']}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/garzon/.local/lib/python3.6/site-packages/pandas/compat/_optional.py:124: UserWarning: Pandas requires version '1.2.1' or newer of 'bottleneck' (version '1.2.0' currently installed).\n",
" warnings.warn(msg, UserWarning)\n"
]
}
],
"source": [
"%%bash\n",
"geoextent -b -t ../tests/testdata/shapefile/ifgi_denkpause.shp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Case study"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Zenodo is a data repository that stores different types of publication materials in form of `records`. The Zenodo API supports the discovery of records by dozens of different parameters stored in the metadata. These parameters include, e.g., `title`, `doi`, `keywords`, and `locations`. Even though geospatial metadata is available for queries via the `locations` parameter, this option seems to be limited to only a small number of records. Therefore, we study the number of records with geospatial metadata available in Zenodo and assess if by using geoextent we could provide the missing geospatial metadata for a significant number of other Zenodo records."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Zenodo geometadata"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first step for this analysis is determining the current state of geometadata in the Zenodo records. Only Open Access repositories are taken into account. For this purpose, we are going to use the Zenodo __[spatial search](https://zenodo.org/api/records/?bounds=-180,-90,180,90)__ function using the `bounds` query parameter, which accepts an area of interest (bounding box) as two coordinate pairs to extract the records within this zone. Zenodo's geospatial information of a record is stored in the `locations` property in the metadata. Each record could have multiple locations, but a location is limited to a single point with the properties `lat`, `lon`, and `place`.\n",
"\n",
"In the following cell, we extract the proportion of Open Access Zenodo records which have a `locations` property value. It is important to note that Zenodo records include 9 different _categories of publication types_: `poster`, `presentation`, `dataset`, `image`, `video`, `software`, `lesson`, `physicalobject`, and `other`, all of which are queried."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**ONLY RUN THE FOLLOWING CELL IF YOU WANT THE CURRENT INFORMATION FROM ZENODO API**\n",
"\n",
"New records are being added every day to Zenodo. We have prepared a data frame with the results from the API at the time of our analysis.\n",
"\n",
"Change the following cell type from `Raw` to `Code` before running it."
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"## Parameters for search\n",
"world_bounds = \"-180,-90,180,90\"\n",
"\n",
"# Type of Records\n",
"types_of_records = [\"publication\",\"poster\", \"presentation\", \"dataset\", \"image\",\n",
" \"video\", \"software\", \"lesson\", \"physicalobject\",\"other\"]\n",
"\n",
"# Dictionary with search parameters\n",
"search_param = {}\n",
"\n",
"# Zenodo search parameter for open records\n",
"search_param[\"record\"] = {\"access_right\":\"open\"}\n",
"\n",
"# Zenodo search parameter for open records with geometadata\n",
"search_param[\"record_geometadata\"] = dict(search_param[\"record\"], bounds = world_bounds)\n",
"\n",
"# Zenodo search parameter for open records with and without geometadata by type of record\n",
"for record_type in types_of_records:\n",
" search_param[record_type] = {\"access_right\":\"open\",\"type\":record_type}\n",
" search_param[record_type + '_geometadata'] = {\"access_right\":\"open\",\"type\":record_type,\"bounds\":world_bounds}\n",
" \n",
"## Extract the number of records by search_parameter\n",
"types_of_records.append('record')\n",
"\n",
"# dict for number of records by type \n",
"dict_type_of_record = {}\n",
"\n",
"for zenodo_record in types_of_records:\n",
" open_access = int(zenodo_api.get_number_of_records(search_param[zenodo_record]))\n",
" with_geometadata = int(zenodo_api.get_number_of_records(search_param[zenodo_record+'_geometadata']))\n",
" dict_type_of_record[zenodo_record] = {'Number open access records':open_access,\n",
" 'Number open access records with geometadata':with_geometadata}\n",
"\n",
"# Create dataframe\n",
"df_api = pd.DataFrame.from_dict(dict_type_of_record,orient='index')\n",
"# Proportion of zenodo records by type with geometadata\n",
"df_api['% records with metadata'] = df_api['Number open access records with geometadata']/df_api['Number open access records']*100\n",
"# Compute proportions of open zenodo record by type over total open zenodo records\n",
"df_api['% proportion over total geometadata']= df_api['Number open access records with geometadata']/df_api.loc['record','Number open access records with geometadata']*100\n",
"df_api.index.name='Type of record'\n",
"df_api.to_csv(OUTPUT_API_STATISTICS)"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | Number open access records | \n", "Number open access records with geometadata | \n", "% records with metadata | \n", "% proportion over total geometadata | \n", "
|---|---|---|---|---|
| Type of record | \n", "\n", " | \n", " | \n", " | \n", " |
| publication | \n", "997743 | \n", "13970 | \n", "1.400160 | \n", "99.849904 | \n", "
| poster | \n", "8397 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| presentation | \n", "21225 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| dataset | \n", "75542 | \n", "19 | \n", "0.025152 | \n", "0.135802 | \n", "
| image | \n", "601753 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| video | \n", "3286 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| software | \n", "53737 | \n", "1 | \n", "0.001861 | \n", "0.007147 | \n", "
| lesson | \n", "2555 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| physicalobject | \n", "28 | \n", "0 | \n", "0.000000 | \n", "0.000000 | \n", "
| other | \n", "6168 | \n", "1 | \n", "0.016213 | \n", "0.007147 | \n", "
| record | \n", "1770435 | \n", "13991 | \n", "0.790258 | \n", "100.000000 | \n", "
| \n", " | % Successful extractions | \n", "% No extraction | \n", "% Geoxtent failure | \n", "Number success | \n", "Number No extractions | \n", "Number Geoextent Failure | \n", "
|---|---|---|---|---|---|---|
| Geospatial extraction | \n", "14.420063 | \n", "81.191223 | \n", "4.388715 | \n", "46.0 | \n", "259.0 | \n", "14.0 | \n", "
| Temporal extraction | \n", "2.507837 | \n", "93.103448 | \n", "4.388715 | \n", "8.0 | \n", "297.0 | \n", "14.0 | \n", "
| \n", " | % Successful extractions | \n", "% No extraction | \n", "
|---|---|---|
| Geospatial extraction | \n", "2.134463 | \n", "97.865537 | \n", "
| Temporal extraction | \n", "0.080909 | \n", "99.919091 | \n", "
| \n", " | Geospatial extraction | \n", "
|---|---|
| Total number of files | \n", "25955.000000 | \n", "
| Number of files with potential | \n", "12954.000000 | \n", "
| % Files with potential | \n", "49.909459 | \n", "
| % Successful extractions over potential | \n", "4.276671 | \n", "
| % No extractions over potential | \n", "95.723329 | \n", "
| \n", " | Geospatial extraction | \n", "
|---|---|
| Total repositories | \n", "319.000000 | \n", "
| Repositories explored | \n", "301.000000 | \n", "
| Number of repositories with potential | \n", "164.000000 | \n", "
| % Repositories with potential | \n", "51.410658 | \n", "
| % Successful extractions over potential | \n", "28.048780 | \n", "
| % No extractions over potential | \n", "71.951220 | \n", "
| \n", " | Files in repository | \n", "Potential files with geoextent | \n", "Files with geoextent | \n", "
|---|---|---|---|
| Mean | \n", "86.229236 | \n", "43.036545 | \n", "1.840532 | \n", "
| Standard deviation | \n", "457.793327 | \n", "400.699765 | \n", "11.620147 | \n", "
| Min | \n", "1.000000 | \n", "0.000000 | \n", "0.000000 | \n", "
| 25% | \n", "3.000000 | \n", "0.000000 | \n", "0.000000 | \n", "
| 50% | \n", "12.000000 | \n", "1.000000 | \n", "0.000000 | \n", "
| 75% | \n", "31.000000 | \n", "4.000000 | \n", "0.000000 | \n", "
| Max | \n", "6515.000000 | \n", "6515.000000 | \n", "180.000000 | \n", "