Hub upgrade
With our Steering Committee agreeing to continue the Copernicus Australasia Regional Data Hub (the Hub) beyond June 2024 when the current agreement ends, our focus has been to prepare for another five years of operation, particularly in light of the European Commission’s Copernicus Programme transitioning to the new Data Space Ecosystem platform (CDSE). The CDSE replaces the current EU public access (Open/Sci/API) hubs decommissioned October 2023.
Over the coming year, the Copernicus Australasia Regional Data Hub will be making significant upgrades to its systems to ensure it remains compatible with the changes occurring in Europe. The upgrades will also improve system robustness and reliability, provide faster and more guaranteed access to new data, while remaining compatible with other services. The system will provide for more flexible data management and allow the Hub to utilise the latest storage and network technologies.
The major component will be transitioning our infrastructure to the new GAEL Store Service (GSS) platform that has been developed by GAEL Systems. As part of this transition, our data acquisition processes will migrate to the CDSE (as well as maintaining data acquisition from EUMETSAT’s Datastore), and a new user portal will be deployed.
The GSS platform will change how the Hub’s data is stored at the National Computational Infrastructure (NCI). This means the APIs used to connect to the Hub’s data will change to align with the CDSE APIs: https://dataspace.copernicus.eu/analyse/apis
Due data synchronisation issues with the CDSE, the new GSS system is now expected to go live in April 2024. Initially, the new system will only contain data collected from the go-live date onwards, however, it will be updated to hold all historic data over a period of 3-6 months. During this transition period, the current Hub portal will still be accessible, but will only contain historic data that has not yet been migrated to the new system. Once all the historic products have been migrated, we expect to decommission Sentinel Australasia Regional Access (SARA) and our legacy Hub. During this period of transition and decommissioning, we will provide a table on this webpage to allow users to track data locations.
Subscribe to our mailing list to receive more information as it becomes available subscribe to the Hub.
Frequently asked questions
- Over the next 6-12 months the Copernicus Australasia Regional Data Hub (the Hub) will be undergoing several software changes and updates to remain compatible with Copernicus EU’s new Copernicus Data Space Ecosystem platform, which replaces the public access EU hubs decommissioned during October 2023.
- These updates include:
- Transitioning the Hub infrastructure to a new GAEL Store Service (GSS) platform.
- Migration of the Hub’s data acquisition processes to the ESA Copernicus Data Space Ecosystem and EUMETSAT Data Store.
- New user portal interface.
- Changes in the APIs used to access the Hub data.
- Changes to how data is stored at the National Computational Infrastructure.
- With ESA moving to the Copernicus Data Space Ecosystem, the current Hub infrastructure will no longer be compatible. Therefore, we need to migrate to a system that will maintain our ability to synchronise Sentinel data with Europe. The new system will improve robustness and reliability, and allow the Hub to utilise the latest storage and network technologies.
- Transitioning to the new system will allow the Hub to continue to provide open access to Sentinel data for the region. However, the new system will provide a new data portal interface and APIs. This means that access to the Hub will change and you may need to make modifications to scripts or programs in order to continue to download Sentinel data from the Hub.
- In addition, it will take some time for us to migrate our current historic holdings (now over 6 petabytes) to the new system. During this transition phase, you may need to download products from both the old and new systems. Once the data migrations commence, we will provide a table to allow you to track data locations on this webpage.
Yes, you can find more information on the upgrade project timeline
| Phase | Timeframe | What is happening? | Impact to users |
|---|---|---|---|
| 1. Implementation | now – Sept 23 (early Oct 23) |
- GAEL develops new software/platform - NCI deploys new platform - New platform configured to connect to ESA DataSpace & EUMETSAT DataStore - Testing | - None |
| 2. Script migration | Aug – Sept 23 | - Adaption and development of partner (and expert user) auto download scripts to new platform | - Need to ensure user code/scripts are ready prior to new platform going live |
| 3. Go live | April 2024 |
- New platform goes live, sync’ing new data from CDSE - Old hub stops syncing data (may be short 1 week overlap) |
- New Sentinel data now only available from new portal - Must switch auto-download scripts (for new data) to new platform APIs. |
| 4. Data Migration | Mar 24 – Dec 24 |
- Migrate and/or resync historic data from old Hub to new platform - NCI moves data to new data store |
- Scripts to download historic data will need to be duplicated and modified to connect to new platform APIs - Historic data may be available from either old or new systems (see Data Transition table) - Users accessing data directly at NCI will need to change data access points |
| 5. Decommissioning | end 2024 | - Old hub will be turned off and decommissioned | - New and historic data only available via new portal and APIs |
- A number of documents and guides are available to help transition to the new system:
- GSS API documentation – the GSS system will be using APIs that are compatible with the CDSE, more information can be found here: https://dataspace.copernicus.eu/analyse/apis
- GSS GUI/web portal user guide – to come
- Data migration transition matrix
- Python Library documentation (access: https://pypi.org/project/pygssearch/1.0.0/, User documentation)
- FAQ: how to modify old API scripts to new GSS API’s – to come
- We aim to have the new GSS based system online by April 2024. It will initially only contain new synchronised data, however, it will eventually replace the current the Hub once all the historic data has been migrated to the new system.
- The new GSS based system will eventually replace the current the Hub. Once all the historic data has been migrated to the new GSS system, the old system will be decommissioned and no longer available. The decommissioning date will be determined once the data migration is completed.
- The new system will continue to use the file system at the NCI to store the Hub products, however due to the way the GSS software indexes files, products will not be stored in the same way or under the existing folder structure.
- If you use the current file structure at NCI to “predict” where new products will be, this will no longer be available once products are migrated to the new GSS system. The only option to find products you need will be to search via the GSS portal or APIs.
- Once you obtain the location of a product via the GSS portal or API, you may still be able to access the files directly from the file system, however we recommend where possible, to use the new command line tool, Python library, or GSS APIs to download any data from the Hub instead.
- If you need help with accessing the new command line tool, Python library, or GSS APIs on the new system, please contact us via CopernicusAustralasia@ga.gov.au.
- If you have any problems or concerns about the new GSS system please contact us via CopernicusAustralasia@ga.gov.au