Community NextGen Updates
- December 2024 Update
update NGIAB-data-preprocess
Pro Tip: The data-preprocess tool offers VPU subsetting capabilities and can generate configuration files for empirical models like the 'Demonstration LSTM'. Check out these and other new features in the latest release! Discover more updates here.
Pro Tip: The data-preprocess tool offers VPU subsetting capabilities and can generate configuration files for empirical models like the 'Demonstration LSTM'. Check out these and other new features in the latest release! Discover more updates here.
update NGIAB-CloudInfra: NGIAB Docker Image
Version 1.4.0 is now available! This release introduces the “demonstration LSTM” within NGIAB, along with sample AWI-008 input data. The LSTM model serves as an educational example to help users get started, though it’s not optimized for maximum prediction accuracy.” Explore more updates here.
Version 1.4.0 is now available! This release introduces the “demonstration LSTM” within NGIAB, along with sample AWI-008 input data. The LSTM model serves as an educational example to help users get started, though it’s not optimized for maximum prediction accuracy.” Explore more updates here.
update ngen-datastream
Important Update: Datastream has been upgraded to v2.2 hydrofabric and now generates both NextGen outputs and forcings for VPUs 02, 03W, and 16. For more details, visit the Datastream portal at https://datastream.ciroh.org/index.html
Important Update: Datastream has been upgraded to v2.2 hydrofabric and now generates both NextGen outputs and forcings for VPUs 02, 03W, and 16. For more details, visit the Datastream portal at https://datastream.ciroh.org/index.html
update NGIAB-HPCInfra: NGIAB Singularity Image
December updates include 4 PRs adding 'Demostration LSTM' in NGIAB-HPCInfra. Learn more about the updates here.
December updates include 4 PRs adding 'Demostration LSTM' in NGIAB-HPCInfra. Learn more about the updates here.
update NWM BigQuery API Update
Access NWM streamflow prediction data directly through BigQuery instead of downloading netCDF files. Learn more about this efficient data access method in our documentation here:
Access NWM streamflow prediction data directly through BigQuery instead of downloading netCDF files. Learn more about this efficient data access method in our documentation here:
- November 2024 Update
update NGIAB-CloudInfra: NGIAB Docker Image
Release v1.3.0 has been released with several significant improvements: * Integration of forked ngen and t-route repositories from CIROH-UA's GitHub. * Updated sample input data based on hydrofabric v2.2 * 15 PRs merged * TEEHR integration * CI pipeline improvements with unified Dockerfile * Enhanced NGIAB Visualizer. Explore more updates here.
Release v1.3.0 has been released with several significant improvements: * Integration of forked ngen and t-route repositories from CIROH-UA's GitHub. * Updated sample input data based on hydrofabric v2.2 * 15 PRs merged * TEEHR integration * CI pipeline improvements with unified Dockerfile * Enhanced NGIAB Visualizer. Explore more updates here.
update NGIAB-HPCInfra: NGIAB Singularity Image
November updates include: * 4 PRs updating singularity image to use CIROH-UA repositories * Updated README with new sample input data for hydrofabric v2.2 * Alignment with Docker image modifications
November updates include: * 4 PRs updating singularity image to use CIROH-UA repositories * Updated README with new sample input data for hydrofabric v2.2 * Alignment with Docker image modifications
update NGIAB-data-preprocess
Version 3.1.2 released featuring: * 7 PRs merged * Major update for compatibility with hydrofabric v2.2. Discover more updates here.
Version 3.1.2 released featuring: * 7 PRs merged * Major update for compatibility with hydrofabric v2.2. Discover more updates here.
feature NextGen Model Framework (CIROH-UA/ngen) - fork of NOAA-OWP/ngen
NGIAB-CloudInfra and NGIAB-HPCInfra now utilize the main branch of CIROH-UA/ngen.
NGIAB-CloudInfra and NGIAB-HPCInfra now utilize the main branch of CIROH-UA/ngen.
feature T-route (CIROH-UA/t-route) - fork of NOAA-OWP/t-route
NGIAB-CloudInfra and NGIAB-HPCInfra now utilize the datastream branch of CIROH-UA/t-route. Main branch is in sync with NOAA-OWP/t-route
NGIAB-CloudInfra and NGIAB-HPCInfra now utilize the datastream branch of CIROH-UA/t-route. Main branch is in sync with NOAA-OWP/t-route
- September 2024 Update
news Best Practice Guide for NextGen Framework
Technical guidance for the inclusion of models/modules in the NextGen Water Resources Modeling Framework is now available in DocuHub.
Technical guidance for the inclusion of models/modules in the NextGen Water Resources Modeling Framework is now available in DocuHub.
update NextGen Model Framework (NOAA-OWP/ngen)
2 PRs merged in September. Explore more updates here.
2 PRs merged in September. Explore more updates here.
news NextGen Community Office Hours
We are excited to invite you to join our NextGen Community Office Hours. It is an excellent opportunity for everyone to stay up to date with the latest developments in hydrologic modeling at CIROH. Join us to engage with the team and learn more about ongoing projects and advancements.
We are excited to invite you to join our NextGen Community Office Hours. It is an excellent opportunity for everyone to stay up to date with the latest developments in hydrologic modeling at CIROH. Join us to engage with the team and learn more about ongoing projects and advancements.
feature NGIAB-CloudInfra : NGIAB Docker Image
We are happy to announce that NGIAB version 1.2.1 has been released, available for both ARM and x86 architectures. This release is up to date with the latest ngen commit ID: f321889 and t-route commit ID: d0982a5. Learn more about the updates here.
We are happy to announce that NGIAB version 1.2.1 has been released, available for both ARM and x86 architectures. This release is up to date with the latest ngen commit ID: f321889 and t-route commit ID: d0982a5. Learn more about the updates here.
- August 2024 Update
update NextGen Model Framework (NOAA-OWP/ngen)
Five PRs merged in August. Explore more updates here.
Five PRs merged in August. Explore more updates here.
update NGIAB-CloudInfra : NGIAB Docker Image
NGIAB version 1.2.0 has been released, available for both ARM and X86 architectures. This release is up to date with the latest ngen commit ID: 2ffedf8 and t-route commit ID: 9d9d711
NGIAB version 1.2.0 has been released, available for both ARM and X86 architectures. This release is up to date with the latest ngen commit ID: 2ffedf8 and t-route commit ID: 9d9d711
update NGIAB-HPCInfra: NGIAB Singularity Image
In August, a new Singularity image was released for NGIAB-HPCInfra. Learn more about the updates here.
In August, a new Singularity image was released for NGIAB-HPCInfra. Learn more about the updates here.
update ngen-datastream
In August, nine pull requests were merged. These include the addition of the forcingprocessor feature, increased test coverage, updates to the research datastream Terraform, and new documentation.
In August, nine pull requests were merged. These include the addition of the forcingprocessor feature, increased test coverage, updates to the research datastream Terraform, and new documentation.
update NGIAB-data-preprocess
Five PRs were merged in August. We are excited to announce the new release v2.0.0! This release uses catchment ids in place of waterbody ids.
Five PRs were merged in August. We are excited to announce the new release v2.0.0! This release uses catchment ids in place of waterbody ids.
- July 2024 Update
update NextGen Model Framework (ngen)
Four PRs merged in July. Explore more updates here.
Four PRs merged in July. Explore more updates here.
update NGIAB-CloudInfra
In July, one PR was merged. We are preparing to present "Advancing Hydrological Modeling: CIROH’s NextGen In A Box (NGIAB) and Enhanced Tools for Community-Driven Research" at AGU24. You can find more updates here.
In July, one PR was merged. We are preparing to present "Advancing Hydrological Modeling: CIROH’s NextGen In A Box (NGIAB) and Enhanced Tools for Community-Driven Research" at AGU24. You can find more updates here.
update ngen-DataStream
In July, 6 PRs were merged. We are also presenting "NextGen National Water Model Framework Datastream" paper at AGU24. More updates can be found here.
In July, 6 PRs were merged. We are also presenting "NextGen National Water Model Framework Datastream" paper at AGU24. More updates can be found here.
- June 2024 Update
update ngen-DataStream
In June, 9 PRs were merged. Ngen-datastream was updated to be compatible with the new hydrofabric data format provided by hfsubset version 2.1.1.
In June, 9 PRs were merged. Ngen-datastream was updated to be compatible with the new hydrofabric data format provided by hfsubset version 2.1.1.
news NGIAB-data-preprocessor
Four PRs were merged in June. We are happy to announce the new release v1.0.1. This release includes various improvements and bug fixes.
Four PRs were merged in June. We are happy to announce the new release v1.0.1. This release includes various improvements and bug fixes.
feature NextGen Model Framework (ngen)
Eleven PRs merged in June. Explore more updates here.
Eleven PRs merged in June. Explore more updates here.
news CIROH DevCon24 Conference Slides
Access the slides from the 8 NextGen Track workshops held at CIROH DevCon24. Slides can be found here.
Access the slides from the 8 NextGen Track workshops held at CIROH DevCon24. Slides can be found here.
- May 2024 Update
news NGIAB-CloudInfra
The NextGen In A Box (NGIAB) workshop last week at CIROH DevCon24 was a huge success. New sample dataset is created “AWI-006”. We had 50+ attendees and most of them were successfully able to run NGIAB on their local machine. For those without the admin access, we provided them the AWS instances for the workshop. This workshop introduced a new geospatial visualization tool for visualizing outputs from NGIAB runs. During May, 16 PRs were merged. Slides can be found at:
The NextGen In A Box (NGIAB) workshop last week at CIROH DevCon24 was a huge success. New sample dataset is created “AWI-006”. We had 50+ attendees and most of them were successfully able to run NGIAB on their local machine. For those without the admin access, we provided them the AWS instances for the workshop. This workshop introduced a new geospatial visualization tool for visualizing outputs from NGIAB runs. During May, 16 PRs were merged. Slides can be found at:
update NGIAB-HPCInfra
Run NGIAB on HPC using NGIAB-HPCInfra. This repo is upgraded to use AWI-006 dataset. For more details, please refer README at
Run NGIAB on HPC using NGIAB-HPCInfra. This repo is upgraded to use AWI-006 dataset. For more details, please refer README at
news ngen-DataStream
NextGen Simulation Development Tools workshop was a huge success. It provided attendees with a hands-on experience using the ngen-datastream tool. In May, 21 PRs were merged. Explore the updates. Slides for the DevCon can be found at:
NextGen Simulation Development Tools workshop was a huge success. It provided attendees with a hands-on experience using the ngen-datastream tool. In May, 21 PRs were merged. Explore the updates. Slides for the DevCon can be found at:
news NGIAB-data-preprocessor
The NextGen Simulation Development Tools workshop at CIROH DevCon24 showcased the tool's capabilites of simplifying data preparation for next-gen simulations.
The NextGen Simulation Development Tools workshop at CIROH DevCon24 showcased the tool's capabilites of simplifying data preparation for next-gen simulations.
- April 2024 Update
news NGIAB-data-preprocessor
Introducing a tool for NGIAB data preparation. A workshop is planned for CIROH DevCon24 under NextGen Simulation Development Tools.
Introducing a tool for NGIAB data preparation. A workshop is planned for CIROH DevCon24 under NextGen Simulation Development Tools.
news NGIAB-CloudInfra
In April, we are gearing up to present the NextGen In A Box (NGIAB) workshop at CIROH DevCon24
In April, we are gearing up to present the NextGen In A Box (NGIAB) workshop at CIROH DevCon24
news ngen-DataStream
In April, we are getting ready to present NextGen DataStream at CIROH DevCon24. A workshop is planned under NextGen Simulation Development Tools
In April, we are getting ready to present NextGen DataStream at CIROH DevCon24. A workshop is planned under NextGen Simulation Development Tools
feature NextGen Model Framework (ngen)
Thirteen PRs merged in April. Explore the updates.
Thirteen PRs merged in April. Explore the updates.
- March 2024 Update
news Improvement of NGIAB HPCInfra Repositories
Automating build of NGIAB HPCInfra using GitHub Actions
Automating build of NGIAB HPCInfra using GitHub Actions
feature ngen-DataStream
In March, the python internal to the datastream was containerized. This included implementing catchment-specific configuration for PET, CFE, and Noah OWP (https://github.com/CIROH-UA/ngen-datastream/pull/46). The scripts and plotting tools were developed to perform benchmarking and preliminary results have been collected.
In March, the python internal to the datastream was containerized. This included implementing catchment-specific configuration for PET, CFE, and Noah OWP (https://github.com/CIROH-UA/ngen-datastream/pull/46). The scripts and plotting tools were developed to perform benchmarking and preliminary results have been collected.
- Feb 2024 Update
news Improvement of NGIAB CI Pipeline for Pull Requests from Forked Repositories
Three pull requests have been successfully merged. Pull requests submitted using forked repositories are now automatically tested in the CI Pipeline. Upon merging changes to the main repository, another CI pipeline is triggered to retest the build and push the image to DockerHub. Additionally, new AWI_004 sample data is now available for 09 VPU for unit testing.
Three pull requests have been successfully merged. Pull requests submitted using forked repositories are now automatically tested in the CI Pipeline. Upon merging changes to the main repository, another CI pipeline is triggered to retest the build and push the image to DockerHub. Additionally, new AWI_004 sample data is now available for 09 VPU for unit testing.
feature ngen-DataStream
In February, two pull requests were merged. For small runs, you can access ngen-DataStream at: https://github.com/CIROH-UA/ngen-datastream/tree/main/examples
In February, two pull requests were merged. For small runs, you can access ngen-DataStream at: https://github.com/CIROH-UA/ngen-datastream/tree/main/examples
- Jan 2024 Update
news Run NextGen In A Box(NGIAB) with Singularity on HPC without docker support!!!
If you want to use NGIAB on HPC that does not support docker, we have a solution for you. Please follow the steps at this link.
If you want to use NGIAB on HPC that does not support docker, we have a solution for you. Please follow the steps at this link.
news NextGen In A Box (NGIAB)
We have made some improvements to NGIAB: 10 PRs merged. Sample input data updated to use AWI_003(with new data provider names). Boost upgraded to v1.79. Self-hosted Runner can start and stop automatically with GitHub Actions. NGIAB image can run in auto mode. Geopackage support added. Documentation available at CIROH DocuHub
We have made some improvements to NGIAB: 10 PRs merged. Sample input data updated to use AWI_003(with new data provider names). Boost upgraded to v1.79. Self-hosted Runner can start and stop automatically with GitHub Actions. NGIAB image can run in auto mode. Geopackage support added. Documentation available at CIROH DocuHub
feature ngen-DataStream
3 PRs merged in January. Documentation available at: /products/Hydrologic%20Modeling%20Framework/nextgenDatastream/
3 PRs merged in January. Documentation available at: /products/Hydrologic%20Modeling%20Framework/nextgenDatastream/
- Dec 2023 Update
news NGIAB presentation at AGU!!!
NextGen In A Box: Advancing Collaborative Modeling for Enhanced Water Resource Management presented by Arpita. We had a full house!
NextGen In A Box: Advancing Collaborative Modeling for Enhanced Water Resource Management presented by Arpita. We had a full house!
news NextGen In A Box
NGIAB Updates: Merged CI pipeline changes PR#74 by Benjamin Lee, Added case study map and a plot with output results PR#72 by Shahab Alam, PR#70 by Josh Cu
NGIAB Updates: Merged CI pipeline changes PR#74 by Benjamin Lee, Added case study map and a plot with output results PR#72 by Shahab Alam, PR#70 by Josh Cu
feature NextGen Model Framework
5 PRs merged in December.
5 PRs merged in December.
feature T-route
12 PRs merged in December.
12 PRs merged in December.
feature Hydrofabric
Lynker is proud to support CIROH with a freely accessable resource for geospatial data: https://www.lynker-spatial.com/. Updated transects/cross section runners
Lynker is proud to support CIROH with a freely accessable resource for geospatial data: https://www.lynker-spatial.com/. Updated transects/cross section runners
feature NextGen Framework Forcings
December updates include an (almost) complete stream script that will produce daily ngen outputs. Documentation (readme) included. ngen-datastream/subsetting has been deprecated and hfsubset integrated into the stream. Benchmarking has begun.
December updates include an (almost) complete stream script that will produce daily ngen outputs. Documentation (readme) included. ngen-datastream/subsetting has been deprecated and hfsubset integrated into the stream. Benchmarking has begun.
note Community Support
December is always an AGU Month! San Francisco (the traditional home of AGU) brings us to the season where we reflect on our own work for the year and finally learn what our coworkers have been doing all this time! Checkout the presentation slides from CIROH at below link.
December is always an AGU Month! San Francisco (the traditional home of AGU) brings us to the season where we reflect on our own work for the year and finally learn what our coworkers have been doing all this time! Checkout the presentation slides from CIROH at below link.
- Nov 2023 Update
news NextGen Research Lightning Talk
Presented at CIROH NextGen Research Lightning Talk Webinar
Presented at CIROH NextGen Research Lightning Talk Webinar
feature NextGen Model Framework
14 PRs merged in November. Adding new tests for sqlite and new geopackage-based hydrofabric support.
14 PRs merged in November. Adding new tests for sqlite and new geopackage-based hydrofabric support.
feature T-route
13 PRs merged in November. Added additional json/geojson support
13 PRs merged in November. Added additional json/geojson support
feature Hydrofabric
Lynker is proud to support CIROH with a freely accessable resource for geospatial data: https://www.lynker-spatial.com/. Updated hfsubset to support using the v2.0 hydrofabric by default. Added Cross-section support to the hydrofabric. 13 PRs merged in November. Added additional json/geojson support
Lynker is proud to support CIROH with a freely accessable resource for geospatial data: https://www.lynker-spatial.com/. Updated hfsubset to support using the v2.0 hydrofabric by default. Added Cross-section support to the hydrofabric. 13 PRs merged in November. Added additional json/geojson support
feature NextGen Framework Forcings
14 PRs merged towards the Research Data Stream. Fixed lingering pyarrow issues. Added new support for all segments to be dockerized. Added S3 support with regards to the control flow design, to support one reusable, configurable sequence
14 PRs merged towards the Research Data Stream. Fixed lingering pyarrow issues. Added new support for all segments to be dockerized. Added S3 support with regards to the control flow design, to support one reusable, configurable sequence
note Community Support
Singularity support is coming to AWI for HPC users that are able to run Singularity builds. Work on serialization (passing from one run to the next with model states still in memory or imported from a file) is ongoing, relative temporal configurations of NextGen with consideration to model configuration changes between timesteps, and better general support for framework polymorphism.
Singularity support is coming to AWI for HPC users that are able to run Singularity builds. Work on serialization (passing from one run to the next with model states still in memory or imported from a file) is ongoing, relative temporal configurations of NextGen with consideration to model configuration changes between timesteps, and better general support for framework polymorphism.
- Oct 2023 Update
note NGIAB Updates
15 runs of the whole build process, 7 successful, and 2 pending. Over 550 pulls of the container image, number of 'canonical' runs (uploading their metadata and results) is coming in a future version.
15 runs of the whole build process, 7 successful, and 2 pending. Over 550 pulls of the container image, number of 'canonical' runs (uploading their metadata and results) is coming in a future version.
feature NextGen Model Framework
Updates to SUMMA candidate model, performance improvements for memory usage and spatial domain tooling to decouple the currently used geojson Feature and geometry classes into their own simple features interface. This is an abstract interface, with a coupled Boost.Geometry backend that will be used as the default backend.
Updates to SUMMA candidate model, performance improvements for memory usage and spatial domain tooling to decouple the currently used geojson Feature and geometry classes into their own simple features interface. This is an abstract interface, with a coupled Boost.Geometry backend that will be used as the default backend.
note T-route
22 Pull Requests were worked on in October with 11 merged, and a focus on testing and getting the compiling process to be tested and reliable particularly for downstream NGIAB and HPC usage of t-route
22 Pull Requests were worked on in October with 11 merged, and a focus on testing and getting the compiling process to be tested and reliable particularly for downstream NGIAB and HPC usage of t-route