Skip to main content

AORC Data in Your Hands: User-Friendly Jupyter Notebooks for Data Retrieval and Analysis via CIROH JupyterHub Notebooks

· 3 min read
Homa Salehabadi
Postdoctoral Researcher
David Tarboton
Professor at Utah Water Research Laboratory

Screenshot of Hydroshare Resource

A screenshot of the HydroShare resource page for Jupyter Notebooks for the Retrieval of AORC Data for Hydrologic Analysis.

The Analysis of Record for Calibration (AORC) dataset is recognized as a high-value resource for the CUAHSI and CIROH community. This dataset is hosted by NOAA via Amazon Web Services (AWS) and is available in two primary formats: a latitude-longitude gridded dataset and the National Water Model (NWM) projected dataset, part of the NWM Retrospective archive. To enhance accessibility and illustrate analysis capabilities, we developed four user-friendly Jupyter Notebooks that enable data retrieval for both specific points of interest and spatial domains defined by shapefiles:

  • AORC_LL_PointRetrieval.ipynb: For retrieving and aggregating data from the latitude-longitude gridded dataset for a specific point using geographic coordinates.
  • AORC_LL_ZoneRetrieval.ipynb: For retrieving and aggregating data from the latitude-longitude gridded dataset for an area defined by a polygon shapefile.
  • AORC_NWMProj_PointRetrieval.ipynb: For retrieving and aggregating data from the NWM projected dataset for a specific point using geographic coordinates.
  • AORC_NWMProj_ZoneRetrieval.ipynb: For retrieving and aggregating data from the NWM projected dataset for an area defined by a polygon shapefile.

These Jupyter Notebooks, containing instructions and Python code to access the data, enable researchers to retrieve AORC data from AWS. From there, the notebooks offer options to subset and aggregate the data over user-defined time intervals (beyond the original hourly resolution) and spatial area. These serve as examples for how you could write or modify code to access AORC data in your work. The notebooks are publicly available on HydroShare and are compatible with JupyterHub computing platforms such as CIROH 2i2c JupyterHub linked to HydroShare.

To use these notebooks, go to the HydroShare resource, select "Open With" at the top right, and choose "CIROH 2i2c JupyterHub". This will copy the resource contents (notebooks and data) into the CIROH JupyterHub environment, where you can open and work through them to access the data. Note that you will need a CUAHSI HydroShare account to access "Open With" in HydroShare, and you will also need to request CIROH-2i2c JupyterHub access using a GitHub account.

Our work also includes a comparative analysis of the two AORC datasets with a summary of findings. While we mostly observed small differences, mainly due to projections, users should be aware of potential discrepancies between the datasets.

By providing these user-friendly tools and highlighting the characteristics of both AORC datasets, our work aims to support and facilitate more efficient hydrological and climate-related research within the CUAHSI and CIROH community.

References:

Assessing Streamflow Forecast Over the Hackensack River Watershed Using NGIAB

· 3 min read
Jorge Bravo
Graduate Research Assistant
Marouane Temini
Associate Professor

A poster, titled "Assessing streamflow forecast over the Hackensack River Watershed using physics- and AI-driven weather prediction models".

A poster presented by the I-SMART team at the CIROH Developers Conference, held at the University of Vermont in Burlington from May 28 to 30, 2025.

The densely populated Hackensack River watershed lies within the New York City Metropolitan Area, which spans northern New Jersey and southern New York. Accurate streamflow forecasting within this region is therefore essential to enable effective water resource management, flood prediction, and disaster preparedness.

Precipitation data is critical for effective hydrological modeling, making the identification of reliable data sources a key priority. This is why the Integrated Spatial Modeling and Remote Sensing Technologies Laboratory (I-SMART), an interdisciplinary research unit within the Davidson Laboratory at Stevens Institute of Technology in Hoboken, New Jersey, uses the latest developments in both atmospheric and hydrological modeling to address flood risks in the Hackensack Watershed with solutions that could be expanded to the entire New York City Metropolitan Area.

In the past, this work has included key early applications of the Next Generation Water Resources Modeling Framework (NextGen). Notably, the I-SMART group was among the first to force the NextGen framework with multiple atmospheric models for comparative analysis during a real-world event: the passage of Superstorm Ida over the New York metropolitan area in September 2021. The recent advent of NextGen In a Box (NGIAB) has provided an opportunity to accelerate these applications even further by taking full advantage of NGIAB's containerized, user-friendly, and easily deployed environment.

Recently, the I-SMART team has been testing various regional atmospheric models grounded in physical equations, including traditional models like WRF and next-generation atmospheric models such as MPAS. Additionally, given the increasing popularity and adoption of AI/ML-based approaches, the team has also begun exploring their potential. The goal of this work is to assess the performance of these approaches in the Hackensack Watershed, along with investigating the sensitivity of the model to various meteorological forcings, including forcings based on the National Water Model. (The initial and/or boundary conditions for all the models were determined using the Global Forecast System.)

This investigation required handling a large volume of precipitation data from various models, each with different spatial resolutions and in some cases, such as MPAS, using unstructured grids. As such, one of the key challenges was finding a hydrological modeling framework flexible enough to accommodate such diversity. This made the NextGen framework a natural choice, allowing them to integrate precipitation forcings from various sources with the appropriate pre-processing to align them with the model requirements in terms of spatial and temporal scales.

The complex implementation and execution of these models was faciliated by NextGen In A Box (NGIAB), which successfully enabled the integration of diverse precipitation sources. By simplifying local deployment and providing full control over model inputs, configurations, and runtime operations, NGIAB has given the I-SMART team the tools to conduct their groundbreaking research with even greater efficiency.

DevCon 2025: A DevOps and Cyberinfrastructure Success Story

· 3 min read
Arpita Patel
DevOps Manager and Enterprise Architect

The recent DevCon 2025 event showcased not just cutting-edge development practices, but also demonstrated how modern DevOps principles and cloud infrastructure can seamlessly support large-scale technical workshops. Our team had the privilege of providing IT infrastructure and support for over 200 attendees, creating a robust learning environment through an exemplary public-private partnership.

Image of CIROH's Research Cyberinfrastructure and DevOps team. On the left, two graphs are shown depicting usage for the Google Cloud-2i2c and Jetstream2 environments.

CIROH's Research Cyberinfrastructure and DevOps team.
Left to right, top to bottom:
Manjila Singh, Arpita Patel, Nia Minor, Trupesh Patel, James Halgren; Benjamin Lee.

DevCon 2025: Hydroinformatics and Research CyberInfrastructure Keynote

· 5 min read
Arpita Patel
DevOps Manager and Enterprise Architect

Last week, I had the incredible opportunity to co-present a keynote at the CIROH Developers Conference (DevCon 2025), which attracted over 200 attendees. This presentation, which I presented alongside Dan Ames, focused on "CIROH HydroInformatics and Research Cyberinfrastructure." It was a fantastic experience to share insights into the powerful tools and technologies that CIROH engineers, students, researchers have been developing to advance hydrological research and operations.


Application of NOAA-OWP's NextGen Framework: DevCon 2025 and EWRI Congress 2025 Highlights

· 5 min read
Sifan A. Koriche
Research [Hydrologic] Scientist

AWI Science and Technology Team @ CIROH DevCon2025

CIROH-AWI Science and Technology Team.
Left to right: Sagy Cohen, Steven Burian, Manjila Singh, Saide Zand, Savalan N. Neisary, Arpita Patel, Nia Minor, Trupesh Patel, Sifan A. Koriche, Jonathan Frame, Reza S. Alipour, Hari T. Jajula, Chad Perry; Josh Cunningham.

May was a pivotal month for representing the Cooperative Institute for Research to Operations in Hydrology (CIROH) and our collective work in advancing water science. As one of CIROH's Ambassadors, I had the privilege of connecting with the broader scientific community at two key events: the Environmental and Water Resources Institute (EWRI) Congress in Anchorage, Alaska, and the 2025 CIROH Developers Conference in Burlington, Vermont.

δHBV2.0: How NGIAB and Wukong HPC Streamlined Advanced Hydrologic Modeling

· 2 min read
Yalan Song
Research Assistant Professor
Leo Lonzarich
Graduate Researcher
Arpita Patel
DevOps Manager and Enterprise Architect
James Halgren
Assistant Director of Science

Image of graphical outputs from the δHBV2.0 model

Predicting water flow with precision across the vast U.S. landscape is a complex challenge. That's why Song et al. 2024 developed δHBV2.0, a cutting-edge hydrologic model. It’s built with high-resolution modeling of physics to deliver seamless, highly accurate streamflow simulations, even down to individual sub-basins. It's already proven to be a major improvement, performing better than older tools at about 4,000 measurement sites. We also provide a comprehensive 40-year water dataset for ~180,000 river reaches to support this.

Penn State research group pushed δHBV2.0 further, training it with even more detailed river data and integrating other trusted models, aiming to make it a key part of the NextGen national water modeling system (as a potential NWM3.0 successor). But here’s a common hurdle: making powerful scientific tools like this easy and reliable for everyone to use within a larger framework can be tough. Setup issues, runtime errors, and inconsistent results can frustrate users.

NGIAB stepped in to solve exactly this problem. Team has taken the complexity out of using the operations-ready models within NextGen by creating one unified, reliable package. Thanks to NGIAB, users don't have to worry about tricky setups or whether the model will run correctly. NGIAB ensures that our models are compatible everywhere and, most importantly, that they run exactly as designed, consistently and faithfully, every single time, no babysitting required. This means users get the full power of our advanced modeling, without the headaches.

Google Cloud Next 2025: Innovation at Scale ✨

· 5 min read
Arpita Patel
DevOps Manager and Enterprise Architect

Last week at Google Cloud Next representing our CIROH cloud-based computing efforts! With more than 30,000 participants, Google Next always amazes me! It's huge, engaging on so many levels! Engaging booths, networking opportunities, great presentations, workshops, AI coach for basketball, incredible keynote from an amazing team! Event was not just a conference, but a celebration of innovation and a glimpse into the future of cloud computing! Great to see how Gemini is transforming data manipulation in BigQuery. The ability to use natural language to query, transform, and visualize data is revolutionizing how we interact with massive datasets. Gabe Weiss's demo particularly showcased the potential for non-specialists to derive insights from complex data.

If you missed the keynote, I highly recommend watching the recording here: GCN25 Keynote Video

🌟 UA's Alabama Water Institute Showcases 30-Minute Hydrological Modeling Revolution🌟

· One min read
Arpita Patel
DevOps Manager and Enterprise Architect

🌍 AWI News

The Alabama Water Institute (AWI) at the University of Alabama (UA) recently published an article highlighting how NextGen In A Box (NGIAB) could transform hydrological modeling. This article provides great insight into NGIAB's real-world impact:

  • 🚀30-minute setup vs days/weeks of configuration
  • 📖 Provo River Basin Case Study demonstrating rapid deployment
ngiab image

➡️ Read the full press release here!

Pennsylvania State University Researchers Leverage CIROH Cyberinfrastructure for Advanced Hydrological Modeling

· 3 min read
Arpita Patel
DevOps Manager and Enterprise Architect
Yalan Song
Research Assistant Professor
Tadd Bindas
Graduate Researcher

Pennsylvania State University (PSU) researchers have been leveraging CIROH Cyberinfrastructure to tackle complex hydrological modeling challenges. This post highlights their innovative approach using the Wukong computing platform in conjunction with Amazon S3 bucket storage to efficiently process and analyze large-scale environmental datasets. 🚀

CIROH at AGU 2024

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

AGU24 brought together the world’s leading minds in Earth and space sciences. CIROH participated actively, showcasing advances in water prediction, modeling techniques and many more technologies.

Presentations and Posters 📊

The conference provided an excellent platform for CIROH researchers to present their groundbreaking work. Our team delivered impactful presentations and poster sessions highlighting CIROH’s innovative work, including advancements in water prediction systems and community water modeling.

These sessions sparked thought-provoking discussions and fostered collaborations with other researchers. For those who missed it, posters and presentation slides are now available here. Feel free to explore these materials and share your thoughts. 📝

Community NextGen Updates

· 3 min read
Arpita Patel
DevOps Manager and Enterprise Architect

The Community NextGen framework has seen significant advancements in November 2024, with major updates across multiple components and exciting new resources for users. Let's dive into the key developments that are making hydrologic modeling more accessible and powerful than ever.

CIROH Science Meeting 2024

· 4 min read
Arpita Patel
DevOps Manager and Enterprise Architect

The 2024 CIROH Science Meeting was a huge success, bringing together researchers, federal partners, and consortium members both in person and virtually. We're excited to share the valuable resources from this year's meeting with the wider CIROH community.

Slides and pictures from the various sessions, keynotes, and the Federal Town Hall have all been uploaded to a shared drive for easy access. You can find links to these materials here: Access the Shared Drive with Presentation Slides

Accessing National Water Model (NWM) Data via Google Cloud BigQuery API

· 4 min read
Arpita Patel
DevOps Manager and Enterprise Architect
gcp architectrure diagram Image Source: https://github.com/BYU-Hydroinformatics/api-nwm-gcp

Several important historical and ongoing National Water Model (NWM) datasets are now available on Google Cloud BigQuery, which makes them queryable through SQL using Google Cloud console. Some of these data sets are also accessible through an API (e.g. using Python). These datasets and their current status are as follows:

ProductCloud Console SQLCIROH APIHistoricalDaily Updates
Medium-range forecastsXXXX
Long-range forecastsXXXX
Analysis and AssimilationXXXX
Retrospective Data (NWM v3)XX
Return PeriodsXX

CIROH Cloud User Success Story

· 3 min read
Arpita Patel
DevOps Manager and Enterprise Architect

This month, we are excited to showcase two case studies that utilized our cyberinfrastructure tools and services. These case studies demonstrate how CIROH's cyberinfrastructure is being utilized to support hydrological research and operational advancements.

1. ngen-datastream and NGIAB

ngen-datastream image

CIROH Research CyberInfrastructure Update

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

We're excited to share some recent developments and updates from CIROH's Research CyberInfrastructure team:

Cloud Infrastructure

  • CIROH's Google Cloud Account is now fully operational and managed by our team. You can find more information here.
  • We're in the process of migrating our 2i2c JupyterHub to CIROH's Google Cloud account.
  • We've successfully deployed the Google BigQuery API (developed by BYU and Google) for NWM data in our cloud. To access this API, please contact us at ciroh-it-admin@ua.edu. Please refer to NWM BigQuery API to learn more.

CIROH Developers Conference 2024

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

CIROH Developers Conference 2024

DevCon2024

The CIROH team recently participated in the 2nd Annual CIROH Developers Conference (DevCon24), held from May 29th to June 1st,2024. The conference brought together a diverse group of water professionals to exchange knowledge and explore cutting-edge research in the field of hydrological forecasting.

AWRA 2024 Spring Conference

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

AWRA 2024 Spring Conference

The CIROH CyberInfrastructure team recently participated in the AWRA 2024 Spring Conference, co-hosted by the Alabama Water Institute at the University of Alabama.

Themed "Water Risk and Resilience: Research and Sustainable Solutions," the conference brought together a diverse group of water professionals to exchange knowledge and explore cutting-edge research in the field.

Google Cloud Next '24: A Flood of Innovation and Inspiration

· 5 min read
Arpita Patel
DevOps Manager and Enterprise Architect

Google Cloud Next '24

Hello everyone, and thanks for stopping by!

I recently had the incredible opportunity to attend Google Cloud Next 2024 in person for the first time, and it was truly an amazing experience. From insightful keynote presentations and workshops to vibrant booths buzzing with connections, the event was a whirlwind of innovation and inspiration.

Monthly News Update - February 2024

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

Welcome to the February edition of the CIROH DocuHub blog, where we bring you the latest updates and news about the Community NextGen project and CIROH's Cloud and on-premise Infrastructure.

Our team has been hard at work enhancing CIROH's Infrastructure and Community NextGen tools. Here are some highlights from February 2024:

  1. We successfully launched our new On-premises Infrastructure, which is now fully operational. You can find documentation for it here.

NextGen Monthly News Update - January 2024

· 2 min read
Arpita Patel
DevOps Manager and Enterprise Architect

Welcome to the January edition of the CIROH DocuHub blog, where we share the latest updates and news about the Community NextGen project monthly. NextGen is a cutting-edge hydrologic modeling framework that aims to advance the science and practice of hydrology and water resources management. In this month's blog, we will highlight some of the recent achievements and developments of the Community NextGen team.