10 posts tagged with "ciroh"

Pennsylvania State University (PSU) researchers have been leveraging CIROH Cyberinfrastructure to tackle complex hydrological modeling challenges. This post highlights their innovative approach using the Wukong computing platform in conjunction with Amazon S3 bucket storage to efficiently process and analyze large-scale environmental datasets. 🚀

---

💻 The Computing and Storage Infrastructure​

Wukong Computing Platform​

The PSU team has been utilizing Wukong, a high-performance computing (HPC) cluster specifically designed for data-intensive scientific applications, such as the high-resolution physics-informed machine learning for national water modeling (Song et al. 2024[YS1] ). Wukong provides the computational power necessary for running complex simulations and processing large environmental datasets that traditional computing resources would struggle with. 🔍

Key advantages of Wukong include:​

• 🎯 Large GPU capacity for high-resolution ML/differentiable process-based models
• ⚙️ Scalable parallel processing capabilities
• 🚀 Optimized performance for data-intensive workloads
• ⏳ Reduced processing time for big data
• 🌐 Support for multi-node computation to handle larger geographical areas

S3 Bucket Integration:​

To complement Wukong’s computational power, the PSU researchers and AWI DevOps staff implemented Amazon S3 (Simple Storage Service) buckets as their secondary data storage solution. This integration offers several benefits:

• 🗄️ Virtually unlimited storage capacity for growing datasets
• 🔒 Data durability and redundancy
• 💰 Cost-effective long-term storage, with the use of S3 intelligent tiering to automate the storage cost savings by moving data when access patterns change
• 🔄 Seamless data transfer between computing nodes
• 📝 Version control for dataset iterations
• 🤝 Easy data sharing with users not on Wukong

---

🔬 Research Applications​

The PSU team has applied this powerful computing infrastructure to several critical research areas:

1. National Streamflow Modeling 🌊
   Training differentiable hydrologic models with high-resolution forcing and static attribute data across extensive geographical regions using observations from thousands of gauges, followed by whole-domain forwarding.

2. National River Routing 🗺️
   Conducting river routing on MERIT/HydroFabric river networks, combined with neural network-supported routing parameter learning.

3. NextGen Candidate Models & Data Assimilation 🔄
   Applying multiple NextGen candidate models and data assimilation algorithms within the differentiable modeling framework, which supports compliance with BMI.

4. Foundation Model Development 🏞️
   Developing a foundation model to explore co-evolution between landscapes.

---

Thank you to all those who contributed towards this effort.​

🔗 Learn More​

For more details on the Wukong computing platform, check out the official documentation:
👉 Wukong Documentation

For the full research paper by Song et al. (2024), visit:
👉 DOI: 10.22541/essoar.172736277.74497104/v1

---

AGU24 brought together the world’s leading minds in Earth and space sciences. CIROH participated actively, showcasing advances in water prediction, modeling techniques and many more technologies.

Presentations and Posters 📊​

The conference provided an excellent platform for CIROH researchers to present their groundbreaking work. Our team delivered impactful presentations and poster sessions highlighting CIROH’s innovative work, including advancements in water prediction systems and community water modeling.

These sessions sparked thought-provoking discussions and fostered collaborations with other researchers. For those who missed it, posters and presentation slides are now available here. Feel free to explore these materials and share your thoughts. 📝

The 2024 CIROH Science Meeting was a huge success, bringing together researchers, federal partners, and consortium members both in person and virtually. We're excited to share the valuable resources from this year's meeting with the wider CIROH community.

Slides and pictures from the various sessions, keynotes, and the Federal Town Hall have all been uploaded to a shared drive for easy access. You can find links to these materials here: Access the Shared Drive with Presentation Slides

Image Source: https://github.com/BYU-Hydroinformatics/api-nwm-gcp

Several important historical and ongoing National Water Model (NWM) datasets are now available on Google Cloud BigQuery, which makes them queryable through SQL using Google Cloud console. Some of these data sets are also accessible through an API (e.g. using Python). These datasets and their current status are as follows:

Product	Cloud Console SQL	CIROH API	Historical	Daily Updates
Medium-range forecasts	X	X	X	X
Long-range forecasts	X	X	X	X
Analysis and Assimilation	X	X	X	X
Retrospective Data (NWM v3)	X		X
Return Periods	X		X

This month, we are excited to showcase two case studies that utilized our cyberinfrastructure tools and services. These case studies demonstrate how CIROH's cyberinfrastructure is being utilized to support hydrological research and operational advancements.

1. ngen-datastream and NGIAB​

We're excited to share some recent developments and updates from CIROH's Research CyberInfrastructure team:

Cloud Infrastructure​

• CIROH's Google Cloud Account is now fully operational and managed by our team. You can find more information here.
• We're in the process of migrating our 2i2c JupyterHub to CIROH's Google Cloud account.
• We've successfully deployed the Google BigQuery API (developed by BYU and Google) for NWM data in our cloud. To access this API, please contact us at ciroh-it-admin@ua.edu. Please refer to NWM BigQuery API to learn more.

CIROH Developers Conference 2024

The CIROH team recently participated in the 2nd Annual CIROH Developers Conference (DevCon24), held from May 29th to June 1st,2024. The conference brought together a diverse group of water professionals to exchange knowledge and explore cutting-edge research in the field of hydrological forecasting.

Accelerating Innovation: CIROH's March 2024 Update

The CIROH team has been diligently accelerating research cyberinfrastructure capabilities this month. We're thrilled to share key milestones achieved in enhancing the Community NextGen project and our cloud/on-premises platforms.

Welcome to the February edition of the CIROH DocuHub blog, where we bring you the latest updates and news about the Community NextGen project and CIROH's Cloud and on-premise Infrastructure.

Our team has been hard at work enhancing CIROH's Infrastructure and Community NextGen tools. Here are some highlights from February 2024:

1. We successfully launched our new On-premises Infrastructure, which is now fully operational. You can find documentation for it here.

Adding posts​

What file name to use?​

DocuHub will extract a YYYY-MM-DD date from many patterns such as YYYY-MM-DD-my-blog-post-title.md or YYYY/MM/DD/my-blog-post-title.md. This enables you to easily group blog posts by year, by month, or to use a flat structure.