7 posts tagged with "Google Cloud"

The recent DevCon 2025 event showcased not just cutting-edge development practices, but also demonstrated how modern DevOps principles and cloud infrastructure can seamlessly support large-scale technical workshops. Our team had the privilege of providing IT infrastructure and support for over 200 attendees, creating a robust learning environment through an exemplary public-private partnership.

CIROH's Research Cyberinfrastructure and DevOps team.
Left to right, top to bottom:
Manjila Singh, Arpita Patel, Nia Minor, Trupesh Patel, James Halgren; Benjamin Lee.

The Power of Collaboration​

DevCon 2025 represented an outstanding example of what's possible when public institutions and private enterprises work together:

• Corporate Sponsors: AWS and Google Cloud provided funding and infrastructure for the event.
• Infrastructure Partners: NSF JetStream and 2i2c JupyterHub delivered the computational backbone.
• Technical Implementation: The CIROH Research Cyberinfrastructure and DevOps Team managed IT access and software packaging.

Our Technical Approach​

Supporting 200+ workshop attendees required careful planning and execution. Our DevOps team implemented:

Infrastructure as Code​

• Automated deployment of workshop environments
• Scalable JupyterHub instances configured for concurrent users
• Pre-packaged software environments ensuring consistency across all workstations

Cloud-Native Solutions​

• Leveraged AWS S3 bucket and Google Cloud BigQuery
• Implemented auto-scaling to handle peak workshop loads
• Ensured high availability across multiple availability zones

Seamless User Experience​

• Single sign-on authentication for 100+ participants using CIROH 2i2c JupyterHub
• Pre-configured JetStream2 environments with all necessary tools
• Real-time support channels for immediate issue resolution

Key Achievements​

• Zero Downtime: Maintained 100% uptime throughout the entire event
• Rapid Onboarding: All 200 attendees were able to access their environments within minutes
• Consistent Experience: Every participant had identical, fully-functional development environments
• Real-time Support: Our team resolved technical issues with average response time under 5 minutes

Lessons Learned​

This event perfectly demonstrated how DevOps principles—bridging development and operations—combined with cloud computing can transform organizational capabilities. Key takeaways include:

1. Automation is Essential: Pre-event automation allowed us to focus on attendee support rather than infrastructure management.
2. Public-Private Partnerships Work: Combining NSF's research infrastructure with commercial cloud providers created a best-of-both-worlds solution.
3. Preparation Prevents Problems: Our extensive pre-event testing and redundancy planning paid dividends during the live event.

Looking Forward​

The success of DevCon 2025's IT infrastructure demonstrates that large-scale technical events no longer need to be limited by traditional IT constraints. By embracing DevOps practices and leveraging cloud partnerships, we can create learning environments that scale effortlessly while maintaining reliability and performance.

We're proud to have supported the DevCon 2025 community and look forward to applying these lessons to future events. The combination of public research infrastructure, private cloud resources, and dedicated DevOps expertise created an environment where 200+ developers could focus on learning and innovation without worrying about technical barriers.

Last week, I had the incredible opportunity to co-present a keynote at the CIROH Developers Conference (DevCon 2025), which attracted over 200 attendees. This presentation, which I presented alongside Dan Ames, focused on "CIROH HydroInformatics and Research Cyberinfrastructure." It was a fantastic experience to share insights into the powerful tools and technologies that CIROH engineers, students, researchers have been developing to advance hydrological research and operations.

Our keynote aimed to showcase the comprehensive ecosystem that CIROH offers. We highlighted four key pillars:

• Computing Resources: CIROH offers accress to public cloud infrastructure, on-premises HPC, and NSF ACCESS resources.
• Data Management: CIROH members handle and share vast datasets crucial for hydrological modeling. We've provided platforms to do so through HydroShare and CIROH AWS S3 buckets, as well as streamlining access and analysis of this data through tools like Google BigQuery API and Tethys Platform.
• Model Development: CIROH develops tools and frameworks for developing and refining hydrological models, including the NGIAB ecosystem.
• Knowledge Sharing: CIROH disseminates findings and best practices through the DocuHub and Portal platforms.

Value and Impact​

During the keynote, we emphasized the value that CIROH brings to new students and researchers. This includes access to:

• Computational resources that would normally cost thousands of dollars.
• Datasets that would take months to compile.
• Tools that streamline research and increase its impact.

We also highlighted several key tools within the CIROH ecosystem that are pushing hydrology forward:

• NGIAB (NextGen In A Box) ecosystem: Provides power and portability to the NextGen water modeling framework through leading open-source software development.
• CIROH's DocuHub: A central repository for documentation, as well as a platform for monthly blog and news updates.
• CIROH Portal: Facilitates to data, tools, and research findings.
• Google BigQuery NWM API: An efficient alternative for accessing and querying National Water Model data.
• Tethys Platform: Empowers intuitive, accessible web applications to deliver data and model results.
• Pantarhei and Wukong HPC: CIROH's high-performance computing cluster for demanding computations.
• And many more!

Our overall message was that CIROH’s hydroinformatics and research cyberinfrastructure ecosystem is designed to support and amplify research efforts. We encouraged attendees to explore these resources and consider how they could be applied to their own work. Whether it's streamlining data workflows, tackling computationally intensive tasks, or sharing findings, CIROH provides the tools and infrastructure to push the boundaries of hydrological science.

We want to thank everyone who attended our keynote and engaged in the Mentimeter quiz. It’s an exciting time for hydroinformatics, and we’re thrilled to be a part of this dynamic community!

Video voiced by Quinn Lee and prepared by Manjila Singh, Nia Minor, and Arpita Patel.

Last week at Google Cloud Next representing our CIROH cloud-based computing efforts! With more than 30,000 participants, Google Next always amazes me! It's huge, engaging on so many levels! Engaging booths, networking opportunities, great presentations, workshops, AI coach for basketball, incredible keynote from an amazing team! Event was not just a conference, but a celebration of innovation and a glimpse into the future of cloud computing! Great to see how Gemini is transforming data manipulation in BigQuery. The ability to use natural language to query, transform, and visualize data is revolutionizing how we interact with massive datasets. Gabe Weiss's demo particularly showcased the potential for non-specialists to derive insights from complex data.

If you missed the keynote, I highly recommend watching the recording here: GCN25 Keynote Video

Image Source: https://github.com/BYU-Hydroinformatics/api-nwm-gcp

Several important historical and ongoing National Water Model (NWM) datasets are now available on Google Cloud BigQuery, which makes them queryable through SQL using Google Cloud console. Some of these data sets are also accessible through an API (e.g. using Python). These datasets and their current status are as follows:

Product	Cloud Console SQL	CIROH API	Historical	Daily Updates
Medium-range forecasts	X	X	X	X
Long-range forecasts	X	X	X	X
Analysis and Assimilation	X	X	X	X
Retrospective Data (NWM v3)	X		X
Return Periods	X		X

This month, we are excited to showcase two case studies that utilized our cyberinfrastructure tools and services. These case studies demonstrate how CIROH's cyberinfrastructure is being utilized to support hydrological research and operational advancements.

1. ngen-datastream and NGIAB​

We're excited to share some recent developments and updates from CIROH's Research CyberInfrastructure team:

Cloud Infrastructure​

• CIROH's Google Cloud Account is now fully operational and managed by our team. You can find more information here.
• We're in the process of migrating our 2i2c JupyterHub to CIROH's Google Cloud account.
• We've successfully deployed the Google BigQuery API (developed by BYU and Google) for NWM data in our cloud. To access this API, please contact us at ciroh-it-admin@ua.edu. Please refer to NWM BigQuery API to learn more.

Google Cloud Next '24

Hello everyone, and thanks for stopping by!

I recently had the incredible opportunity to attend Google Cloud Next 2024 in person for the first time, and it was truly an amazing experience. From insightful keynote presentations and workshops to vibrant booths buzzing with connections, the event was a whirlwind of innovation and inspiration.