Maintenance Downtime Logs for 2025

December 2025 Downtime

Date of downtime: Tuesday, December 30, 2025

This downtime has been cancelled.

November 2025 Downtime

Date of downtime: Tuesday, November 25, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, login-future - formerly vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None

October 2025 Downtime

Date of downtime: Wednesday, October 29, 2025 NOTE: This has been moved to accomodate the relocation of compute nodes as part of the data center expansion project.

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, login-future - formerly vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users: Racks h22, h23, h24, h25 in the ub-hpc cluster and racks m24, v10, and v11 in the faculty cluster will remain offline and be relocated on Thursday, October 30. They will be brought online as quickly as possible following re-cabling and re-installation.

September 2025 Downtime

Date of downtime: Tuesday, September 30, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Compute node image updates
  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, login-future - formerly vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
- The RStudio Server app has been deprecated will be removed from OnDemand. RStudio users should begin using the "NEW - RStudio Server" app. To continue using RStudio as you have been with this new app, select "CCR R 4.2 software module" from the "Where would you like to run R from?" drop-down list.

September 2025 Data Center Power Outage

Date of downtime: September 2 - September 5

Approximate time of outage: Beginning at 3pm Tuesday, September 2 we will shut down affected servers and services. As soon as electrical work is completed, CCR staff will return the HPC infrastructure to production. This is anticipated to be the morning of Friday, September 5. Rack relocation #2 of the data center move will take place during this time period.

CCR resources affected by power outage:

  • UB-HPC cluster (all partitions - except nodes moved to new data center)
  • Faculty cluster (all partitions - except nodes moved to new data center)

Empire AI resources affected by power outage:

  • Alpha cluster (all partitions)
  • Alpha login nodes

Resources NOT affected by power outage:

  • LakeEffect research cloud
  • Vast storage system (home, project, global scratch directories)
  • Login & compile nodes
  • Globus data transfer service
  • Portals: ColdFront, Identity management, OnDemand

Specific Effects to CCR users:

  • There will be very limited availability of CCR HPC infrastructure during this time. Compute nodes in both the UB-HPC and Faculty cluster that have been relocated to the new data center will be available for jobs.
  • Users WILL be able to login to OnDemand, the login nodes, and Globus during this outage
  • No Empire AI HPC resources will be available for the duration of this outage.

What will be done: Additional electrical updates will be done to prepare for the installation of a new UPS.

August 2025 Data Center Power Outage

Date of downtime: 7am Friday 8/8/25 through Monday 8/11/25

Approximate time of outage: Beginning at 7am Friday, 8/8 we will shut down affected servers and services. Beginning Monday morning, 8/11 we will bring the majority of the servers and services back online. Phase 1 of the data center move will begin at this time so some racks of compute nodes will be offline longer.

CCR resources affected by power outage:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Login & compile nodes
  • Portals: OnDemand

Empire AI resources affected by power outage:

  • Alpha cluster (all partitions)
  • Alpha login nodes

Resources NOT affected by power outage:

  • LakeEffect research cloud
  • Portals: ColdFront, Identity management

What will be done: Data center UPS will be decommissioned and electrical updates will be done to prepare for the installation of a new UPS.

Specific Effects to CCR users:
No available CCR or Empire AI HPC resources for the duration of this outage

July 2025 Downtime

Date of downtime: Tuesday, July 29, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, login-future - formerly vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None expected

June 2025 Downtime

Date of downtime: Tuesday, June 24, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None expected

May 2025 Downtime

Date of downtime: Tuesday, May 27, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None expected

April 2025 Downtime

Date of downtime: Tuesday, April 29, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None expected

March 2025 Downtime

Date of downtime: Tuesday, March 25, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates

Specific Effects to CCR users:
None expected

February 2025 Downtime

Date of downtime: Tuesday, February 25, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates
  • Documentation updates to be published: Update to CCR's software policy, new information on the "R" Statistical package for software release ccrsoft/2024.04, additional information on using QOS (Quality of Service) values for clarification, added a "change logs" section to our documentation site which includes downtime announcements. All future announcements will be in the documentation.
  • VSCode OnDemand app form changes - removed field to load additional software modules when starting. This doesn't propagate to the VSCode environment properly so it is being removed. Documentation is updated to reflect this change.
  • VMD OnDemand app form changes - added fields to allow users to request memory, CPUs, and GPUs rather than capping jobs at 64GB of RAM.
  • Release of new "Using Python at CCR" course in UB Learns. Self-register on the UB Learns website

Specific Effects to CCR users:
None expected

January 2025 Downtime

Date of downtime: Tuesday, January 28, 2025

Approximate time of outage: 7am-5pm

Resources affected by downtime:

  • UB-HPC cluster (all partitions)
  • Faculty cluster (all partitions)
  • Portals: OnDemand, ColdFront (temporarily), IDM (temporarily)

What will be done:

  • Reboot of all cluster nodes
  • Updates of front-end login nodes (login1, login2, vortex-future)
  • Infrastructure Updates
  • Release of additional software packages in ccrsoft/2024.04
  • Default software release on vortex-future changing to ccrsoft/2024.04 - you can pin your account to a specific software release, following the "Pin to a specific release" section of CCR's Software Environment Documentation
  • Updated container documentation to be published today
  • See the January section below for special update

Specific Effects to CCR users:
None expected

January - Special Update

Release of ccrsoft/2024.04

This new software release contains updated compilers, toolchains, and many of the most popular software applications used on CCR's systems.

Users may test this out using the command: module load ccrsoft/2024.04 You can use the vortex-future login node where this environment is now the default: vortex-future.ccr.buffalo.edu
The default software environment on the login (vortex.ccr.buffalo.edu), compile, and compute nodes, as well as in OnDemand, remains ccrsoft/2023.01 at this time.
You can pin your account to a specific release follow the "Pin to a specific release" section of CCR's Software Environment Documentation

NOTE: We will no longer be including machine learning type codes such as Pytorch, Transformers, etc in the CCR software environments. They change so quickly and are difficult to install with Easybuild, making keeping them up-to-date nearly impossible. Instead, we recommend using NVIDIA containers for these software packages. They are portable, up-to-date, and available in many configurations. You can access the NVIDIA container library here and CCR’s container documentation here. We've provided an example of a workflow in which you might pull an NVIDIA container and use a virtual environment to install additional packages. In the new Python course (see below) additional guidance will be provided.

New compute nodes available in the UB-HPC cluster

There are new compute nodes in the industry compute pool that are also available in the scavenger partition for academic users. They include:

  • 2 NVIDIA DGX nodes with 8 H100 GPUs and 2TB of RAM
  • 8 high-bandwidth memory nodes
  • 10 nodes with two NVIDIA L40S GPUs and 1TB of RAM
  • 8 nodes with 2TB of RAM
  • 62 standard compute nodes with 512GB of RAM

You can see the node hardware specs with the following commands: snodes all ub-hpc/scavenger |grep CPU-Platinum-8562Y

For instructions on how to use the scavenger partitions, please refer to CCR's job scavenger documentation.

For a full list of specs for the equipment in the UB-HPC academic partitions, see CCR's "ub-hpc" cluster hardware documentation.

For the industry partitions (and ub-hpc scavenger partition), see CCR's "ub-hpc" cluster hardware documentation.

Intro to CCR Course

A new course was published in UB Learns for the Fall 2024 semester to provide new users with background information on how to effectively and efficiently use CCR's HPC resources. The course was updated based on student feedback and republished for the Spring 2025 semester. All students taking a course that uses CCR are required to complete the Intro to CCR course first and provide their instructor with a certificate of completion before getting access to their course's allocations. Anyone using CCR for research purposes is highly encouraged to check out the course. You can self-enroll on the UB Learns website.

Using Python at CCR Course

A new course will soon be available in UB Learns that guides CCR users through the various methods for properly using Python in CCR's HPC environment. We'll make the link available here when the course is published. Check back here for updates!

CCR Migrating to TeamDynamix

You may be aware that UBIT has migrated to the TeamDynamix help desk platform, along with other IT units at UB. This spring CCR will be joining them! We'll make sure all CCR users are aware of any impact this will have on them well in advance of the cutover.