Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: January 9, 2025, 6:00 p.m.

End Date: January 10, 2025, 11:00 p.m.

UPDATE 01/10 - The service has been fully restored at this time. There is a unplanned partial outage for ACCESS XDMoD from approximately 12:00 EDT on Thursday, January 9th until 17:00 EDT on Friday, January 10th. This is a partial outage that impacts viewing job performance data in the Single Job Viewer. An update will be posted once the issue is resolved.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: January 6, 2025, 3:00 p.m.

End Date: January 9, 2025, 3:00 p.m.

On Monday, January 6, 2025 at 9AM EST, Jetstream2 will begin a maintenance outage that will last through approximately 9AM EST on Thursday, January 9 (subject to change). This infrastructure maintenance is being done in conjunction with a third party vendor to update Jetstream2’s cooling system in order to accommodate a resource expansion. This maintenance outage will affect all primary Jetstream2 resources (CPU, GPU, Large Memory, and Storage) and user interfaces (Exosphere, CACAO). While existing instances at satellite regions will not be affected by the maintenance, the Exosphere and CACAO user interfaces will be inaccessible for all regions. When maintenance is complete, all instances will be returned to either a shelved or active state. If your instance was in an active, unshelved state, it will be brought back to that state. If it was shelved, it will remain shelved. NOTE: If your instance was in an errored, shutoff, or suspended state, it will be restored to an active state. We strongly advise all Jetstream2 users to review the states of their existing instances, as well as save and close their work prior to January 6. You can preserve your work by: - Safely shelving any active instances - Backing up essential data outside of Jetstream2 or creating images (https://docs.jetstream-cloud.org/general/instancemgt/#image) of your instances During the outage, please refer to the Jetstream2 status page (https://jetstream.status.io/) for the most up-to-date information. We appreciate your understanding and hope to mitigate any inconvenience this might cause. If you have any questions, please contact the Jetstream2 Support team at help@jetstream-cloud.org (mailto:help@jetstream-cloud.org?subject=).

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: identity.access-ci.org

Start Date: January 2, 2025, 7:00 p.m.

End Date: January 2, 2025, 7:30 p.m.

On January 2, 2025, the ACCESS CI Identity Provider (https://idp.access-ci.org/idp/) will be updated to the latest Tomcat version. No downtime is expected.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: January 2, 2025, 2:00 p.m.

End Date: January 2, 2025, 3:00 p.m.

On January 2, 2025, a new "Terms and Conditions" document (a.k.a., Acceptable Use Policy (https://access-ci.org/acceptable-use/)) will be configured for the ACCESS User Registry (https://registry.access-ci.org/). The ACCESS CI Acceptable Use Policy (AUP) requires users to re-acknowledge responsibilities regarding use of ACCESS systems and services every 12 months. Configuring a new Terms and Conditions document in the ACCESS User Registry essentially resets the 12-month clock so that the next time a user logs in to https://registry.access-ci.org/ , they will be prompted to read and agree to the AUP. Enforcement of the 12-month AUP agreement has not yet been put in place, so users should not notice any change in behavior for other ACCESS websites or HPC resources. However, users will begin to receive email notifications that they need to agree to the updated Terms and Conditions document. At some point in the future, enforcement of the 12-month AUP agreement will be put in place requiring users to visit https://registry.access-ci.org/ and agree to the new Terms and Conditions document. No downtime is expected for this configuration update.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: December 27, 2024, 12:00 a.m.

End Date: December 27, 2024, 9:00 p.m.

Delta Resource Users: A campus wide facilities event triggered a partial power outage at our data center. To safely restore full power to the system we must shutdown Delta now. We hope Delta will be back online at sometime tomorrow but will send an update in the morning when we know more. The Delta Team.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta.ncsa.access-ci.org

Start Date: December 27, 2024, 12:00 a.m.

End Date: December 27, 2024, 9:00 p.m.

DeltaAI Resource Users: A campus wide facilities event triggered a partial power outage at our data center. To safely restore full power to the system we must shutdown DeltaAI now. We hope DeltaAI will be back online at sometime tomorrow but will send an update in the morning when we know more. The DeltaAI Team.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: December 18, 2024, 10:30 a.m.

End Date: December 18, 2024, 7:00 p.m.

Update: The Lustre OSS with the problems was fixed and returned to service and the filesystem is accessible on Expanse now. Dear Expanse User We are currently seeing issues with the Expanse Lustre filesystem. This is leading to very slow responses or timeouts on access. We will update once the problem is resolved. SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: December 3, 2024, 2:00 p.m.

End Date: December 3, 2024, 2:00 p.m.

On December 3, 2024, the "Reply-to" mail header for email sent from the ACCESS User Registry (https://registry.access-ci.org/) will be changed from "registry@cilogon.org" to "support@access-ci.atlassian.net". This enables users who have problems with their new ACCESS IDs to easily open an ACCESS Help Ticket (instead of contacting CILogon support staff). Note that this only affects the "Reply-to" mail header. The "From" mail header will remain "registry@cilogon.org" so as not to affect any existing email filters.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: November 25, 2024, 4:30 p.m.

End Date: November 26, 2024, 1:00 a.m.

Dear Expanse User, A maintenance is ongoing to update part of the datacenter power infrastructure. This has impacted more Expanse racks than we originally anticipated (due to cooling considerations) and as a result some of the jobs running on Expanse were impacted. The jobs will get a NODE_FAIL error in Slurm so they will not be charged SUs. We will update once the maintenance is complete and the nodes are returned to service. In the interim, Expanse will have fewer available nodes than normal so wait times will likely increase today. We are sorry for the unexpected impact and please send us a ticket (either ACCESS or SDSC ticketing system) if you have any questions. Thanks SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: anvil.purdue.access-ci.org

Start Date: November 19, 2024, 6:00 a.m.

End Date: March 31, 2025, 5:00 a.m.

Anvil CPU is now offering a discounted exchange rate for researchers with Explore, Discover, and Accelerate allocations. The revised exchange rate is now 1 ACCESS credit = 1 Anvil CPU service unit (core hour). This new rate will be applicable for all exchanges to Anvil CPU starting 11/19/2024. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: globus.org

Start Date: November 2, 2024, 2:00 p.m.

End Date: November 2, 2024, 3:30 p.m.

Globus services will be unavailable for up to 1.5 hours beginning at 9:00 AM US Central on Saturday, November 2, 2024 due to planned maintenance that includes database upgrades. More details available here (https://www.globus.org/blog/maintenance-downtime-for-database-upgrades-saturday-november-2-2024).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-storage.ncsa.access-ci.org

Start Date: October 30, 2024, 11:00 p.m.

End Date: October 31, 2024, 1:00 p.m.

The /work file system emergency maintenance will continue overnight. A small portion of the /work file system remains unavailable. The job scheduler will remain paused until the /work file system is whole again. The NCSA Help Portal (http://help.ncsa.illinois.edu) is the preferred method to contact us. If you run into problems using the portal, you can still email help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu)for support. - The Delta and DeltaAI Project ________________________________________ The /work file system is experiencing issues at this time. We have contacted the vendor to help resolve the issue. This may impact new login sessions if you load a conda base from /work/hdd (formerly known as /scratch). Try to control-c once your login session makes it past the Duo prompt. An update will be sent when more information is available.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: October 30, 2024, 1:00 p.m.

End Date: November 1, 2024, 10:00 p.m.

Dear Bridges-2 User, We believe that all Bridges-2 Ocean issues have been resolved at this point. This is a reminder that Bridges-2 will be unavailable beginning at 8:00AM Eastern time on Wednesday, October 30 in order to install and test new hardware to add capacity to both the Bridges-2 Ocean and Jet filesystems. Maintenance is scheduled to end at Noon on Friday, November 1 but we hope to complete the upgrade and return the system earlier than that if possible. During the outage, Bridges-2, including all VMs and filesystems, will be unavailable. Please contact help@psc.edu with any problems or concerns.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: duo.access-ci.org

Start Date: October 28, 2024, 1:00 p.m.

End Date: October 28, 2024, 1:10 p.m.

On Monday, October 28, 2024, Duo Mobile Passcodes (https://guide.duo.com/universal-prompt#duo-mobile-passcode) for ACCESS will be reconfigured to use TOTP (https://en.wikipedia.org/wiki/Time-based_one-time_password) instead of HOTP (https://en.wikipedia.org/wiki/HMAC-based_one-time_password). The TOTP configuration will be pushed to users with the Duo security app (https://duo.com/product/multi-factor-authentication-mfa/duo-mobile-app#download) v4.49.0 or later. Users with older versions of the Duo security app will continue to use HOTP. For more information, please see TOTP vs HOTP (https://duo.com/blog/totp-vs-hotp).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: October 22, 2024, 11:00 a.m.

End Date: October 24, 2024, 1:00 a.m.

Delta Resource Users: This is a reminder of the scheduled maintenance event on Delta and DeltaAI on Tuesday October 22nd from 6AM to 8PM. There are some IMPORTANT updates to bring to your attention. 1. To address recent GPU XID errors, the Delta GPU nodes will move to a newer NVIDIA driver: moving from the 535 driver family to the 550 driver family. Initial testing did not indicate any need for recompilation or linking. Please check your GPU jobs as they run tomorrow for any unexpected behavior. 2. The maintenance will require the /scratch and /work/hdd file systems to be offline and unavailable during the maintenance. 3. The login nodes will need to be rebooted to unmount /scratch and /ime. 4. Please note that the /ime cache front-end to /scratch is currently being flushed and jobs that use the ime constraint will not be allowed to run before the maintenance. 5. The home and projects file system will be available, and jobs not using /scratch or /work/hdd will be allowed to run if submitted to one of the following reservations: On Delta, to run jobs that DO NOT require /scratch, please submit jobs using the no_scratch_requirements reservation by adding: --reservation=no_scratch_requirements to your sbatch, srun and Open OnDemand scripts. On DeltaAI, to run jobs that DO NOT require /work/hdd, please submit jobs using the no_workhdd_requirements reservation by adding: --reservation=no_workhdd_requirements to your sbatch, srun and Open OnDemand scripts. A final sync'ing of data on /scratch to /work/hdd will be done during the maintenance period. Upon maintenance completion, the Delta /scratch file system will be replaced by /work/hdd, which will also be accessible from DeltaAI, and the new NVME portion of /work called /work/nvme (currently available on DeltaAI) will also be accessible from Delta. On Delta a symbolic link will be set to link /scratch/hdd/abcd to /work/hdd/abcd to maintain functionality of existing scripts.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: October 21, 2024, 11:00 a.m.

End Date: October 22, 2024, 1:00 a.m.

UPDATE: October 21, 2024 09:00 PM EDT As of 9:00PM EDT, the maintenance work on Anvil has been completed and job scheduling has been resumed. If you encounter any issues post maintenance, please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket. Original Post: The Anvil system will be unavailable from Monday, October 21st at 7:00am to Tuesday, October 22nd at 6:00pm EDT, 2024 for scheduled maintenance. During the maintenance, we will perform cooling work for the system to integrate a new NSF funded AI resource into Anvil. Any Slurm jobs which request a walltime which would take them past Monday, October 21st at 7:00am EDT will not start and will remain in the queue until after the maintenance is completed. Anvil will return to full production by Tuesday, October 22nd at 6:00pm, EDT 2024. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: October 15, 2024, 2:00 p.m.

End Date: October 16, 2024, 1:00 a.m.

The FASTER cluster will be unavailable during maintenance from 9am to 8pm CDT on Tuesday October 15. A reservation is in place to prevent jobs from running past the start time of the maintenance period.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: October 15, 2024, 1:00 p.m.

End Date: October 21, 2024, 4:00 p.m.

The Bridges-2 issues are persisting in spite of dedicated efforts from both the PSC team and the hardware vendor. We are working towards a long term solution but the issue has been more complicated than we had expected. We understand your concerns and appreciate your patience while we work through the issue. This is of the highest priority for our team. We will let you know when we have a resolution and/or we will update you early next week with any developments. PSC User Services Team

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: October 14, 2024, 1:00 p.m.

End Date: October 14, 2024, 1:30 p.m.

On October 14, 2024 at 8:00am CDT, the ACCESS User Registry at https://registry.access-ci.org/ will be upgraded to COmanage Registry v4.3.5 (https://spaces.at.internet2.edu/display/COmanage/Registry+4.3.5+Release+Announcement). This is a maintenance release containing various bug fixes. No downtime is expected for this upgrade. The upgrade has been deployed to the TEST ACCESS User Registry instance at https://registry-test.access-ci.org/ . ACCESS OIDC Client Administrators can test their client by logging on to https://registry-test.access-ci.org/ , creating a new OIDC Client (or using an existing one), and reconfiguring their client with the OIDC client_id/client_secret associated with https://registry-test.access-ci.org/ . For questions or concerns with this update, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: October 13, 2024, 1:00 p.m.

End Date: October 13, 2024, 9:00 p.m.

Update: As of 5:00PM EST Oct 13, 2024, most compute nodes have been brought back online and Anvil has been back to normal operations. If you had jobs running around 09:00AM when the power interruption occurred, those jobs were terminated, so you may need to check the state of running jobs that were dependent upon work that was interrupted. Original Post: Shortly before 9:00AM EST Oct 13, 2024, Anvil has experienced a power interruption due to a campus power outage. It has caused compute nodes down and scheduling paused. Our engineers have arrived on campus and found additional impacts from today's power interruption. At of 12:18PM EST, scheduling has been resumed and we are still working on bringing remaining nodes back. We are at reduced capacity now but users should see stable systems at the moment. Additional efforts to restore full service are undertaken. Will post updates before the end of today. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket if you have any questions.

Posted: March 20, 2026