Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: January 9, 2026, 10:00 p.m.

End Date: January 12, 2026, 3:00 p.m.

The NCSA Identity portal, https://identity.ncsa.illinois.edu , will be unavailable from Friday, January 9th at 4PM to Monday, January 12th at 9AM due to planned power maintenance in the NCSA building. This outage will impact setting or resetting passwords for local NCSA user accounts. Processing of NCSA accounts, which require setting a password, may be paused or delayed during the outage. Please send questions or comments to NCSA via the NCSA help portal at https://help.ncsa.illinois.edu or by email to help@ncsa.illinois.edu.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: aces.tamu.access-ci.org

Start Date: January 8, 2026, 9:30 p.m.

End Date: January 23, 2026, 1:30 p.m.

The Globus Connect Service on TAMU ACES cluster has been restored. The Globus Connect Service on TAMU ACES cluster is down. We are working to restore the service. No estimated time for restoration at this time.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: January 7, 2026, 3:00 p.m.

End Date: January 7, 2026, 5:27 p.m.

This issue has been resolved. Please check your jobs. --- The /work file system is temporarily unavailable due to an unexpected power issue during data center maintenance. The /work is up but not accessible from Delta. A system reservation will be put in place to prevent new jobs from starting. An update will be provided when the issue is resolved. Please submit comments or questions via the NCSA Help portal at https://help.ncsa.illinois.edu or email to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) -Delta Project.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: kerberos.access-ci.org

Start Date: January 7, 2026, 12:50 p.m.

End Date: January 12, 2026, 1:00 p.m.

Due to power work at NCSA, one of the Kerberos replica servers (tg-kdc1) will be offline through the weekend. The master server will remain online and the replica server hosted at PSC should not be affected. Normal Kerberos service is expected to not be impacted by this work. If you notice any issues, please submit a help ticket https://support.access-ci.org/help-ticket

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta.ncsa.access-ci.org

Start Date: January 7, 2026, 12:00 p.m.

End Date: January 8, 2026, 4:00 a.m.

DeltaAI Resource Users, This is a reminder that the DeltaAI system will undergo planned maintenance on Wednesday, January 7th starting at 6AM (central time)and ending at 10PM (central time), approximately. Changes will include: - Upgrade of Slingshot FMS from 2.2.0 to 2.3.1. - Upgrade of Slurm from 23.02 to 25.11. - Maintenance on facility power feed for DeltaAI. Available environment modules will NOT change. Nevertheless, please check your jobs once the scheduler resumes job scheduling. During the maintenance period: - All compute and login nodes will be unavailable. - DeltaAI Open OnDemand will be unavailable. - Globus endpoints will remain available. The work and projects filesystems will also be accessible from Delta. - A reservation will be in place to prevent jobs from running into the maintenance period. Please be sure to check job time requirements as January 7th approaches so that jobs can be scheduled as the reservation drains the available nodes. Adjusting the time limit to account for the start time of the reservation will allow jobs to run. The resource scheduler will resume once the maintenance is complete. The other previously announced upgrades have been deferred: - OS upgrade from COS 24.8.0 / SLES 15 SP5 to SLES 15 SP6. - Upgrade of the HPE Cray Programming Environment from 24.07 to 25.09. - Upgrade of NVIDIA HPC SDK from 24.03 (CUDA 11.8 + 12.3) to 25.5 (CUDA 11.8 + 12.9). - Upgrade of NVIDIA driver from 550 series to 570 series. - Upgrade of Slingshot SHS from 11.0.0 to 13.0.0. We invite users to test their codes on nodes where these deferred upgrades have been applied. Please create a ticket for additional information. Available environment modules in the test environment WILL change. Thank you!

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: aces.tamu.access-ci.org, launch.tamu.access-ci.org

Start Date: January 6, 2026, 2:00 a.m.

End Date: January 6, 2026, 12:00 p.m.

Reminder: The ACES and Launch clusters will be inaccessible during 8p CST Monday January 5 until 6a CST Tuesday January 6 for the university to perform network maintenance. During this maintenance, job scheduling will continue and not be impacted on the clusters.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: January 2, 2026, 11:00 a.m.

End Date: January 2, 2026, 11:00 p.m.

On Friday, January 2, from approximately 6AM to 6PM ET, Jetstream2 will be offline during plumbing maintenance of the IU Bloomington Data Center. This maintenance will cause a widespread outage for all Jetstream2’s CPU, GPU, and large memory resources. We strongly advise all Jetstream2 users to save and close their work prior to January 2. You can preserve your work by: - Safely shutting down any active processes or jobs - Backing up essential data outside of Jetstream2 During the outage, please refer to the Jetstream2 status page (https://jetstream.status.io/) for the most up-to-date information. We appreciate your understanding and hope to mitigate any inconvenience this might cause. If you have any questions or concerns, please contact Jetstream2 Support at help@jetstream-cloud.org (mailto:help@jetstream-cloud.org).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: December 22, 2025, 1:00 p.m.

End Date: December 22, 2025, 5:00 p.m.

The Delta Slurm resource scheduler software and job database will be upgraded on Monday, December 22nd starting at 7AM and ending at 11AM, approximately. The Slurm software will be upgraded from 23.11.9 to 25.11.0. The most noticeable change will be how Slurm sees numa memory domains as sockets by setting the Slurm numa_node_as_socket parameter, which is helpful for CPU-GPU affinitization. If you use the sbatch, srun, or salloc -per-socket options such as --cores-per-socket or --gpus-per-socket then please check your jobs once the scheduler resumes job scheduling. See the Slurm socket affinity page (https://slurm.schedmd.com/gres.html#Socket_Affinity)for more information. After the maintenance: - The RH9 Slurm reservation will be removed and no longer needed when submitting jobs from dt-login02, dt-login03 or dt-login04. - Login node, dt-login01, and 1/4 of the compute nodes will be available for users who have not migrated to the upgraded OS. - The RH8 reservation will be set for jobs submitted from dt-login01. During the maintenance period: - All computes nodes will be unavailable. - Login nodes, file systems, Globus endpoints and other non-job related services will remain available. - The Delta OnDemand service will be available but interactive applications that require a compute node such as Jupyter notebook, VS Code server, and X Desktop will not be able to run. A reservation will be in place to prevent jobs from running into the maintenance period. Please be sure to check job time requirements as December 22nd approaches so that jobs can be scheduled as the reservation drains the available nodes. Adjusting the time limit to account for the start time of the reservation will allow jobs to run. The resource scheduler will resume once the maintenance is complete. Please submit comments or questions by using the NCSA Help portal (https://help.ncsa.illinois.edu/) or by email to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: launch.tamu.access-ci.org

Start Date: December 17, 2025, 3:00 p.m.

End Date: December 18, 2025, 6:00 p.m.

The TAMU Launch maintenance has completed. Previous message: TAMU Launch maintenance is extended till noon, Dec 18, 2025, due to unexpected issues. The TAMU Launch maintenance for Dec 17, 2025 has started. Original message: The TAMU Launch cluster will be unavailable from 9a to 11pm CST Wednesday December 17, 2025 for regular and storage maintenance.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: December 17, 2025, 1:00 a.m.

End Date: December 18, 2025, 2:30 a.m.

>>> Update 2>>> Dear Expanse User, The Lustre filesystem issues on Expanse have been resolved and we have mounted it back on the login nodes and on most compute and GPU nodes. The remaining nodes have been drained from the queue and will be rebooted to fix the mounts. The system is available for logins now and jobs can run on the nodes that are already fixed. Users are reminded that the Lustre filesystem locations (both scratch and projects) are **not** backed up. So please make offsite copies of anything critical. Also, it is important to control Lustre usage levels to help with performance and stability. Users are requested to clean up any data that is not in use for active runs or not needed in the near future. The filesystem **should not** be used as an archival location and users are strongly urged to make offsite copies of anything they want to keep long term. Thanks SDSC User Services Staff >>> Update 1>>>> Dear Expanse User, We are continuing to work on the Lustre filesystem issue. We were able to recover the metadata servers last night but there are still continuing issues with one of the metadata servers that we are working on. Additionally, logins to Expanse are being impacted (with errors regarding malformed TOTP) and we are looking into the source of the problem. Updates will be provided as we resolve both of these problems. Thanks SDSC User Services Staff >>>>> Dear Expanse User, We are currently seeing problems with the metadata servers of the Expanse Lustre filesystem. The issue started around 5PM (PT), December 16, 2025, and is currently resulting in filesystem access problems on all Expanse nodes. This will impact logins if your default path has Lustre locations in it. We will update once the issue is resolved. Thanks SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: December 11, 2025, 7:00 p.m.

End Date: December 12, 2025, 7:30 a.m.

The hardware and software issues encountered during the upgrade of the /work file system have been resolved. The file system is now available and the job scheduler was resumed. All but one root cause analysis has been completed, and we will work with vendor engineering on the remaining root cause. We apologize of the inconvenience caused by this unplanned outage. -Delta and DeltaAI projects. The Delta and DeltaAI /work file system is currently unavailable due to an unforeseen hardware failure during a software update. NCSA continues to work with the vendor to replace the hardware and complete the software update. New jobs are being blocked from starting, which includes jobs submitted via Delta and DeltaAI Open OnDemand portals. Additional updates will be sent when a return to service is known. We apologize for the inconvenience. -Delta and DeltaAI projects.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: December 11, 2025, 12:00 p.m.

End Date: December 11, 2025, 10:51 p.m.

UPDATE: December 11, 2025 05:51 PM ET As of 5:51pm ET, the maintenance work on Anvil has been completed and job scheduling has been resumed. If you encounter any issues post maintenance, please contact ACCESS Help Desk (https://support.access-ci.org/help-ticket). Original Post: The Anvil system will be unavailable on December 11th, 2025, from 7:00 AM to 6:00 PM ET for scheduled maintenance. During this maintenance, we will perform NVIDIA driver upgrades and rebuild Slurm with PMIx. Any Slurm jobs requesting a walltime that would extend past Thursday, December 11th at 7:00 AM ET will not start and will remain in the queue until after maintenance is completed. If you have any questions, please submit a ticket through the ACCESS Help Desk (https://support.access-ci.org/help-ticket).

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: jetstream2.indiana.access-ci.org

Start Date: December 2, 2025, 12:00 p.m.

End Date: December 3, 2025, 2:00 a.m.

On December 2, Jetstream2 engineers will be performing updates to Jetstream2's networking infrastructure to enable DNS resolution over IPv6. There is no expected downtime or impact to users for this maintenance. If you have any questions, please contact Jetstream2 Support by emailing help@jetstream-cloud.org

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: aces.tamu.access-ci.org

Start Date: November 18, 2025, 7:40 p.m.

End Date: November 18, 2025, 9:55 p.m.

TAMU ACES had a storage outage at 1:40 pm ~ 3:55 pm CST, Nov 18, 2025. ACES storage is now back to normal.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: identity.access-ci.org

Start Date: November 14, 2025, 2:00 p.m.

End Date: Not Specified

Dear ACCESS Community, The telephony credits available in the ACCESS DUO account unexpectedly ran out the evening of November 13, 2025. Those employing phone-call-based authentication for DUO have been getting denied by DUO as a result. We are working as quickly as possible to purchase additional credits to resolve this issue, but due to university purchasing policies there may be a delay before the credits become available. We regret the inconvenience of this sudden suspension of phone-call-based DUO authentication and direct all affected members of the community to select one of the several alternative methods of DUO authentication described at https://guide.duo.com/ (https://6fax6u7ab.cc.rs6.net/tn.jsp?f=0013UWUxb0WeMD2uTMupnTB5qASFg_xjuvzRv6psbGE-sHyF1oKWQHpPQPMXpJQTcmCsz4DD0QMrM_Zr_-CuaS16vi7zC0LbABPDsWRaN5qEDYmxKVBPwE-rPE1kySrVmUydiFBxNXlBNEAXRlH6OjmcA==&c=10ka7qroOSdY1qz0BlkWJMDU9CFOeZlw2SrKzSFTMJhvlleLJztGtg==&ch=U2-7NbP4COXJ3gs0XcWUE4Sc_viAYFAi2FOggUXfAc3nNQT82t1-RQ==). More than 90% of ACCESS community members already employ Duo Push (https://guide.duo.com/universal-prompt#duo-push) or a Platform Authenticator (e.g., passkey), and we strongly recommend that those currently configured for phone-call-based DUO authentication switch to one of these better methods. If you need assistance to update your DUO authentication method, please open a ticket at https://support.access-ci.org/help-ticket and select "I need help logging into ACCESS website" as the ticket type. We anticipate a surge in help desk tickets to help those affected, and we appreciate your patience as we work to help you as soon as we can. Regards, ACCESS Cybersecurity Team

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: identity.access-ci.org

Start Date: November 10, 2025, 2:30 p.m.

End Date: November 10, 2025, 2:40 p.m.

On November 10, 2025, the ACCESS Identity Provider (https://idp.access-ci.org/idp) (idp.access-ci.org) was updated to Tomcat v10.1.48 to address a security issue (https://nvd.nist.gov/vuln/detail/CVE-2025-61795).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: launch.tamu.access-ci.org

Start Date: October 25, 2025, 2:00 p.m.

End Date: October 27, 2025, 4:00 a.m.

Updated: 10:25 pm CDT, Oct 26, 2025. Launch cluster maintenance has completed. All queued jobs are running. The TAMU Launch cluster will be unavailable from 9a CDT Saturday October 25 to 11pm CDT Sun October 26 for regular and storage maintenance.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: dashboard.operations.access-ci.org

Start Date: October 24, 2025, 7:00 p.m.

End Date: October 24, 2025, 8:00 p.m.

The ACCESS Operations Dashboard was down for about an hour to apply fixes and upgrades. The following application(s) were affected: - Resource Provider Integration Dashboard (https://dashboard.operations.access-ci.org/dashboard/integration/) The following new applications were released during the upgrade: - ACCESS Staff Dashboard PREVIEW (https://dashboard.operations.access-ci.org/dashboard/integration/concierge/dashboard) - ACCESS Resource Integration Summary View (https://dashboard.operations.access-ci.org/dashview/resource_summary)

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta.ncsa.access-ci.org

Start Date: October 23, 2025, 2:04 p.m.

End Date: October 23, 2025, 6:00 p.m.

A facility event has caused a full outage on DeltaAI and a partial outage on Delta. We will return the affected nodes to service as soon as we can. -The Delta and DeltaAI projects

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: xdmod.access-ci.org

Start Date: October 16, 2025, 2:00 p.m.

End Date: October 17, 2025, 2:00 p.m.

The ACCESS XDMoD portal will temporarily be unavailable on 10/16/2025, from approximately 09:00 EST until the following day, 10/17/2025, 09:00 EST. The service will be completely unavailable for routine infrastructure updates. An update will be posted once the update is complete. We apologize for any inconvenience this might cause. UPDATE: The update has completed successfully and the service is available again.

Posted: March 20, 2026