Infrastructure News Type: Reconfiguration

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: April 26, 2023, 6:00 a.m.

End Date: April 27, 2023, 11:30 p.m.

Update 04/27/2023: We have restored the overlay functionality on all Expanse nodes. Original message: In response to a security advisory, we have disabled the overlay kernel module on all Expanse nodes. This will prevent the use of some Singularity containers and possibly CVMFS mounts on Expanse. We are working on a mitigation approach and will update once this functionality is restored. Please send a ticket via ACCESS support (https://support.access-ci.org (https://urldefense.com/v3/__https://support.access-ci.org__;!!Mih3wA!DwrHmYkjt_poeQCAi67dZKlq73yxtpBB5_DWxOIFYI2TQKhrssmOLNDvj573Rwg2DVkkiWov67YTxMaz1S2M$)) or SDSC's support email (consult@sdsc.edu) if you have any questions. Thanks, SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: April 19, 2023, 1:00 p.m.

End Date: April 20, 2023, 1:00 a.m.

The Delta resource will undergo maintenance from 8:00AM to 8:00PM CDT on Wednesday April 19th, 2023. Additional notices will be sent if changes to the plan occur. During the maintenance period the following changes will be made: - Operating system (OS) on the Delta login and compute nodes will be updated from RHEL 8.4 to 8.6 with no expected impact to the existing software stack. - Software for the Delta high-speed network (HSN) switches and fabric manager will be upgraded. - Lustre file system client software will be updated to address known issues. - NVIDIA GPU driver will be upgraded to CUDA 12, while the default CUDA software module will remain at CUDA 11.6.1 until the next planned maintenance. - The hostname command will provide the public-facing hostname and IP address rather than the internal hostname and IP address. - Workflows that use the hostname --ip command need to check that references to 172.x are changed to the matching 141.x addresses. All Delta resources will be unavailable during the maintenance period including: - Delta login nodes - unavailable - Delta compute nodes - unavailable - Delta services - Open OnDemand - unavailable - Delta Globus Online endpoint - unavailable All running jobs will have been drained in advance of the maintenance. Queued jobs will persist and be eligible to run once maintenance is complete. A follow-up message will be sent once maintenance is complete. Please send questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta in the subject.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: tickets.access-ci.org

Start Date: April 3, 2023, 10:00 a.m.

End Date: April 3, 2023, 2:45 p.m.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: March 30, 2023, 10:45 p.m.

End Date: March 31, 2023, 2:02 a.m.

UPDATE: March 31, 2023 10:02pm EDT As of 10:02pm EDT, the Anvil system has been returned to normal service. Job queues have been enabled and job scheduling has been resumed. We apologize for the disruption of service. Please report any issues through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket ORIGINAL: March 31, 2023 6:45pm EDT The Anvil cluster began experiencing issues with its scratch filesystem around 6:45pm EDT. Access to scratch directories may be slow or hang. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide an update by 10pm Eastern time, or sooner as we have more information. This outage is being tracked at https://www.rcac.purdue.edu/news/5704

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: ranch.tacc.access-ci.org

Start Date: March 26, 2023, 1:00 p.m.

End Date: March 26, 2023, 7:00 p.m.

TACC network infrastructure will not be available from 8:00 AM to 2:00 PM (CDT) on Sunday, 26 March 2023. Network maintenance will be performed during this time. Please submit any questions you may have via the TACC User Portal. https://portal.tacc.utexas.edu/tacc-consulting

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: tickets.access-ci.org

Start Date: March 24, 2023, 2:00 p.m.

End Date: May 1, 2023, 4:59 a.m.

On March 24, 2023, ACCESS introduced a new ticket system based on Jira Service Management (JSM), replacing the previous Resource Tracker (RT) system. ACCESS users can submit tickets following the links on the ACCESS Support site at: - https://support.access-ci.org As additional documention is available it will be listed here: - General Ticket System FAQ (https://operations.access-ci.org/faqs) -- ACCESS Support and Operations

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: March 21, 2023, 12:00 p.m.

End Date: March 22, 2023, 12:00 a.m.

As of 8:27pm EDT, engineers have completed maintenance and have returned the Anvil system back to normal service. All queues have been enabled and jobs have resumed scheduling. Please see detailed list of changes (https://www.rcac.purdue.edu/news/5689) in a separate article. Please report any issues through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket ————————————————————— We have extended the maintenance until 8PM ______________________________________________________________ The Anvil system will be unavailable Tuesday, March 21, 2023 from 8:00am - 6:00pm EDT for scheduled maintenance. Any Slurm jobs which request a walltime which would take them past Tuesday, March 21st, 2023 at 8:00am EDT will not start and will remain in the queue until after the maintenance is completed. How does this maintenance impact you? - The default queue will be changed from wholenode to shared. - NVIDIA GPU driver will be updated and CUDA 12.0.0 will be added as a module. Anvil will return to full production by Tuesday, March 21st, 2023 at 6:00pm EDT. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: February 24, 2023, 3:00 p.m.

End Date: February 26, 2023, 4:50 a.m.

Anvil scratch has been returned to a fully operational state , and scheduling has been resumed. ** Update 5:40 PM 02/25/23" Our engineers continue to collaborate with vendors to return Anvil scratch to service. Work will continue throughout the evening and possibly overnight. We will send an update by 10 AM EST 02/26/23 _______________________________________________________________________________________________ ** Update 9:30AM 02/25/23" Logins to Anvil via Open OnDemand gateway (https://gateway.anvil.rcac.purdue.edu/) are affected by the outage. Other login methods (direct SSH (https://www.rcac.purdue.edu/knowledge/anvil/access/login) and Thinlinc Remote Desktop (https://www.rcac.purdue.edu/knowledge/anvil/access/login/thinlinc)) continue to function as expected provided you have previously enabled (https://www.rcac.purdue.edu/knowledge/anvil/access/login/sshkeys) SSH keys-based authentication for your account. Storage engineers continue to work with the vendor to return scratch and project filesystems to full functionality. Scheduling remains paused while we address the problem. We will provide an update by 8:00PM EST 02/25/23 **Update 8PM 02/24/23** Storage engineers continue to work with the vendor to return scratch to full functionality. Scheduling remains paused. We will provide an update by 10AM EST 02/25/23 Around 10AM EST on Friday, February 24th, Anvil scratch began experiencing performance issues.. Storage engineers are working with the storage vendor for a resolution. Scheduling has been paused until scratch is returned to service

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: lyric.uky.access-ci.org

Start Date: February 24, 2023, 9:00 a.m.

End Date: February 25, 2023, 12:00 a.m.

The KyRIC system will be offline on Friday, 2/24/2023, 3AM-6PM ET due to maintenance of the storage system. All nodes and data operations will not be available in that period. We aim to bring the system back on the same day. KyRIC Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: February 21, 2023, 4:00 p.m.

End Date: February 22, 2023, 1:00 a.m.

We will have a maintenance period on Expanse 8AM-5PM (PT), Feb 21, 2023. During this maintenance, we will work on the NFS home filesystems. We have a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation. Thanks SDSC User Support Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: February 8, 2023, 3:00 p.m.

End Date: February 9, 2023, 2:00 a.m.

FASTER will be unavailable from 9a to 8pm for software and hardware maintenance.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: stampede2.tacc.access-ci.org

Start Date: January 28, 2023, 7:19 p.m.

End Date: January 29, 2023, 5:59 a.m.

Stampede2 is currently experiencing issues with the /scratch filesystem and queues will remain down for a prolonged duration as the issues are investigated.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: December 24, 2022, 6:01 a.m.

End Date: December 27, 2022, 5:59 a.m.

The /projects file system has been returned to service and is again available. Users who had blocks on their workloads due to dependencies on the /projects file system have been removed. The slurm Feature for projects has been added back to the compute nodes. We apologize for the delay in getting the /projects file system back in service. Please send questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta in the subject or message body. -------------------------------------- The Delta projects (/projects) file system is currently unavailable. NCSA staff are working with the vendor to return the file system as soon as possible. Jobs already in the queue have been put on hold that reference the /projects as a slurm Feature. Users with jobs in the queue are currently not able to run jobs if they have jobs that have a slurm job variable which references /projects such as a working directory, a submit directory or a location for job stdout/stderr. Jobs that do not require the /projects or /taiga file system as a Feature will be allowed to run. See File System Dependency Specification for Jobs (https://wiki.ncsa.illinois.edu/display/DSC/Delta+User+Guide#DeltaUserGuide-FileSystemDependencySpecificationforJobs) for information on how to specify file systems as a job Feature. We will provide updates on the status of the projects file system as they become available. Please send questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta in the subject or message body.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: December 20, 2022, 2:00 p.m.

End Date: December 20, 2022, 3:00 p.m.

Update: Maintenance is complete. The Delta core router will undergo preventative maintenance from 8:00AM to 9:00AM CST on Tuesday December 20th, 2022. The Delta core router provides connectivity between the Delta resource and the outside world. External access to or from Delta compute resources (dt-login and compute nodes) and to Delta services (Open OnDemand, Globus Online endpoint) will be unavailable during the maintenance period. Non-interactive jobs should not be impacted as internal connectivity between compute nodes and storage will not be affected. A follow-up message will be sent once maintenance is complete. Please send questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta in the subject or message body. --Delta Project Office

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: November 30, 2022, 7:01 p.m.

End Date: December 13, 2022, 4:59 a.m.

The work to bring the new Allocations information online has now been completed. ---------------------------------------------------------------------------------------------------------- ACCESS XDMoD (https://xdmod.access-ci.org/ ) may experience some temporary performance degradation while work is done to bring the new Allocations information online. Chart load times may be slower than usual. Another notice will be sent when the work is complete.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: November 30, 2022, 1:45 a.m.

End Date: November 30, 2022, 2:23 p.m.

UPDATE - 9:23am Eastern Network engineers have restored access to the 149.165.168.0/21 CIDR block. While they are still working on the failed networking equipment, that network is now routed via other means and connectivity appears to be fully restored. ---- UPDATE - 10:53pm Eastern There is an outage on the research network hardware that presently serves the 149.165.168.0/21 routes for Jetstream2. Network engineers are assessing the issue. There is no estimated time for return to full operations at this time. ----- It appears that the network routing outside Jetstream2 for our newest subnet (149.165.168.0/21) is no longer functioning. We have alerted our Network Operations Center for diagnosis and resolution as it is outside of Jetstream2. VMs with IPs in other ranges (i.e. anything under 149.165.168.0 are not presently affected). We will update as soon as we have more information.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: November 10, 2022, 6:13 p.m.

End Date: November 13, 2022, 6:13 p.m.

Jetstream2 is upgrading compute nodes and infrastructure in preparation for an OpenStack upgrade in December. You may see brief pauses in VMs as they are live migrated to other compute nodes. There should not be downtimes. GPU nodes are being updated as they empty since we cannot presently live migrate GPU instances. We will likely have a planned outage in coming weeks for GPU node maintenance and will notify users here and via https://jetstream.status.io/ as well.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: ranch.tacc.access-ci.org

Start Date: November 1, 2022, 2:00 p.m.

End Date: November 2, 2022, 2:00 a.m.

Ranch will be undergoing tape storage expansion from 9:00 AM (CDT) until 9:00 PM on Tuesday, 1 November 2022. Ranch will be up for data ingestion but unavailable for data recovery during this time. Users will be delayed in reading their older data off of tape during this downtime. Please submit any questions you may have via the TACC User Portal. https://portal.tacc.utexas.edu/tacc-consulting

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: identity.access-ci.org

Start Date: October 28, 2022, 7:10 p.m.

End Date: October 28, 2022, 9:15 p.m.

Duo has been re-enabled on the ACCESS IdP at https://idp.access-ci.org/ (https://idp.access-ci.org/idp/) . ------------------------------------------------- Previously: Due to ongoing issues with Duo licensing, Duo has informed us that they will be downgrading our service until the issue gets resolved. As a result we will have to suspend the Duo authentication component of the ACCESS IdP at idp.access-ci.org. We expect this to be a seem-less transition and for web authentication to continue to function with your ACCESS identity. Any issues encountered should be reported to the ACCESS ticket system at: https://support.access-ci.org/open-a-ticket

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: October 20, 2022, 2:00 p.m.

End Date: October 21, 2022, 1:00 a.m.

The FASTER cluster will be unavailable from 9am to 8pm on Thursday, October 20th. Software and hardware maintenance will be performed during this downtime. Queued jobs will not start if they overlap with this maintenance window. For questions or concerns, please e-mail help@hprc.tamu.edu

Posted: March 20, 2026