Communications Archive

Link to this item

Link to this item

Link to this item

We are pleased to announce that OARC and the Office for Research reached the agreement to continue support for MOE license for the year 2022-2023.      
After this year, the management of the license, including the cost and hosting of the license server will be transferred entirely to the Office for Research.
OARC will continue the technical support of MOE on the OARC clusters and assist MOE users with scientific consultations.

Please contact us at help@oarc.rutgers.edu if you have any questions.

Thanks,

Vlad Kholodovych, PhD
Sr Scientist,Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Link to this item

Amarel Cluster Users,

Amarel has been returned to service. All jobs that were waiting on queue are now being allocated resources. New jobs can be submitted for resources. Thank you again for patience, and happy computing.

Best,

Link to this item

Amarel Cluster Users,

The maintenance tasks are complete, but we will need to extend the reservation to run some additional tests before we return Amarel to regular service.

We anticipate that these tests will be complete before 9pm.

I apologize for the inconvenience. A message will be sent once the tests are complete.

Thank you,

Paul Guillermo Arias, Ph.D.

 Senior Scientist, Office of Advanced Research Computing

Link to this item

Amarel Cluster Users,

The monthly maintenance for Amarel will be on Tues, August 16 to Wed, August 17:

*** Start: Tues, August 16, 2022, 8am ***

*** End: Wed, August 17, 2022, 7pm ***

The maintenance will include the following hardware and software tasks:

Correct Exxact/RAVE IPMI console redirect problems

Resolve UID/GID collisions between Warewulf master machines and images

DNS work on global_old_systems August Maintenance

Any wiring changes/installations needed to support Amarel Phase V(b)

Change ownership of files for users that have UID/GID collisions with Rutgers LDAP UID/GIDs

Deprovision amareln:/etc/modules-load.d/ib.conf from WW

Operating system patching/GPFS upgrade to 5.1.2-5 for Warewulf images, 202208

Upgrade infrastructure/non-stateless machines to GPFS 5.1.2-5

Remove Ganglia from 202208 Warewulf images

Disable pam_pfrintd.so PAM module in 202208 Warewulf image

Operating system patching for CentOS/Debian Infrastructure, 202208

A message will be sent to all uses once the maintenance has been completed.

Thank you,

Paul Guillermo Arias, Ph.D.

 Senior Scientist, Office of Advanced Research Computing

Link to this item

Link to this item

Link to this item

Amarel Cluster Users,

Summary: Amarel returned to service, but pending jobs were purged.

Amarel has been returned to service. The issue regarding connectivity to storage has been resolved.

Unfortunately, the pending jobs on the queue were purged. You will need to resubmit your jobs to secure Amarel resources again.

Thank you again for patience, and happy computing.

Best,

Paul Guillermo Arias, Ph.D.

 Senior Scientist, Office of Advanced Research Computing

Link to this item

Amarel Cluster Users,

Summary: Maintenance window extended until storage access issue resolved.

Due to a connectivity issue with the storage system, we will need to delay returning Amarel beyond our originally scheduled window. Details of the work that was to be done is included here: https://oarc.rutgers.edu/communications-archive/07-18-2022/

The issue does not affect any data that resides in the storage system. It only affects the connectivity to it. OARC is in communications with the system provider to resolve the problem.

 We will provide regular updates and return Amarel usage to you as soon as possible.

Thank you for your patience.

Best regards,

Link to this item

Link to this item

Amarel Cluster Users,

The monthly maintenance for Amarel will be on Tues. July 19 and Wed. July 20:

*** Start: Tues, July 19, 2022, 8am ***
*** End: Wed, July 20, 2022, 7pm ***

The maintenance will include the following hardware and software tasks:
Hill Ethernet – reconcile switch uplinks and spanning-tree
Configure networking and monitoring for existing Hill Center PDUs
Upgrade Lenovo Amarel equipment to Lenovo Best Recipe 22B (https://support.lenovo.com/us/en/solutions/HT513895)
Migrate GPFS client cluster configurations to release=LATEST (minReleaseLevel 5.1.2.0)
Intermittent H/W problems on amarel[1-2]; physically reseat and expel dust around connector

Operating system patching for CentOS/Debian Infrastructure, 202207
The following systems will be patched:
strep
chernobog
biomech
leif2, saul
squid
arba, kirk
wwulfc

A message will be sent to all uses once the maintenance has been completed.

Thank you,

Paul
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

Amarel users,

Yesterday, July 13, 2022, we experienced unexpected rebooting of amarel login servers in Piscataway. Amarel1 rebooted at 19:47 and amarel2 at 23:23. We are not sure yet what caused the interruption. We received some network switch alerts in the same timeframes but unsure if issues are related. OARC team is investigating the cause.
If you had any processes running, e.g. interactive jobs or connections using terminal multiplexers (screen, tmux) on the login nodes at that time they may be lost.
Please check your connections.

All jobs submitted through slurm scheduler, both running and queued, should not be affected by this interruption.

Vlad Kholodovych, PhD
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Users,

Maintenance is complete and the cluster is back in service.

Please let us know if you spot any problems.

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Cluster users,

The monthly maintenance for Amarel will be on Tuesday and Wednesday:

*** Start: Tues, June 21, 2022, 8am ***
*** End: Wed, June 22, 2022, 7pm ***

A message will be sent to all uses once the maintenance has been completed.

Thank you,

Paul Arias
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Users,

Maintenance is complete and the cluster is back in service.

Please let us know if you spot any problems.

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

ATTENTION STUDENTS – undergrad or grad students from any discipline:

The CAREERS (Cyberteam to Advance Research and Education in Eastern Regional Schools) Cyberteam Program is an initiative led by Yale University and funded by the NSF to grow the pool of Research Computing Facilitators (RCFs) available to support researchers at small and midsized institutions in our region: Connecticut, Delaware, New Jersey, New York, Pennsylvania, and Rhode Island.

We are looking for graduate or undergraduate students who have some computing experience and are interested in learning about research computing facilitation through an experiential training program where participants (virtually) join the project teams of researchers working on a compute-intensive or data-intensive projects for 3-6 month engagements. If matched to a project, you will be assigned an experienced mentor, and together you will provide assistance on research computing and/or data science to a researcher to help to move their project forward. Typically, a time commitment of 10-20 hours per week is required, and students receive a trainee stipend ranging from $3000-$6000 depending on the nature of the project and the experience or skills of the student.

** For many students, this could be a summer project opportunity that looks great on a resume.

Currently available projects can be viewed at the CAREERS Cyberteam Portal (https://careers-ct.cyberinfrastructure.org). We especially encourage women and minorities to participate, as increasing diversity in the RCF community (and in advanced computing and data science, more generally) is an important goal for this program.

To apply to become a student facilitator (or a mentor, if you have the experience) for any of the projects listed on the Portal, or to be considered for future projects, click on “Join the Team” to register for the CAREERS Cyberteam and create a profile on the Portal. If you prefer, you can also send an email expressing your interest along with a brief bio to our program managers, Kaylea Nelson kaylea.nelson@yale.edu and Julie Ma, jma@mghpcc.org.

Of course, you can also reach-out to me with questions. I’m already serving as a mentor for a few of the projects.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Users,

The monthly maintenance for Amarel will be on next Tuesday and Wednesday:

*** Start: Tues, May 17, 2022, 8am ***
*** End: Wed, May 18, 2022, 7pm ***

A message will be sent to all uses once the maintenance has been completed.

Cheers,
Ehud (Udi) Zelzion,
Senior Scientist, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Phase 5A Owners,

In a message from February (see below), I requested that the Phase 5A nodes be loaded carefully, not using all cores.

That concern has been obviated. Please feel free to use the full capacity of those nodes — no need for restrained loading.

Thank you for your patience.

Galen

From: Galen Collier
Sent: Tuesday, February 8, 2022 1:59 PM
To: Galen Collier <galen.collier@rutgers.edu>
Subject: Amarel Phase 5A Nodes Entering Production</galen.collier@rutgers.edu>

Amarel Phase 5A Owners,

Your Phase 5A nodes and storage are nearly ready for release. Each you will be hearing more from me about this within the next 24 hours.

We’re facing a power distribution issue in the Hill Data Center that could impact the racks where these new compute nodes reside. Rather than delay release, I want to explain the situation and encourage you to simply be mindful of the issue until it’s resolved (hopefully very soon):
* The new racks are being fed by a power system that’s missing some back-ordered components, so it’s currently lacking the capacity we prefer to have for racks of nodes that may reach near-peak power consumption.
* Half of the in-rack power distribution equipment is back-ordered due to widespread shipping delays, so the entire electrical load on those racks travels a single path.
* This configuration is susceptible to drawing too much current and triggering an automatic shut-off. That means we must run at a slightly lower power draw than usual, which in-turn means that we need to try to limit the load your jobs place on those nodes until this issue is resolved.
* Until the needed parts arrive, please try to avoid fully loading your new Phase 5A nodes. This can be done by using less than the full 64 cores onboard. We’ve been running tests with sample jobs that efficiently use all 64 cores and the system seems stable, but spikes in utilization from poorly-configured jobs could risk an isolated power outage for these racks.

Of course, let me know what questions or concerns you have.

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Cluster users,

The earlier reported issue with the slurm controller failure is resolved now. You may continue to use Amarel cluster as usual.

As mentioned before, we noticed that not all jobs reported with sacct command as RUNNING at the time of the first announcement ran to the full completion. Please check outputs from your recently submitted jobs. As a precaution we would recommend to re-submit these jobs again.

We apologize for inconvenience and for the delay that affects your research.

Message sent on: Thu Feb 17 14:47:39 EST 2022

Amarel Cluster users,

We continue to experience the slurm controller failure issues. Our tech team successfully identified the cause of the problem and now working on patching it.

Unfortunately though, we noticed that not all jobs reported with sacct command as RUNNING at the time of the first announcement ran to the full completion. Please check outputs from your recently submitted jobs. As a precaution we would recommend to re-submit these jobs again when the normal operation of the cluster resumes.

We apologize for inconvenience and for the delay that affects your research.

Message sent on: Thu Feb 17 12:07:02 EST 2022

Amarel Cluster users,

This morning, February 17, 2022, we experience an unexpected failure of slurm controller that prevents new jobs to be submitted to the cluster. Already submitted and running jobs are not affected.

You can check the status of your currently running jobs by using the following command: sacct -u $USER

Our tech team is aware of the problem and investigating the cause. We are working to resolve the problem as soon as possible. We will update you when the issue is resolved. We apologize for inconvenience that it may cause.

Message sent on: Thu Feb 17 09:11:46 EST 2022

Link to this item

Amarel Cluster users,

We continue to experience the slurm controller failure issues. Our tech team successfully identified the cause of the problem and now working on patching it.

Unfortunately though, we noticed that not all jobs reported with sacct command as RUNNING at the time of the first announcement ran to the full completion. Please check outputs from your recently submitted jobs. As a precaution we would recommend to re-submit these jobs again when the normal operation of the cluster resumes.

We apologize for inconvenience and for the delay that affects your research.

Message sent on: Thu Feb 17 12:07:02 EST 2022

Amarel Cluster users,

This morning, February 17, 2022, we experience an unexpected failure of slurm controller that prevents new jobs to be submitted to the cluster. Already submitted and running jobs are not affected.

You can check the status of your currently running jobs by using the following command: sacct -u $USER

Our tech team is aware of the problem and investigating the cause. We are working to resolve the problem as soon as possible. We will update you when the issue is resolved. We apologize for inconvenience that it may cause.

Message sent on: Thu Feb 17 09:11:46 EST 2022

Link to this item

Amarel Cluster users,

This morning, February 17, 2022, we experienced an unexpected failure of slurm controller that prevents new jobs to be submitted to the cluster. Already submitted and running jobs are not affected.

You can check the status of your currently running jobs by using the following command: sacct -u $USER

Our tech team is aware of the problem and investigating the cause. We are working to resolve the problem as soon as possible. We will update you when the issue is resolved. We apologize for inconvenience that it may cause.

Message sent on: Thu Feb 17 09:11:46 EST 2022

Link to this item

Amarel Phase 1 Owners,

As announced previously, Amarel Phase 1 compute nodes and storage expired in July 2021.

We postponed removal of the Phase 1 hardware, in part, to compensate for delays we’ve faced in acquiring and provisioning Phase 5A. Now that the Phase 5A equipment is in-production, we need to remove the Phase 1 equipment as soon as possible.

We are planning to remove Phase 1 nodes and storage from owner allocations during next week’s maintenance window (Feb 15-16).

How will these changes be implemented?

(Case 1) Owners of later-phase nodes (Phase 2-5) will see Phase 1 nodes removed from their job partitions. Storage quotas for the /projects filesystem will be reduced from current values, accounting for the loss of Phase 1 storage. Note that actively owning any number of Amarel compute nodes (i.e., being an Amarel owner) qualifies one for 1 TB of /projects storage, so there is no situation where an owner of compute nodes does not have any /projects storage.

If your /projects storage utilization is over your quota after a quota adjustment, that renders your /projects storage space read-only (unwriteable) until you can move or delete enough data to get back under quota.

(Case 2) Those who own only Phase 1 nodes and storage will simply lose access to those resources. Please note that loss of /projects storage will result in loss of data stored there. If this case applies to you, I urge you to begin moving your data to an appropriate alternative storage resource immediately. Box.rutgers.edu is a good option, but the data transfer process may be slow.

Of course, let me know what questions or concerns you have.

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

January 5, 2022

Amarel Users,

Please be advised that the monthly Amarel maintenance that was originally scheduled for January 18-19 has been rescheduled to January 11-12.

Please let us know if you have any questions.

Link to this item

Amarel Users,

Please be advised that the monthly Amarel maintenance that was originally scheduled for January 18-19 has been rescheduled to January 11-12.

Please let us know if you have any questions.

Ehud (Udi) Zelzion
Senior Scientist, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Link to this item

Link to this item

Amarel Cluster users,
Due to an interruption in power in Newark around 12pm, today, December 20, 2021 we experience loss of performance for AmarelN cluster in Newark. Some cluster components are not reachable, i.e. Open On Demand interface and a job scheduler. As a result jobs submitted before 12pm today may be lost and new jobs can’t be scheduled at this time.
We are working to resolve the problem as soon as possible. Meanwhile if you can please try to use amarel.rutgers.edu cluster and ondemand.hpc.rutgers.edu in Piscataway.
We apologize for inconvenience that it may cause.

Link to this item

Amarel users,
Please be advised that all OARC staff members will be attending multiple full day trainings from SchedMD (a slurm scheduler vendor) next week, Tuesday-Thursday, December 14-16, 2021.
Both the technical support and the helpdesk communication may be slow during this time. We apologize for inconvenience that it may cause.

Link to this item

Link to this item

Wednesday, November 17, 2021, 7:05 am

Morning,

Monday, November 15, 2021, 6:46 PM

Dear Amarel-N users,

There will be a scheduled overnight power outage occurring at the Rutgers Newark campus which will affect OARC services and Amarel-N use.

Amarel-N will be unavailable beginning Tuesday, 11/16/2021 at 8:00 PM.
OARC services should be back on-line Wednesday, 11/17/2021 at 8:00 AM.

Please let us know what your questions are.

Sue

Susan Oldenburg
Research Support/GIS Specialist
OARC 40.521456, -74.461249
NBL 40.505103, -74.452339
gis-help@rutgers.edu

Friday, November 12, 2021, 4:40 pm

Amarel Users,

We just learned that all buildings on the Newark campus will be affected by a 20 minutes power outage sometime between 1:00 am – 3:00 am on Wednesday 11/17/2021.

We are still unsure how the outage will affect OARC’s systems in Newark (Amarel-N) and will provide updates as we receive more information.

Link to this item

Monday, November 15, 2021, 6:46 PM

Dear Amarel-N users,

There will be a scheduled overnight power outage occurring at the Rutgers Newark campus which will affect OARC services and Amarel-N use.

Amarel-N will be unavailable beginning Tuesday, 11/16/2021 at 8:00 PM.
OARC services should be back on-line Wednesday, 11/17/2021 at 8:00 AM.

Please let us know what your questions are.

Sue

Susan Oldenburg
Research Support/GIS Specialist
OARC 40.521456, -74.461249
NBL 40.505103, -74.452339
gis-help@rutgers.edu

Friday, November 12, 2021, 4:40 pm

Amarel Users,

We just learned that all buildings on the Newark campus will be affected by a 20 minutes power outage sometime between 1:00 am – 3:00 am on Wednesday 11/17/2021.

We are still unsure how the outage will affect OARC’s systems in Newark (Amarel-N) and will provide updates as we receive more information.

Link to this item

Amarel Users,

We just learned that all buildings on the Newark campus will be affected by a 20 minutes power outage sometime between 1:00 am – 3:00 am on Wednesday 11/17/2021.

We are still unsure how the outage will affect OARC’s systems in Newark (Amarel-N) and will provide updates as we receive more information.

Have a great weekend,

Ehud (Udi) Zelzion,
Senior Scientist, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Amarel Users,

It appears that work in or near the datacenter caused power to one of our core switches to be interrupted at 16:49. That switch rebooted and returned to service at 16:52, but the interruption disrupted our network fabric, causing processes (or jobs) using /projectsp, /home, /opt/sw, or /projects (the one with the symlinks) to be momentarily interrupted.

The problem has been resolved, but some users (or some jobs) may have noticed an interruption in filesystem activity at that time.

Please let me know if you have questions.
Galen
Galen Collier, PhD

Director, Research Support

Office of Advanced Research Computing (OARC)

Link to this item

Amarel CCB Nodes and Storage Users,
The nodes comprising the ccb_1 job partition and the storage file set at /projects/ccb have been in-service since July 2017 and they’re about 3 months overdue for retirement. We must now remove those resources from service.
On Thursday, Nov. 18 (two weeks from today) at 12:00 PM ET, the ccb_1 job partition will no longer accept jobs and it will be removed once its queue has cleared.
On Thursday, Dec. 2 (four weeks from today) the /projects/ccb storage file set will be purged and removed. There are currently about 72 GB of data there now.
Please adjust plans for any ongoing work accordingly and remove any data you wish to keep well before Dec. 2nd.
If you have questions or need help, let me know.
Galen
Galen Collier, PhD

Director, Research Support

Office of Advanced Research Computing (OARC)

Link to this item

Amarel is back in service now.

Maintenance is complete.

As always, please let us know if you spot any problems.

Galen

October 20, 2021

Amarel Users,

Our OCT maintenance window (underway now) must continue longer than scheduled so that our IBM Spectrum Scale storage system vendor can continue with some troubleshooting and configuration work.

We will follow-up with further details as we have them and we will return Amarel to normal service as soon as we are able to do so.

Please let me know if you have questions.

Galen

Link to this item

Amarel Users,

Our OCT maintenance window (underway now) must continue longer than scheduled so that our IBM Spectrum Scale storage system vendor can continue with some troubleshooting and configuration work.

We will follow-up with further details as we have them and we will return Amarel to normal service as soon as we are able to do so.

Please let me know if you have questions.

Galen

Link to this item

Tuesday, August 24, 6:30 pm

Caliburn Users,

Caliburn and ELF will be permanently removed from service on Tuesday, August 31.

If you haven’t already, please begin removing any data you wish to retain. The storage systems will be unavailable after these systems are taken offline and data lost will be unrecoverable.

Most of the options for moving data to/from the Amarel system will work for Caliburn as well, so some guidance can be found here: https://sites.google.com/view/cluster-user-guide#h.3jyxmx35y8y2

Galen

Thursday, August 12, 2021 4:40 PM

Caliburn Users,

Further analysis and testing have revealed that the cooling system in the data center is not sufficiently stable to permit continued operation of the Caliburn and ELF systems. As a result, we are very sad to report that those systems will be permanently removed from service on August 31.

Please plan any remaining workflows accordingly and begin removing valuable data from those systems. Data residing there on or after August 31 will be lost.

Galen

Link to this item

Caliburn Users,

Further analysis and testing have revealed that the cooling system in the data center is not sufficiently stable to permit continued operation of the Caliburn and ELF systems. As a result, we are very sad to report that those systems will be permanently removed from service on August 31.

Please plan any remaining workflows accordingly and begin removing valuable data from those systems. Data residing there on or after August 31 will be lost.

Galen

Link to this item

Dear Amarel users,

We have an update on the regular monthly maintenance which was schedule to start from 08:00 ET August 17, to 19:00 ET on Aug. 18.

Due to the software upgrade performed by other Dept. of the University, the network router servicing Amarel cluster will have to be shut down from 05:00 ET to 07:00 ET on Aug. 17, therefore our maintenance reservation on August 17 is revised accordingly. Now it will start from 05:00 ET August 17, to 19:00 ET on Aug. 18. There is a three-hour difference. Fortunately, no running jobs are overlapping with this three-hour period.

We hope this will not cause any inconvenience. Thank you!

In addition, a gentle reminder about the maintenance on Aug. 10, from 9:50am to 12:00pm noon, which we communicated earlier.

Have a nice weekend!

Janet Chang, PhD
Senior Scientist
Office of Advanced Research Computing

Link to this item

Thursday, August 5, 2:07 pm

Dear Amarel users:

Apologies, there is another piece information to update that maybe helpful to your jobs on Amarel.

Not all nodes will be offline during the maintenance on Aug. 10, from 9:50am to 12:00pm noon. The following is the list of nodes that will be impacted: gpu[007-016],hal[0001-0238],mem[001-005],pascal[001-010],volta[001-003]

Thank you

Best regards

Janet

Thursday, August 5, 1:33 pm

Dear Amarel users:

Here is the most recent update regarding the maintenance scheduled on Aug. 10.

As our team obtain more information on the planned work, the reservation has been revised. It will start on Aug. 10, from 9:50am to 12pm noon. We hope this may help your job planning on the cluster. Thank you

Sincerely

Janet

Tuesday, August 3, 7:02 pm

Dear Amarel users:

Please be aware that there are two scheduled maintenances for the month of August.

The regular monthly maintenance on Aug. 17, from 08:00am to Aug. 18, 07:00pm, which was announced earlier.

Another maintenance is schedule for Aug. 10, starting at 00:00 (midnight), ending at 12:00 (noon), this is for the Data Center facility team to perform some electrical work, to solve the power problem that has been affecting the data center users .

Generally, any jobs scheduled that overlaps with the maintenance window will not start until after the maintenance is completed, and jobs that are already running will be terminated when we have to shut down the cluster.

The duration we reserved is 12 hours, but the maintenance likely will be completed in less than 12 hours. We will keep you all updated.

Sincerely

Janet

Janet Chang, PhD
Senior Scientist
Office of Advanced Research Computing

Link to this item

Thursday, August 5, 1:33 pm

Dear Amarel users:

Here is the most recent update regarding the maintenance scheduled on Aug. 10.

As our team obtain more information on the planned work, the reservation has been revised. It will start on Aug. 10, from 9:50am to 12pm noon. We hope this may help your job planning on the cluster. Thank you

Sincerely

Janet

Tuesday, August 3, 7:02 pm

Dear Amarel users:

Please be aware that there are two scheduled maintenances for the month of August.

The regular monthly maintenance on Aug. 17, from 08:00am to Aug. 18, 07:00pm, which was announced earlier.

Another maintenance is schedule for Aug. 10, starting at 00:00 (midnight), ending at 12:00 (noon), this is for the Data Center facility team to perform some electrical work, to solve the power problem that has been affecting the data center users .

Generally, any jobs scheduled that overlaps with the maintenance window will not start until after the maintenance is completed, and jobs that are already running will be terminated when we have to shut down the cluster.

The duration we reserved is 12 hours, but the maintenance likely will be completed in less than 12 hours. We will keep you all updated.

Sincerely

Janet

Janet Chang, PhD
Senior Scientist
Office of Advanced Research Computing

Link to this item

Dear Amarel users:

Please be aware that there are two scheduled maintenances for the month of August.

The regular monthly maintenance on Aug. 17, from 08:00am to Aug. 18, 07:00pm, which was announced earlier.

Another maintenance is schedule for Aug. 10, starting at 00:00 (midnight), ending at 12:00 (noon), this is for the Data Center facility team to perform some electrical work, to solve the power problem that has been affecting the data center users .

Generally, any jobs scheduled that overlaps with the maintenance window will not start until after the maintenance is completed, and jobs that are already running will be terminated when we have to shut down the cluster.

The duration we reserved is 12 hours, but the maintenance likely will be completed in less than 12 hours. We will keep you all updated.

Sincerely

Janet

Janet Chang, PhD
Senior Scientist
Office of Advanced Research Computing

Link to this item

Caliburn Users,

Caliburn and ELF are accessible again. The cooling system in the data center has been returned to service and it’s keeping the environment there stable.

As noted earlier, the JAN-JUN 2021 allocation period for Caliburn ended on June 30. There will not be another allocation period because support agreements for Caliburn’s and ELF’s infrastructure have ended, continuation funding is not available, and the hardware vendors have declined to further extend their support or warranty coverage. Valuable data should be removed from the Caliburn/ELF environment immediately because data stored there are not backed-up.

We want to researchers make the most of any compute resources we have to offer, so Caliburn and ELF will remain available for open-access use as long as we can keep those systems operational without support. With an open-access configuration, any NJ-based researchers may gain access and there will be no dedicated compute or storage allocations. Ongoing use of Caliburn and ELF must be done “at your own risk” because the storage systems are not backed-up, the infrastructure and software are no longer supported, and a hardware failure could end access to those systems at any time. We hope to keep these systems operational for a few months.

Again, I strongly recommend exploring alternative compute and storage resources for your research. For Rutgers researchers, Amarel may be a good alternative. For both Rutgers and non-Rutgers researchers, the scale and diversity of XSEDE compute resources or commercial cloud resources may be good alternatives. The support team at OARC can help with assessment of, and access to, all of those systems.

Please let me know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

07/13/2021, 9:56 am

Caliburn and ELF Users,

Caliburn and ELF are still offline due to the data center cooling issue announced on July 6. The University’s facilities team is working to determine whether the data center where Caliburn is located can safely handle normal operations (the cluster generates a lot of heat, even when only a portion of it is in operation). Right now, we still don’t have any news or an estimated time when researchers may be able to access the system.

Please note that the JAN-JUN 2021 allocation period for Caliburn ended on June 30. The current outage has interrupted the process of users removing their data upon the termination of that allocation period. So, we are certainly planning to restore access to Caliburn’s storage systems as soon as possible to enable users to finish removing their data.

Unfortunately, there will not be another allocation period because support agreements for Caliburn’s and ELF’s infrastructure have ended, continuation funding is not available, and the hardware vendors have declined to further extend their support or warranty coverage. As a result, even if we can restore operation temporarily (for weeks or months), we can no longer guarantee its availability to support research. We obviously want researchers to make the most of any compute resources we have to offer, but ongoing use of Caliburn and ELF must be done “at your own risk” because the storage systems are not backed-up, the infrastructure and software are no longer supported, and a hardware failure could end access to those systems at any time.

At this time, I strongly recommend exploring alternative compute and storage resources for your research. For Rutgers researchers, Amarel may be a good alternative. For both Rutgers and non-Rutgers researchers, the scale and diversity of XSEDE compute resources may be a good alternative. The support team at OARC can help with assessment of, and access to, all of those systems.

Of course, we will send updates when we have any useful news about Caliburn and ELF.

Please let me know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

07/06/2021, 10:56 pm

Caliburn and ELF Users,

Tonight at around 8:40pm the Caliburn and ELF clusters lost their cooling ability and by 10 pm the compute nodes started shutting down due to high temperatures.
All running and pending job have been lost.

We will provide updates as we receive more information.

Ehud (Udi) Zelzion,
Senior Scientist Research Computing
Office of Advanced Research Computing (OARC)

Link to this item

Caliburn and ELF Users,

Tonight at around 8:40pm the Caliburn and ELF clusters lost their cooling ability and by 10 pm the compute nodes started shutting down due to high temperatures.
All running and pending job have been lost.

We will provide updates as we receive more information.

Ehud (Udi) Zelzion,
Senior Scientist Research Computing
Office of Advanced Research Computing (OARC)

Link to this item

This is a reminder that Amarel monthly maintenance is taking place today and tomorrow, April 20-12, 2021.
The maintenance schedule can be found here: https://oarc.rutgers.edu/events/
Please let us know if you have any questions.
Thank you.

OARC team
HPC Help <help@oarc.rutgers.edu>

Link to this item

OARC Systems Users,

Rutgers IT has implemented the ServiceNow platform for management of IT-related service delivery (https://it.rutgers.edu/service-management/servicenow) and OARC has begun using this platform to manage help tickets — some of you may have noticed this already.

Messages sent to help@oarc.rutgers.edu automatically create ServiceNow incidents. These are basically the same as “support tickets” and each has a unique identifier (e.g., Rutgers IT incident INC0021383). Any replies to an email from our ServiceNow system will be included in the referenced support incident, so just reply to those messages in the same way you would reply to an email sent directly from our team.

Also, as with regular emails, if you’d like to request input or support for a new issue,

(1) It would be best if you could create a new message to help@oarc.rutgers.edu instead of simply replying to a previous message. That will help us keep support threads organized.

(2) Please try to direct your messages to our entire team via help@oarc.rutgers.edu instead of emailing specific individuals directly. This helps us ensure that your message won’t be missed if a particular team member isn’t available.

Thanks for your patience as we roll-out this change in the way we deliver support. Let me know if you have questions or if you encounter any problems with this new approach.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Reserve your Amarel research computing resources now

The Office of Advanced Research Computing (OARC) is preparing to purchase its annual hardware, compute node, and storage assets for Amarel, a high-performance research computing cluster serving the university research community. To ensure that there’s enough Amarel inventory to satisfy demand, OARC is asking university researchers, schools, and departments to let it know what they plan to buy as part of Amarel Phase V.

IN PARTICULAR, WE WANT TO HEAR FROM YOU IF YOU HAVE SPECIAL EQUIPMENT NEEDS INCLUDING GPU’S OR HIGH MEMORY NODES.

More computing power, more storage

Phase V’s exact node pricing is still approximate, but anticipated costs per recent vendor negotiations indicate they will be approximately:
* $7,600 per standard node
* $12,500 per 1TB high memory node
* $26,000 per 2x PCI-e A100 GPU or $46,000 per 4x PCI-e A100 GPU

Node ownership includes a four-year warranty, and new owners receive one complimentary terabyte of storage with their first purchase. Storage costs will remain $150 per terabyte.

Why use Amarel?

The Amarel computing cluster — which is named for former Rutgers professor and artificial intelligence pioneer Saul Amarel — allows Rutgers researchers to access world-class computing power without the need to buy and maintain their own hardware.

The architectural approach followed by OARC assures the continuity of each new phase with earlier Amarel phases, but it also allows new phases to take advantage of increased core counts, improved energy efficiency, and reduced cost per unit of compute.

Why buy Amarel?

While general-access Amarel use is free to the entire Rutgers community, owners have the highest priority and are guaranteed access commensurate with their investment. Free access is available on a first-come, first-served basis to the Rutgers community, but jobs undertaken by owners immediately bump non-owner jobs that are underway at any given time. Owner allocations are designed to feel as close to a locally maintained and managed system as possible. Unused cycles are available to the general pool, benefitting the entire user community.

Questions?

Contact Kristin Lepping at klepping@rutgers.edu to indicate your intent to buy Amarel Phase V or for further information.

Kristin Lepping, MLIS
Director, Business Operations, Grants, & Workforce Development
Office of Advanced Research Computing

Link to this item

03/22/2021, 5:37 pm

Dear OARC Users,

The performance issues were resolved and now the system is back to normal.

Thank you for your patience.

Bala

03/22/2021, 3:54 pm

OARC Research Computing Users,

Since this morning, we have received intermittent email regarding system performance issues. The OARC tech team is currently investigating this issue that is affecting file access and performance on Amarel in Piscataway. We will let you know as soon as the issue is resolved.

We apologize for the inconvenience,
Bala

—
Balamurugan (Bala) Desinghu, PhD
Senior Scientist. Office of Advanced Research Computing (OARC)

Link to this item

Dear Amarel Users,

A quick reminder about the upcoming scheduled monthly maintenance of Amarel cluster. For this month, the maintenance begins on March 19, 2021 at 8am (Friday) and ends on March 21, 2021 at 7pm (Sunday).

You can always check the current status of Amarel system and a schedule for anticipated maintenances on the Amarel status page
oarc.rutgers.edu/amarel-system-status

Please contact us if you have any questions or concerns.

Thank you.
—
Vlad Kholodovych, PhD
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

03/11/2021, 12:20 pm

Amarel users,

Our tech team identified the network issue on the infiniband fabric and now it has been corrected.
All nodes are back online and resumed to normal operation.

Please resume your normal activity on Amarel.

We apologize for inconvenience that this disruption might cause to your research.

—
Vlad Kholodovych, PhD

03/11/2021, 11:10 am

Amarel users,

We are experiencing a degradation in network connectivity between Amarel cluster nodes. As a result you may see some lags and slow responsiveness from a cluster.

Access to /home, /projects, and /scratch spaces are limited at the moment and some running jobs, including Open on Demand desktops may fail due to lost of connectivity. Please monitor your currently running jobs.

It is highly advisable to refrain from submitting new jobs till further notice.

Our tech team is investigating and working on mitigation of the problem.

Sorry for inconvenience and thank you for your understanding.

Vlad Kholodovych, PhD
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

Amarel users,

We are experiencing a degradation in network connectivity between Amarel cluster nodes. As a result you may see some lags and slow responsiveness from a cluster.

Access to /home, /projects, and /scratch spaces are limited at the moment and some running jobs, including Open on Demand desktops may fail due to lost of connectivity. Please monitor your currently running jobs.

It is highly advisable to refrain from submitting new jobs till further notice.

Our tech team is investigating and working on mitigation of the problem.

Sorry for inconvenience and thank you for your understanding.

Vlad Kholodovych, PhD
Senior Scientist
Office of Advanced Research Computing (OARC)

Link to this item

Note: The following email was sent to each Phase 1 owner individually and included specifics of their Amarel Phase 1 hardware acquisition. Not all information below applies to each owner.)

Dear [Phase 1 owner],

The Office of Advanced Research Computing is grateful to count you among the earliest adopters of a centralized, university-wide research compute, data, storage, and network system which has supported innovative research across all Rutgers campuses and many diverse disciplines in its short lifespan.

Your Amarel Phase 1 compute nodes and storage are set to expire in July 2021, when their four-year warranty reaches the end of its life. I am reaching out now to gauge your interest in refreshing the X node(s) and (X)TB /project storage that you purchased in Phase 1 with the next generation of Amarel equipment, Phase 5, which will be available in 2021. Pricing for Phase 5 is currently being negotiated with vendors, but is expected to remain comparable to Phase 4, which was $7,200/node and $150/TB storage.

In addition to your Phase 1 purchase of (X) node and (X)TB storage, you also own (X) Phase 2 (or other Phase) node but (no) additional /project storage from later purchases. Your subsequent equipment will remain active until its own expiration date(s), whether you renew your Phase 1 equipment or not.

Should you decide not to renew node ownership your /home directory(s) will remain intact and your accounts will transition to open access.  Any unexpired /project storage will remain available.  Please review https://oarc.rutgers.edu/resources/amarel/ if you have questions regarding the differences between Amarel ownership and Amarel open access privileges.

Since you own a Phase 2 node, you may decide to renew your storage only, if you wish. However, if you choose not to renew, the quota for your /projects space will be reduced to 0TB. Your data will remain readable to enable moving it somewhere else if you become over-quota when your /projects storage is reduced. If you choose not to retain any /projects storage ownership, your data will remain in-place and readable for 90 days to enable moving those data elsewhere. After that, any remaining data must be purged to create space for new /projects storage ownership.

At your convenience, please respond to let me know whether to count you in our Phase 5 purchase, or if we should begin working together to prepare to move your data elsewhere in the coming months.

It is our pleasure to work with you and your team, and we look forward to continuing our partnership. Thank you!

– Kristin

Kristin Lepping, MLIS
Director, Business Operations, Grants, & Workforce Development
Office of Advanced Research Computing

Link to this item

Monday, February 5, 2021 at 10:45 am

Dear Amarel Users,

A quick reminder about the upcoming scheduled monthly maintenance of Amarel cluster. For this month, the maintenance begins on Tuesday (tomorrow) and ends on Wednesday.

Forwarding the previous announcement that has the dates of schedules and further details. Please feel free to contact us if you have any questions or concerns.

Thank you

Bala
—
Balamurugan (Bala) Desinghu, PhD
Senior Scientist. Office of Advanced Research Computing (OARC)

Thursday, January 28, 2021 at 4:53 pm

CORRECTION:

The maintenance window for April will be APR 20-21

Thursday, January 28, 2021 at 12:31 pm

Amarel Users,

In an effort [to] avoid long, intensive maintenance periods, we are shifting to a regular monthly maintenance schedule.

The new schedule is a tentative schedule with maintenance spanning up to 2 days each month, starting on the 3rd Tuesday of the month. We may not need the full 2 days each month, but we’ll reserve that time in case it is needed. Otherwise, the cluster will be returned to service early, immediately after the planned maintenance is complete. When possible, we will skip a month’s maintenance window entirely.

Each maintenance period is expected to begin at about 08:00 ET on the first day and end by about 19:00 ET on the 2nd day.

Here is the anticipated maintenance schedule for the next 12 months:

FEB 16-17
MAR 16-17
APR 13-14
MAY 18-19
JUN 15-16
JUL 20-21
AUG 17-18
SEP 21-22
OCT 19-20
NOV 16-17
DEC 21-22 (may be shifted 1 week earlier)
JAN 18-19 (may be shifted 1 week earlier)

Maintenance is usually scheduled for weekdays because it often involves coordination with University facilities teams or external vendors.

Please note: there may be times when Amarel must be unavailable due to externally-scheduled data center facilities, power, or telecommunications maintenance. When this happens, we will do our best to avoid adding to a month’s downtime with our own maintenance work (e.g., by working in parallel or postponing some tasks).

Let me know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Thursday, January 28, 2021 at 4:53 pm

CORRECTION:

The maintenance window for April will be APR 20-21

Thursday, January 28, 2021 at 12:31 pm

Amarel Users,

In an effort [to] avoid long, intensive maintenance periods, we are shifting to a regular monthly maintenance schedule.

The new schedule is a tentative schedule with maintenance spanning up to 2 days each month, starting on the 3rd Tuesday of the month. We may not need the full 2 days each month, but we’ll reserve that time in case it is needed. Otherwise, the cluster will be returned to service early, immediately after the planned maintenance is complete. When possible, we will skip a month’s maintenance window entirely.

Each maintenance period is expected to begin at about 08:00 ET on the first day and end by about 19:00 ET on the 2nd day.

Here is the anticipated maintenance schedule for the next 12 months:

FEB 16-17
MAR 16-17
APR 13-14
MAY 18-19
JUN 15-16
JUL 20-21
AUG 17-18
SEP 21-22
OCT 19-20
NOV 16-17
DEC 21-22 (may be shifted 1 week earlier)
JAN 18-19 (may be shifted 1 week earlier)

Maintenance is usually scheduled for weekdays because it often involves coordination with University facilities teams or external vendors.

Please note: there may be times when Amarel must be unavailable due to externally-scheduled data center facilities, power, or telecommunications maintenance. When this happens, we will do our best to avoid adding to a month’s downtime with our own maintenance work (e.g., by working in parallel or postponing some tasks).

Let me know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Thursday, January 28, 2021 at 12:32 pm

Amarel Users,

In an effort [to] avoid long, intensive maintenance periods, we are shifting to a regular monthly maintenance schedule.

The new schedule is a tentative schedule with maintenance spanning up to 2 days each month, starting on the 3rd Tuesday of the month. We may not need the full 2 days each month, but we’ll reserve that time in case it is needed. Otherwise, the cluster will be returned to service early, immediately after the planned maintenance is complete. When possible, we will skip a month’s maintenance window entirely.

Each maintenance period is expected to begin at about 08:00 ET on the first day and end by about 19:00 ET on the 2nd day.

Here is the anticipated maintenance schedule for the next 12 months:

FEB 16-17
MAR 16-17
APR 13-14
MAY 18-19
JUN 15-16
JUL 20-21
AUG 17-18
SEP 21-22
OCT 19-20
NOV 16-17
DEC 21-22 (may be shifted 1 week earlier)
JAN 18-19 (may be shifted 1 week earlier)

Maintenance is usually scheduled for weekdays because it often involves coordination with University facilities teams or external vendors.

Please note: there may be times when Amarel must be unavailable due to externally-scheduled data center facilities, power, or telecommunications maintenance. When this happens, we will do our best to avoid adding to a month’s downtime with our own maintenance work (e.g., by working in parallel or postponing some tasks).

Let me know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Friday, January 15, 2021 at 7:00 am

All maintenance work is now complete and the Amarel cluster is back in service.

Please let us know if you encounter any issues.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Tuesday, December 22, 2020 at 10:31 am

Amarel Users,

The entire Amarel system (all campuses) will be offline for routine maintenance starting at 10:00am ET on Monday, January 11 and continuing until Friday, January 15 at midnight. Jobs held in a pending state due to intersection with this maintenance window will resume after the maintenance is complete.

Work being completed during this maintenance window is expected to include firmware updates, software updates, rewiring/recabling of some equipment.

If you have questions or need help, please let us know.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Friday, January 8, 11:31 am

Dear OARC Users,

A quick reminder that Amarel will be offline for maintenance during Jan 11 – Jan 15. I am forwarding the initial announcement (see below my signature) that has more details about the upcoming maintenance.

Please contact us if you have any questions or need any help.

Thank you
Bala

Tuesday, December 22, 2020 at 10:31 am

Amarel Users,

The entire Amarel system (all campuses) will be offline for routine maintenance starting at 10:00am ET on Monday, January 11 and continuing until Friday, January 15 at midnight. Jobs held in a pending state due to intersection with this maintenance window will resume after the maintenance is complete.

Work being completed during this maintenance window is expected to include firmware updates, software updates, rewiring/recabling of some equipment.

If you have questions or need help, please let us know.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Friday, January 8, 2021 at 7:56 am

Morning,

The scheduled power outage in Camden is complete.

To my knowledge, all Amarel related services should be available. Any error or access problem encountered is unexpected and should be reported.

Return to service was a little behind schedule due to problems encountered by PSE&G. The necessary repair work was completed yesterday afternoon around 4pm; however, PSE&G’s main electrical switch into the power grid did not come online. We were notified that power was fully restored shortly before 7pm. Equipment was brought online and available by 8pm.

Remaining issues are related to the legacy Golova cluster. The following should be restored this morning. These do NOT affect most users.

CCIB Workstations (Camden labs+owners)

* /lustre/scratch fileset

CIIPRO Website

* VM hosted on server ccib-bsb-170 which did not reboot

We appreciate your patience and understanding during this outage. We were all in the same boat. Copy of last notification of outage provided below.

Monday, January 4, 2021 at 2:04 pm

REMINDER!

Electrical work is scheduled by PSE&G for this coming Thursday 1/7/21 which will require power be removed from many buildings on the Camden campus. One affected building is the Business & Science Building (BSB). This building houses the Data Center which supplies compute resources for the Amarel HPC cluster as well as home+data directories for many CCIB Lab workstations using Golova legacy resources.

The outage is scheduled from 1/7/21 @06:00 until around 1/7/21 @18:00. To plan for this outage, services and systems will be taken offline starting Wednesday 1/6/21 at 16:00 (4pm). Some services will return to service the evening of 1/7/21; however, some will not become available until Friday morning 1/8/21. An email will be sent out as services return.

The Amarel cluster in Piscataway will remain available. If your data does not reside in Camden, your jobs and access should continue normally during this outage.

Following is a list of some affected services.

1) Amarel resources physically in Camden
* amarelc login server
* halc* nodes
* gpuc* nodes
* /projects filesets in Camden

ccib
f_grigorie_1
f_hfuchs_1
f_jn322
f_km1243_1
f_rucdr
jdb252_1

* job partitions using Camden nodes
p_ccib_1
cmain
p_grigorie_1
p_jn322
p_jdb252_1
p_rucdr

* /scratch
only the Camden-specific scratch normally accessible via amarelc, halc* or gpuc*.

2) Legacy Golova/kestrel data+services
* home+data directories
/g1
/g2
/g3
/g5

* scratch directory
/lustre/scratch

* ciipro website

3) Home directories for CCIB Linux workstations
* will contact each affected owner separately

Following is an excerpt of the original announcement to the Camden Campus sent on 12/23/2020.

To the Campus Community:

On Thursday, Jan. 7, the following buildings will be without power due to electrical repair work and will be closed during 6 a.m. to 6 p.m.:

Artis Building
Business and Science Building
Camden Apartments
Camden Tower
Campus Center
Mail Room Building
Science Building
401 Cooper

For more information, please contact Mike Fitzgerald in University Facilities at mike.fitzgerald@rutgers.edu.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Skipper
HPC & System Administrator
Office of Advanced Research Computing (OARC)

Monday, December 21, 2020 at 11:57 am

Hi Amarel Users,

PSE&G have chosen Thursday 1/7/2021 for the work in Camden that will result in a day long power outage. This will affect ALL compute resources in the Camden Data Center (BSB-114). Therefore, all amarel nodes in Camden will be shutdown.

Anyone using amarelc or halc nodes AND /scratch and /project on that campus will be affected.

Sorry for the inconvenience that may cause an interruption to your work.

—
Vlad Kholodovych, PhD
Senior Scientist
vlad.khol@rutgers.edu

Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey

Link to this item

Monday, January 4, 2021 at 2:04 pm

REMINDER!

Electrical work is scheduled by PSE&G for this coming Thursday 1/7/21 which will require power be removed from many buildings on the Camden campus. One affected building is the Business & Science Building (BSB). This building houses the Data Center which supplies compute resources for the Amarel HPC cluster as well as home+data directories for many CCIB Lab workstations using Golova legacy resources.

The outage is scheduled from 1/7/21 @06:00 until around 1/7/21 @18:00. To plan for this outage, services and systems will be taken offline starting Wednesday 1/6/21 at 16:00 (4pm). Some services will return to service the evening of 1/7/21; however, some will not become available until Friday morning 1/8/21. An email will be sent out as services return.

The Amarel cluster in Piscataway will remain available. If your data does not reside in Camden, your jobs and access should continue normally during this outage.

Following is a list of some affected services.

1) Amarel resources physically in Camden
* amarelc login server
* halc* nodes
* gpuc* nodes
* /projects filesets in Camden

ccib
f_grigorie_1
f_hfuchs_1
f_jn322
f_km1243_1
f_rucdr
jdb252_1

* job partitions using Camden nodes
p_ccib_1
cmain
p_grigorie_1
p_jn322
p_jdb252_1
p_rucdr

* /scratch
only the Camden-specific scratch normally accessible via amarelc, halc* or gpuc*.

2) Legacy Golova/kestrel data+services
* home+data directories
/g1
/g2
/g3
/g5

* scratch directory
/lustre/scratch

* ciipro website

3) Home directories for CCIB Linux workstations
* will contact each affected owner separately

Following is an excerpt of the original announcement to the Camden Campus sent on 12/23/2020.

To the Campus Community:

On Thursday, Jan. 7, the following buildings will be without power due to electrical repair work and will be closed during 6 a.m. to 6 p.m.:

Artis Building
Business and Science Building
Camden Apartments
Camden Tower
Campus Center
Mail Room Building
Science Building
401 Cooper

For more information, please contact Mike Fitzgerald in University Facilities at mike.fitzgerald@rutgers.edu.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Skipper
HPC & System Administrator
Office of Advanced Research Computing (OARC)

Monday, December 21, 2020 at 11:57 am

Hi Amarel Users,

PSE&G have chosen Thursday 1/7/2021 for the work in Camden that will result in a day long power outage. This will affect ALL compute resources in the Camden Data Center (BSB-114). Therefore, all amarel nodes in Camden will be shutdown.

Anyone using amarelc or halc nodes AND /scratch and /project on that campus will be affected.

Sorry for the inconvenience that may cause an interruption to your work.

—
Vlad Kholodovych, PhD
Senior Scientist
vlad.khol@rutgers.edu

Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey

Link to this item

Tuesday, December 22, 2020 at 10:31 am

Amarel Users,

The entire Amarel system (all campuses) will be offline for routine maintenance starting at 10:00am ET on Monday, January 11 and continuing until Friday, January 15 at midnight. Jobs held in a pending state due to intersection with this maintenance window will resume after the maintenance is complete.

Work being completed during this maintenance window is expected to include firmware updates, software updates, rewiring/recabling of some equipment.

If you have questions or need help, please let us know.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Monday, December 21, 2020 at 11:57 am

Hi Amarel Users,

PSE&G have chosen Thursday 1/7/2021 for the work in Camden that will result in a day long power outage. This will affect ALL compute resources in the Camden Data Center (BSB-114). Therefore, all amarel nodes in Camden will be shutdown.

Anyone using amarelc or halc nodes AND /scratch and /project on that campus will be affected.

Sorry for the inconvenience that may cause an interruption to your work.

—
Vlad Kholodovych, PhD
Senior Scientist
vlad.khol@rutgers.edu

Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey

Link to this item

Monday, December 7, 2020 at 8:01 pm

Caliburn Users,

The June 1 – Nov 30 allocation period has ended and last week’s extension of service for this allocation period has also ended. We hope you’ve been able to make significant progress on your projects, make discoveries, or build a foundation for future investigation. It’s important to OARC that your experience with using Caliburn has been a good one. Please share any feedback you may have.

With the end of this allocation period, Caliburn is now officially no longer available for computing and all projects not continuing with renewals into the next allocation period must be removed from the system. Storage directories and user accounts associated with those projects will be purged before the next allocation period begins on January 1.

Caliburn will be offline for maintenance starting at 10:00 am ET on Tuesday, December 8. If you have not already removed your data, logging-in to Caliburn for data egress will be possible after this maintenance is complete (most likely Friday, December 11). We’ll send a follow-up announcement when access to the system is available again.

If you are having trouble removing your data from the system, let us know ASAP.

Please delete data that you have successfully moved so it will be easier for us to identify “stragglers” within Caliburn’s storage system.

Special note: We are working to minimize the full December inter-allocation maintenance task list, which means we may be able to provide limited access to compute resources between December 11 and 31. The timeframe cannot be guaranteed because maintenance may require more time than expected. Also, since Caliburn is officially not in service during this time, user support and infrastructure support will not be available. So, the “lights will be on” but users must work independently if they wish to use the system during this time.

Please let me know if you have questions or need help.

Galen

Monday, November 30, 2020 at 10:05 am

Caliburn Users,

The allocation period scheduled to end today (November 30) has been extended through December 6 to enable additional time for active projects.

At a minimum, we are expecting Caliburn to be offline for 2 days during the week of December 7-11. Caliburn may be offline longer, but we are working to minimize the time when the system will be unavailable.

The maintenance window between the current allocation period and the next one starting January 1 will not require as much maintenance work as anticipated, so we are coordinating with our infrastructure team to keep Caliburn available in production mode for additional days. Once the specific dates when Caliburn must be offline are determined, we will send an update regarding Caliburn’s availability between these allocation periods.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Caliburn Users,

The allocation period scheduled to end today (November 30) has been extended through December 6 to enable additional time for active projects.

At a minimum, we are expecting Caliburn to be offline for 2 days during the week of December 7-11. Caliburn may be offline longer, but we are working to minimize the time when the system will be unavailable.

The maintenance window between the current allocation period and the next one starting January 1 will not require as much maintenance work as anticipated, so we are coordinating with our infrastructure team to keep Caliburn available in production mode for additional days. Once the specific dates when Caliburn must be offline are determined, we will send an update regarding Caliburn’s availability between these allocation periods.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

November 16, 2020

A final reminder: the /scratch quota for all Amarel users will be reduced to 1 TB at noon tomorrow. See previous announcements and details below.

Galen

November 11, 2020

Amarel Users,

Reminder: Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB

That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

With this transition to a 1 TB /scratch quota happening at a time when the provisioning of owner-purchased /projects space has been delayed, there has been a significant increase in demand for temporary special allocations of /scratch storage. We have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Once the delayed /projects space is available (within the next few weeks), we’ll have more flexibility with special allocation requests. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

November 4, 2020

Amarel Users,

This is just a reminder about the transition to 1 TB of /scratch space for all Amarel users that will occur on November 17. See the previous message (below) for details.

Information about how to check your storage utilization is presented in our user guide: https://sites.google.com/view/cluster-user-guide/amarel#h.kc5vkty50xan

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support

From: Galen Collier
Sent: Thursday, August 6, 2020 11:50 PM
To: oarc_amarel_users@email.rutgers.edu <oarc_amarel_users@email.rutgers.edu>
Subject: Transitioning to a 1 TB /scratch quota

Amarel Users,

The expansion of storage space comprising our /scratch file system has slowed temporarily due to various constraints. This has created a situation where we must make significant changes to the space allocated to each user.

Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB. That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

Starting immediately, new Amarel accounts will be provisioned with the 1 TB /scratch quota.

When the transition to a 1 TB /scratch quota is completed in November, there will be increased demand for special allocations of storage. We will have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
https://oarc.rutgers.edu

Link to this item

Graduate and Undergraduate Students:

The CAREERS (Cyberteam to Advance Research and Education in Eastern Regional Schools) Cyberteam Program is a 3-year initiative led by Yale University and funded by the NSF to grow the pool of Research Computing Facilitators (RCFs) available to support researchers at small and midsized institutions in Connecticut, Delaware, New Jersey, New York, Pennsylvania, and Rhode Island.

RCFs are experts at figuring out how to match the right computing resources and techniques to the task at hand, something that can stymie researchers who are, for example, sifting through billions of records to find a specific pattern of genes that correlates with a particular form of cancer; or examining massive quantities of sensor data to understand movements of the sea floor; or modeling complex molecular structures. When computing needs exceed the capacity of the researcher’s desktop or if some research software engineering help is needed, the RCF’s role is to help make use of local, regional, and national high-performance computing resources to meet the researcher’s needs.

We are looking for graduate or undergraduate students who have some computing experience and are interested in learning about research computing facilitation through an experiential training program where participants (virtually) join the project teams of researchers working on a computationally intensive projects for 3-6 month engagements. If matched to a project, you will be assigned an experienced mentor, and together you will provide assistance on research computing and/or data science to a researcher to help to move their project forward. Typically, a time commitment of 10-20 hours per week is required, and students receive a trainee stipend ranging from $3000-$6000, depending on the nature of the project and the level of the student. As part of the program, student RCFs will join a community of facilitators that has up-to-the-minute visibility into research computing projects and programs taking place in the region. In addition to project work, student RCFs will also have the opportunity to work on a live help desk with a mentor, honing their consultative skills while getting exposure to a broad range of research computing technology and topics.

Currently available projects can be viewed at the CAREERS Cyberteam Portal (https://careers-ct.cyberinfrastructure.org). Since we’ve just started, there aren’t many projects listed yet, but we expect the number of projects and their scientific diversity to grow rapidly. We have a few Rutgers and NJIT projects there already. The location of each researcher is noted, but co-location is not required in most cases. We especially encourage qualified women and minorities to participate, as increasing diversity in the RCF community (and in advanced computing and data science, more generally) is an important program goal.

To apply to become a student facilitator (or a mentor, if you have the experience) for any of the projects listed on the Portal, or to be considered for future projects, click on “Join the Team” to register for the CAREERS Cyberteam and create a profile on the Portal. If you prefer, you can also send an email expressing your interest along with a brief bio to our program managers, Kaylea Nelson kaylea.nelson@yale.edu and Julie Ma, jma@mghpcc.org.

Of course, you can also reach-out to me with questions. I’ll be serving as a mentor for a few of the projects.

Galen

Galen Collier, PhD

Link to this item

Amarel Users,

Reminder: Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB

That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

With this transition to a 1 TB /scratch quota happening at a time when the provisioning of owner-purchased /projects space has been delayed, there has been a significant increase in demand for temporary special allocations of /scratch storage. We have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Once the delayed /projects space is available (within the next few weeks), we’ll have more flexibility with special allocation requests. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

November 4, 2020

Amarel Users,

This is just a reminder about the transition to 1 TB of /scratch space for all Amarel users that will occur on November 17. See the previous message (below) for details.

Information about how to check your storage utilization is presented in our user guide: https://sites.google.com/view/cluster-user-guide/amarel#h.kc5vkty50xan

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support

From: Galen Collier
Sent: Thursday, August 6, 2020 11:50 PM
To: oarc_amarel_users@email.rutgers.edu <oarc_amarel_users@email.rutgers.edu>
Subject: Transitioning to a 1 TB /scratch quota

Amarel Users,

The expansion of storage space comprising our /scratch file system has slowed temporarily due to various constraints. This has created a situation where we must make significant changes to the space allocated to each user.

Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB. That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

Starting immediately, new Amarel accounts will be provisioned with the 1 TB /scratch quota.

When the transition to a 1 TB /scratch quota is completed in November, there will be increased demand for special allocations of storage. We will have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
https://oarc.rutgers.edu

Link to this item

November 4, 2020

Amarel Users,

This is another a reminder about the transition to 1 TB of /scratch space for all Amarel users that will occur on November 17. See the previous message (below) for details.

Information about how to check your storage utilization is presented in our user guide: https://sites.google.com/view/cluster-user-guide/amarel#h.kc5vkty50xan

Please let me know if you have questions or need help.

Galen

October 6, 2020

Amarel Users,

This is just a reminder about the transition to 1 TB of /scratch space for all Amarel users that will occur on November 17. See the previous message (below) for details.

Information about how to check your storage utilization is presented in our user guide: https://sites.google.com/view/cluster-user-guide/amarel#h.kc5vkty50xan

Please let me know if you have questions or need help.

Galen
Galen Collier, PhD
Director, Research Support

From: Galen Collier
Sent: Thursday, August 6, 2020 11:50 PM
To: oarc_amarel_users@email.rutgers.edu <oarc_amarel_users@email.rutgers.edu>
Subject: Transitioning to a 1 TB /scratch quota

Amarel Users,

The expansion of storage space comprising our /scratch file system has slowed temporarily due to various constraints. This has created a situation where we must make significant changes to the space allocated to each user.

Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB. That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

Starting immediately, new Amarel accounts will be provisioned with the 1 TB /scratch quota.

When the transition to a 1 TB /scratch quota is completed in November, there will be increased demand for special allocations of storage. We will have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
https://oarc.rutgers.edu

Link to this item

The Rutgers Office of Advanced Research Computing invites applications for the allocation of computing resources on Caliburn. Caliburn, New Jersey’s largest academic supercomputing facility, provides high-performance computing capabilities to academic researchers across the state to accelerate research programs that use or develop highly scalable computing applications.

The call for proposals for Caliburn allocations application can be found at: https://easychair.org/conferences/?conf=caliburn-2021-1. Instructions for the proposals are here: https://oarc.rutgers.edu/wp-content/uploads/sites/32/2020/10/OARC_Caliburn_CFP_Nov.2020_v2.pdf

In addition to applications for awarded allocations, startup allocations are also welcome. Startup applications are provided as means to have full access with a limited time usage allocation; they can be converted into awarded allocations during the next call for proposal cycle. Applications from any academic/research institution in New Jersey are welcome.

Application process timeline:
– Proposals must be received no later than Tuesday, 12/15/2020
– Notification of awards will be on Friday, 12/18/2020
– Awarded allocations will run from 1/1/2021 – 6/30/2021

For current awardees:

We would like to inform current awardees that current research awards will expire on Nov 30th, 2020. We request that if you are interested in renewing your allocation for the next award cycle that you submit an updated proposal. You may use one from previous cycles with a new requested resources. Please include updated publications, presentations and grants awarded due to Caliburn-enabled research.

Feel free to send us an email at help@oarc.rutgers.edu (subject line: Caliburn Proposal) for more information so we can work with you on preparing the documentation.

Paul Guillermo Arias, Ph.D.

Senior Scientist, Office of Advanced Research Computing
Rutgers, the State University of New Jersey
96 Frelinghuysen Road
Piscataway, NJ 08854
Office: CoRE Bldg, Rm 712
Tel: 848-445-8562

Link to this item

Rutgers Research Computing Community,

Women of Color (WoC) – which includes Black, indigenous, and Latinx minorities – are the most underrepresented demographic from high paying industries, including software engineering. Yet WoC are flush with the talent, creativity, and social insight to solve complex problems and lead in the software industry.

The WoCCode program seeks to unite WoC and other members of minoritized groups, to battle feelings of isolation and marginalization that are prevalent in educational institutions and industry settings. Using a dedicated Slack space and cohorts of 4-5 people, the WoCCode program will create a peer mentoring network for WoC and members of other minoritized groups to engage, receive feedback, and troubleshoot issues around their participation in open source software communities.

ANYONE may apply to join WoCCode.

See the flyer for details: https://drive.google.com/file/d/1CvtB8UIvsEDEQft3ukdKpEnWOwRr6UHe

Applications submitted before November 24, 2020 will receive full consideration.

Galen

Link to this item

Rutgers Research Computing Community,

With a variety of storage-related challenges and changes at hand, I’d like to take a moment to provide a status update and some details about our current storage management efforts.

Some background

Hardware added during the past few years has configured Amarel’s computing environment with 6 PB of IBM Spectrum Scale (GPFS) storage that’s divided among 3 key file sets: /home, /projects/, and /scratch. All Amarel users have 100 GB of backed-up storage in their /home directory and 1 TB of space in their /scratch directory to serve as temporary expansion space (not backed-up) for actively running jobs. The standard /home and /scratch allocations are intended to be fixed. Additional storage space is available to Amarel owners (those who invest in our shared infrastructure by purchasing 1 or more compute nodes). Amarel owners purchase TBs of backed-up storage in private /projects directories. See our user guide for details: https://sites.google.com/view/cluster-user-guide/amarel/owner-guide

As initially configured, those 6 PB of storage were sufficient for quite a long time, but we now support a much larger user population and we have more detailed information about their current and anticipated storage needs. Our current storage has been fully consumed, but we are quickly adding additional storage where it is most urgently needed. In response to recent equipment provisioning setbacks due to COVID-19-related acquisition delays and workplace/datacenter restrictions, a top priority for us has become rushing to provide ample online storage to support the explosive growth of data-centric computing taking place on the Amarel cluster.

Current state

This week, we are (finally) getting our new Phase IV storage hardware installed. In this round of expansion, we’re adding 4 PB of storage. Most of that storage will be used to satisfy existing and anticipated /projects storage purchases.

Managing the very large weekly backups for our /home and /projects file sets has become a significant challenge. We are transitioning to storing our weekly backups off-site using commercial cloud storage services. If that turns out to be a safe and cost-effective approach, we may continue to manage our backups that way.

Next few weeks

As the current storage installations (4 PB) are completed and the newly available space is configured for use, owners waiting for their new /projects storage will have earliest access to that space. There are many tasks and variables that influence the schedule for this storage availability. For example,

– We’re waiting for the vendor technicians currently on-site to finish installing the hardware
– We must provision the bare metal (OS, boot configuration, network configuration)
– We must install and configure the GPFS software
– We must upgrade storage nodes to a common version of the GPFS software
– We must integrate the newly configured storage into our existing production file sets
– All filesystems will have to be moved and rewritten
– All of this will be done in a very active production environment

The expected timing for the availability of the new /projects storage is as follows:

Best case = the day after the presidential election (Nov 3)
Median case = the day after Thanksgiving
Worst case = the first work day after the new year, if we encounter problems with all of the tasks listed above, which would be caused by external factors like a vendor’s schedule or performance, etc.

Some of the storage upgrade procedures temporarily use the remaining available space, so we would appreciate users avoiding filling-up their /projects allocations during the next few weeks, if that can be avoided.

Since our user population has grown into the thousands, our /scratch system is undergoing a long overdue restructuring to improve sustainability. Personal /scratch quotas will change to a consistent 1 TB for all users on Nov 17. This transition was scheduled and started before the current /projects storage provisioning delays were realized, but proceeding will help us recover much needed storage space. This transition will also bring our user storage allocations into closer alignment with those of national research computing resources.

Months ahead

Going forward, large storage expansions will be a key part of our annual infrastructure purchase plan: a larger portion of our equipment budget will be dedicated to storage. If you anticipate making a very large /projects storage purchase, let us know as soon as possible so we can integrate your expectations into our storage acquisition planning.

As always, if you have questions or comments, we would very like to hear them.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Dear former HPCC owners and account holders,

While HPCC nodes were retired as of July 1 of this year, storage allocation for prior HPCC owners is being retained through October 31, 2020. At the July 1 deadline, all accounts associated with HPCC ownership became Amarel Open Access accounts. Project storage, as noted, was retained. This information was communicated to you in early June and can be reviewed at https://oarc.rutgers.edu/communications-archive/6-3-2020/. Please note that this effects both owners and accounts associated with the ownership. Former HPCC owners are eligible to renew their storage up to the allocation in place as of July 1 at a cost of $150/TB including backup for a four year period beginning November 1, 2020. This is the same cost we assess current Amarel owners. You are not required to purchase new Amarel Node(s) to renew your storage. Of course, we hope you will consider the purchase of Amarel Phase IV nodes if you have not already placed a pre-order. Phase IV nodes are vastly more powerful than even the last generation of HPCC nodes and are significantly less expensive as a consequence of the subsidy the Office of Advance Research Computing is able to offer. They are expected to be in production in late Fall. You can learn more about Phase IV at https://oarc.rutgers.edu/resources/amarel/#ownership.

Please contact Kristin Lepping (klepping@rutgers.edu) to notify OARC of your intention to renew your Project storage before the October 31 deadline to prevent its removal and loss. Alternatively, should you elect not to renew your storage allocation or to reduce it, be sure to copy any files you intend to keep to your own or university-provided general-purpose storage, e.g., Box. The document at https://sites.google.com/view/cluster-user-guide/amarel#h.6bb8ylmm9bzz provides comprehensive information for moving files to and from Amarel. Should you elect to retain less storage than your current allocation, your applications will be denied write access if your actual utilization is greater than your new allocation. In this case, removing or copying files from your Amarel storage to an alternate location will be necessary. Once reduced, your allocation cannot be increased without the purchase of new Amarel node(s). Please note that your home and scratch storage (the latter subject to the 90 day required access policy) is not impacted by any decision you make regarding Project storage.

To check your Project directory allocation and utilization use the following command at the shell prompt:

mmlsquota -j –block-size=auto projectsp

Please direct any technical questions regarding your storage and its renewal or off-loading to help@oarc.rutgers.edu.

Thank you once more for your pioneering HPCC ownership. OARC trusts that you will continue using Amarel as an Open Access user or hopefully as a new Amarel owner.

Regards,

Les
—
Leslie P. Michelson, Ph.D.
Senior Diretor

Leslie.Michelson@rutgers.edu
Office of Advanced Research Computing
Rutgers University

Link to this item

October 6, 2020

Amarel Users,

This is just a reminder about the transition to 1 TB of /scratch space for all Amarel users that will occur on November 17. See the previous message (below) for details.

Information about how to check your storage utilization is presented in our user guide: https://sites.google.com/view/cluster-user-guide/amarel#h.kc5vkty50xan

Please let me know if you have questions or need help.

Galen
Galen Collier, PhD
Director, Research Support

From: Galen Collier
Sent: Thursday, August 6, 2020 11:50 PM
To: oarc_amarel_users@email.rutgers.edu <oarc_amarel_users@email.rutgers.edu>
Subject: Transitioning to a 1 TB /scratch quota

Amarel Users,

The expansion of storage space comprising our /scratch file system has slowed temporarily due to various constraints. This has created a situation where we must make significant changes to the space allocated to each user.

Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB. That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

Starting immediately, new Amarel accounts will be provisioned with the 1 TB /scratch quota.

When the transition to a 1 TB /scratch quota is completed in November, there will be increased demand for special allocations of storage. We will have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Our special allocation request form is here: https://docs.google.com/forms/d/1GDLR21IV5gbhDVZcaWcZNkN9bW3lHrZaBCNSnyTnHX4

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
https://oarc.rutgers.edu

Link to this item

Kristin Lepping
Tuesday, August 25, 2020 at 5:58 pm

Dear Amarel community,

GPU nodes are increasing in popularity and research computing necessity. While OARC has some stock of GPU’s, most are already owned by Rutgers researchers and very few are available for open access users. We were unable to increase the GPU inventory with our recent Phase 4 node and storage acquisition due to the need to prioritize those items; however, researchers do continue to request them.

With this in mind, we are pursuing quotes from vendors with the goal of making an autumn GPU purchase which would be specifically researcher-owned (i.e., we may not add any or perhaps very few general access GPU’s due to data center constraints and tightened resources). We are trying to gauge the interest of the community. If you or your lab/department would like to be part of this GPU purchase, please let us know and we will work with you to ensure you are included.

The NVIDIA Ampere (A100) is about to come to market and would integrate well with our existing infrastructure. Lenovo has provided us with the following approximate costs:

1x A100 GPU: $17,500 + ~ 2K for 4 year maintenance (total cost)
2x A100: $26,000 + $2,700 for maintenance
3x A100: $34,500 + $3,500 for maintenance
4x A100: $43,000 + $4,200 for maintenance

Each of the above configurations is supported by a server with the same specifications (including memory) as our new Phase 4 compute nodes (available to researchers soon). Configurations with fewer than four A100’s can later be upgraded up to a full complement of A100 GPU’s.

Again, if you wish to purchase GPU’s in this fiscal year, please let me know. Thank you for your continued support!
– Kristin
Kristin Lepping, MLIS
Director, Business Operations, Grants, & Workforce Development

Link to this item

Bala Desinghu
Wednesday, August 19, 2020
Dear All,

As notified earlier, the HPCC cluster and associated nodes are being retired. The HPCC associated nodes are slepner nodes (old nodes) in Amarel cluster.

Today we off-lined HPCC cluster. Tomorrow, HPCC associated nodes, old slepner nodes, will be removed from the cluster.

Please let us know if you have any questions or need more information.
Thank you,
Bala Desinghu

Link to this item

Galen Collier
Thursday, August 6, 2020 at 11:51 pm

Amarel Users,

The expansion of storage space comprising our /scratch file system has slowed temporarily due to various constraints. This has created a situation where we must make significant changes to the space allocated to each user.

Starting November 17, each Amarel user’s /scratch file system quota (limit) will be set to 1 TB. That 1 TB quota will be a soft quota with a 2 TB hard quota (firm limit) and a 2-week grace period. That means a user can have up to 1 TB of data in the /scratch file system, but if more than 1 TB is written there, the user will have 2 weeks to remove files and get back under 1 TB before they will no longer be able to write to /scratch. So, if a job using /scratch generates more output than expected and the quota is exceeded, that job can still run to completion. The 2 TB hard quota means that it will not be possible to put more than 2 TB of data in /scratch.

Starting immediately, new Amarel accounts will be provisioned with the 1 TB /scratch quota.

When the transition to a 1 TB /scratch quota is completed in November, there will be increased demand for special allocations of storage. We will have a fixed space, approximately 200 TB, dedicated to hosting temporary allocations of /scratch space to enable projects that require it to move forward. Our special allocation request form is here: https://oarc.rutgers.edu/amarel-special-allocation-request/

Guidance for moving data from Amarel is available in the Amarel User Guide: https://sites.google.com/view/cluster-user-guide/amarel#h.3jyxmx35y8y2 Please be aware that network performance outside the Amarel environment can be a limiting factor with transfer speeds of approximately 10 MB/sec in some cases. Plan ahead if you must make large transfers.

Please let me know if you have questions or need help.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
http://it.rutgers.edu/oarc

Link to this item

Galen Collier
Wednesday, July 8, 2020 at 5:18 pm

Greetings Everyone,

We’re adding a new training and education option for our user community: the Weekly Open Workshop

Each week, we’ll host a brief Zoom session that provides (1) basics for new Amarel cluster users and (2) shifts into specific technical topics and Q&A. We’ll have occasional guest presenters (researchers, librarians, scholars, or other professionals) to share info, news, techniques, or best practices for managing data and computational research. We’re also very interested in hearing what topics you’d like to have presented.

These sessions are currently scheduled for 10am-12pm every Thursday starting tomorrow, July 14. See https://rutgers.box.com/v/weekly-open-workshop for details. Please stop by to watch, listen, or join the conversation.

You can download and import the following iCalendar (.ics) files to your calendar system:
Weekly: https://us02web.zoom.us/meeting/tZcpduivrDMvGtMiPweQPSVePWZZCoAJuiHf/ics?icsToken=98tyKuGrqjorHdadtRmARpwqBoj4d-3zplxYjbdwlS_yBhFjUTXzBtV4CJFwMfLF

Join the Zoom Meeting: https://us02web.zoom.us/j/83411728507?pwd=RGpMOFM0MmZ0QmlQeFFsVGpzT0VYZz09
Meeting ID: 834 1172 8507
Password: 768002

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
http://oarc.rutgers.edu

Link to this item

Les Michelson
Wednesday, June 3, 2020 at 6:08 pm

Dear Amarel,

I must apologize once more. In the haste to clarify an earlier message, the HPCC nodes end date was incorrectly given as June 1, 2020; the correct date is not July 1, 2020. Again, Phase I-III nodes remain operational. Thank you for your patience.

Regards,
Les

Link to this item

Les Michelson
Wednesday, June 3, 2020 at 5:56 pm

Dear Amarel owners,

Please accept my apologies for the confusion the term HPCC has apparently caused. HPCC is an older cluster that was integrated into Amarel and is scheduled for retirement on June 1, 2020. This does not impact any Amarel Phase I, II or III purchases. These nodes will remain operational as specified by the SLA applicable to each purchase. Happy computing!

Regards,
Les

Link to this item

Les Michelson
Wednesday, June 3, 2020 at 11:46 am

Former HPCC Owners and Associated Users

The job partitions and ownership configurations associated with the legacy HPCC system will be removed after June 30. We have kept those resources in-place as long as we could and much longer than planned. The reasons for doing this are manifold. The compute resources integrated into Amarel from HPCC are far out of warranty and that old equipment is not reliable. In addition, power consumption is more efficient in newer nodes than in the old HPCC node. We also must liberate space, power and cooling to accomodate new equipment.

HPCC owner storage: current storage allocations will expire October 31, 2020. Storage up to the amount currently owned may be purchased if HPCC PIs wish to retain those storage allocations beyond October 31. Those purchases would be standard Amarel storage purchases at $150/TB for a 4-year term ownership (with that 4 yr ownership starting November 1, 2020). Please contact Kristen Lepping (klepping@rutgers.edu) if you are interested in retaining your storage beyond the October 31 expiration. If so, we ask that you notify us as soon as you can as this information will aid us in our longer term storage planning.

What does this mean for your future use of Amarel?

On July 1, 2020 HPCC accounts will be treated as Open Access subject to pre-emption; i.e., jobs submitted by these accounts may be terminated if insufficient resources are available to satisfy owner entitlement for their own jobs. The non-pre-emptible partition is an available option if your jobs are unable to tolerate pre-emption. As noted earlier, your /home and /scratch directories will be retained and project storage can be renewed if desired before their 11/31/20 expiration. In summary, your jobs on Amarel will continue to run as Open Access, but without the guaranty of a resource dedicated to your account. Please email help@oarc.rutgers.edu if you have any questions regarding the upcoming retirement of HPCC owner resources.

We also invite you to consider the purchase of new nodes to replace your former HPCC asset. OARC has announced its Phase IV ownership opportunity with availability by mid to late Summer 2020. These nodes are far more powerful and energy efficient than the nodes comprising HPCC and will be supported for four years from date of initial owner access.

https://it.rutgers.edu/oarc/resources/amarel/#ownership

Thank you for your HPCC ownership and the impact it has had on the vibrant HPC capability and community it helped to foster over the last several years. Again, do not hesitate to contact us should you have questions regarding Amarel Open Access or new storage and dedicated compute acquisition. We hope you will consider Phase IV ownership.

Warm regards,
Leslie P. Michelson, Ph.D.
Senior Director

Link to this item

Les Michelson
Tuesday, May 5, 2020 at 4:53 pm

Dear investigator,

Please see information below to register for IPA/CLC webinar. Also, note that unrestricted access to Rutgers IPA license has been extended to May 31 for general use and June 15th for Covid-research. Apologies for the late notice- this announcement was just received.

QIAGEN IPA and QIAGEN CLC webinar: “Using QIAGEN Digital Insights to determine how LY6E impairs coronavirus fusion and confers immune control” by external virology experts together with Jean-Noel Billaud, Ph.D., Senior Principal Scientist at QIAGEN. Click here to register and attend on May 6th 11:00 AM EDT and 5:00 PM CET.

Please note that unrestricted access to IPA has been extend thru May 31 for all Rutgers IPA accounts. An extension through June 15 is available if research is Covid-19 related. Please contact Leslie.Michelson@rutgers.edu if your use of IPA qualifies under the Covid-19 extension and you wish to take advantage of the extra unrestricted access period.

Regards,

Les

Link to this item

Galen Collier
Tuesday, March 24, 2020 at 3:06 pm

Rutgers Research Computing Community:

I hope everyone is well and able to proceed with important work under the current circumstances.

If there is anything our office might be able to do to support your COVID-19-related research, please let us know. We have received a few requests, so we have addressed those needs and we are taking needed steps to enable other activities.

LET US KNOW WHAT YOU’RE WORKING ON:

If anyone is using an OARC-managed system (Amarel, ELF-1, Didact, Eastern Regional Network resources, or our network infrastructure to access or share data sets) for COVID-19-related research, we would very much like to hear about it:

— Names of PI and coworkers
— Very brief project description/summary
— Applications or specific resources being used

NEW RESEARCH RESOURCES AVAILABLE:

Google Cloud has made a copy of the widely referenced Johns Hopkins University CSSE COVID-19 data available as a public dataset in BigQuery. You can access the dataset on Cloud Marketplace here, https://console.cloud.google.com/marketplace/details/johnshopkins/covid19_jhu_global_cases, along with a description of the data and sample queries to get your analysis started. This John Hopkins University CSSE dataset is made available for educational and academic research purposes only. Google Cloud will not charge for bytes read from this dataset through September 15, but other charges may apply if you join this dataset with other datasets. See the documentation in the Marketplace listing for this dataset for more details.

Galen

Link to this item

Galen Collier
Tuesday, March 17, 2020 at 7:11 pm

UPDATE: The Amarel-N system has been returned to normal service (just moments ago).

Travel and activity restrictions have eliminated some of the planned data center power upgrade tasks, so we were able to power everything up again and return to work.

/projects data that were temporarily staged at other campuses have been synchronized with their original Amarel-N storage system sources. That means the data stored within the Amarel-N system is the most updated version.

Please let me know if you have questions or if you encounter any problems.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)
Rutgers, The State University of New Jersey
Busch Campus, CoRE Building, 96 Frelinghuysen Road, Room 621
Piscataway, NJ 08854
mobile: 970.948.4858
email: galen.collier@rutgers.edu
http://it.rutgers.edu/oarc

Link to this item

Students, Staff, Faculty who need access to Maple, STATA, and SPSS can now download and use these tools on their laptops FREE until the end of May (for Maple and Stata) or June (for SPSS).

The downloads and licensing details can be found at the Rutgers Software Portal: https://software.rutgers.edu

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Galen Collier
Wednesday, March 11, 2020 at 11:15 am

We’re facing a short-notice maintenance affecting our Newark-based Amarel systems (Amarel-N). The power infrastructure there is being upgraded and that will allow us to add new nodes there, but this will require all of of the Newark-based Amarel-N system to be offline.

Amarel-N systems will be unavailable March 13 at 9:00 AM through March 23 at 7:00 PM (10 days).

Users of Amarel-N compute nodes may continue using compute nodes in Piscataway (amarel.rutgers.edu). All user /home directories will remain available during maintenance, but /scratch directories for the amareln.rutgers.edu system will not be available. If assistance is needed with temporary staging of /projects or /scratch data currently stored in Newark, please let us know.

Here are some details of the anticipated work provided to us by the University Facilities Team in Newark:

• We need OARC to power down that Amarel rack as soon as possible on Friday 3/13 in order for us to prepare for the work.
• Afterward, the data center will be completely shutdown.
• Install and commission new UPS, and possibly complete an emergency power transfer test with the building generator and new UPS.
• Electrical service redistribution from the UPS distribution panel to RPPs 1 and 2. Commission and test the new RPPs.
• Execute the emergency power transfer test.
• Installing multiple 30 and 50 amp rated 3 phase circuits. This will allow us to expedite providing additional power capacity to OARC once we’ve completed our electrical and heat load analyses and have determined what is available.
• Some racks in the room may need to be moved to optimize air flow and heat removal.
  Of course, if you have questions or need help, let us know.

Galen

Link to this item

Galen Collier
Saturday, March 7, 2020 at 2:05 am

ELF Users,

Maintenance for the ELF cluster is nearly complete and the system has been returned to service.

Account holders can access ELF at elf.oarc.rutgers.edu or elf.rutgers.edu (ssh [NetID]@elf.oarc.rutgers.edu).

Work completed includes:

— All hosted VMs, compute nodes, and login nodes have been upgraded to CentOS 7.7 with GPFS 4.2.3.19, SLURM 18.08.8.
— The firmware for all managed and unmanaged InfiniBand switches has been upgraded.
— Node Health Check has been setup to help with monitoring of the compute nodes.
— Added DNS names such that elf.oarc.rutgers.edu and elf[1-2].oarc.rutgers.edu and elf.rutgers.edu are the names used.
— Same list of packages is now installed on each node of each type (improved consistency among the compute nodes).
— Switched from nslcd (deprecated) to System Security Services Daemon (SSSD) for authentication services.
— The Lmod module system has been upgraded to 8.1.6.

For information about Caliburn, please see our Caliburn System Status page: https://oarc.rutgers.edu/caliburn-status We are currently working quickly to have Caliburn ready for production use on March 13.

Please let us know if you have questions or if you encounter any problems: help@oarc.rutgers.edu

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Les Michelson
Tuesday, March 3, 2020 at 5:33 pm

Dear Perceval user,

As previously communicated, Perceval accounts will be supported as Amarel Open Access accounts effective 03/01/20. In order to continue your work without interruption, please review the following guidance.

• Direct login to perceval.rutgers.edu is no longer available. Please log into amarel.rutgers.edu. The location of your /home and /scratch data remains unchanged and you can continue your work as you did when logging directly into Perceval. Your applications will continue to work as they did on Perceval.
• You should be aware that Amarel Open Access jobs run in the MAIN partition and are pre-emptible by default. This means that your jobs may be terminated if the resources they are using are required by individuals and organizations owning Amarel assets. You can avoid this situation by requesting that your job run in Amarel’s non-pre-emptible partition. The trade-off is likely to be longer queue times before your jobs start. Advice regarding using pre-emptible and non-pre-emptible partitions is contained in the links below.
• Please review https://oarc.rutgers.edu/perceval/ for a review of previously communicated information and guidance for continuity using Amarel Open Access.
  Although you should not encounter any functional differences between Perceval and Amarel, https://sites.google.com/view/cluster-user-guide an abundance of detailed information for general and more advanced use of Amarel.
• Do not hesitate to contact us at help@oarc.rutgers.edu with any questions you may have now and in the future. And, of course, contact me should you have additional questions.

Regards,

Les
—
Leslie P. Michelson

Leslie.Michelson@rutgers.edu
Office of Advanced Research Computing
Rutgers University

Link to this item

Galen Collier
Wednesday, January 22, 2020 at 12:35 am

OARC Systems Users,

Our current inventory of Amarel compute nodes (Phase III) and backed-up /projects storage is completely sold-out.

The next buy-in opportunity will take place in the fall and announcements will be made when we have configuration and pricing details.

In the meantime, if you have questions about Amarel ownership, please let me know.

Galen

Galen Collier, PhD
Director, Research Support
Office of Advanced Research Computing (OARC)

Link to this item

Les Michelson
Thursday, January 16, 2020 at 3:31 pm

Please read the attached document which details the changes in the support model and roles for the Perceval Supercomputer to be effective March 1, 2020. My apologies in advance for the length of this communication, but I encourage you to review it as some changes may require action on your behalf or that of your PI/mentor. Please do not hesitate to contact me should you any questions concerning the upcoming changes planned for Perceval. Thank you.

Please share this communication with the PI(s) in your lab as necessary

Dear Perceval User:

As Principal Investigator on the NIH grant that funded the Perceval supercomputer for Rutgers University I am delighted and proud to tell you that over the more than 4.5 years this instrument has been operational, a very large body of work in the life sciences has been facilitated, resulting in numerous publications and undoubtedly leveraging many subsequent and successful grant proposals. Perceval also demonstrated that a large-scale, shared computational facility is viable and capable indeed of supporting and being effectively managed for a sizable community of Rutgers investigators. The Office of Advanced Research Computing and its parent, the Office of Information Technology, deserve great credit for their effort and support which, in very large measure, contributed to this success.

On March 1, 2020, Rutgers’ contractual obligation to support Perceval will officially end. At this time Perceval will be re-purposed to support a broader scope of activities including but not limited to research, teaching and pre-production hardware and software testing for the wider research community. Perceval will also support expanded open access, including a larger non-pre-emption pool for the Amarel Distributed Linux Cluster. OARC will provide more detailed information regarding these changes and opportunities as the March 1 date approaches.

We, of course, want this transition to have as little impact on the continuity of your research as possible. OARC manages several major resources that can assist in achieving this result. Among them are Amarel ownership and open access. Amarel ownership provides discrete resources dedicated to your lab and immediately accessible whenever you need it. Owners purchase nodes and storage with all other infrastructure provided and technical and scientific support delivered and at no cost by OARC. Amarel Open Access is another option and has no associated cost. There are several restrictions regarding Amarel Open Access, nevertheless it is an excellent solution for many research computing requirements. OARC plans to have other advanced computational resources in production over the next year and will communicate them to you as they become available.

Investigators who collaborated on the NIH proposal (major and minor users) that funded Perceval are considered owners for the purpose of retaining all or part of the project storage allocated to them, their students and lab staff by purchasing Amarel project storage up to their allocated capacity as of March 1, 2020. Currently, Amarel storage cost is $150/TB for a four-year term. These investigators will retain their allocations until 10/31/20. Those wishing to purchase storage must do so by 10/31/20 or arrange to move their storage out of the Amarel environment. The four-year storage term will begin 11/1/20 for these purchases. Home directory storage and scratch storage are not impacted by the Perceval transition and no action is required by the account holder to retain those allocations. Please contact Kristin Lepping (klepping@oarc.rutgers.edu) for further information if you want to retain all or part of your storage allocation after 10/31/20. A separate notice will be communicated to the Perceval proposal collaborators to alert them to the transition and review available options.

Please note that as of 3/1/20, Perceval accounts will be treated as Open Access subject to pre-emption; i.e., jobs submitted by these accounts may be terminated if insufficient resources are available to satisfy owner entitlement for their own jobs. The non-pre-emptible partition is an available option if your jobs are unable to tolerate pre-emption. OARC plans to expand the non-pre-emptible partition utilizing a portion of the Perceval resources.

• In summary:

• Perceval transitions from an instrument supporting only NIH-funded work to a more general role;
• Investigators who were collaborators on the proposal that funded Perceval are eligible to retain all or part of the storage allocated to them as of 02/28/20. This storage must be purchased by 10/31/20;
• Home directory and scratch storage are not impacted- only project storage;
• Jobs submitted to Perceval by non-Amarel owners are subject to pre-emption. Use the non-pre-emptible partition to avoid re-running jobs or if automatic re-queuing upon pre-emption is not a suitable option for your applications. Although the non-pre-emptible partition will be expanded, users should expect longer queue times versus the pre-emptible main partition. Please contact us if you have questions regarding pre-emption. We will be happy to work with you to find the best approach.

The shelf life of technology, as we all know, is fleeting. Computational technology is perhaps among the more extreme examples of this. On the other hand, we continue to look for new opportunities, e.g., NIH S10, NSF MRI and similar programs to expand and enhance Rutgers cyberinfrastructure arsenal. The proposal that funded Perceval could not have succeeded without collaboration across the Rutgers research communities. We look forward to exploring new programs with you.

Thank you once more for helping to make Perceval a great success through the work you have done on it. Do not hesitate to contact me should you have any questions about this or other changes that have been detailed in this communication.

Regards,

Les

—
Leslie P. Michelson, Ph.D.

Leslie.Michelson@rutgers.edu
Office of Advanced Research Computing
Rutgers University

Link to this item

Galen Collier
Wednesday, January 15, 2020 at 12:43 am

Folks,

The Open OnDemand system has been updated and seems to be working normally again.

Please let us know if you spot any problems.

Galen

Link to this item

Galen Collier
Tuesday, January 14, 2020 at 9:36 am

Folks,

We’ve identified a problem with accessing Amarel and Perceval’s storage systems from our browser-based Open OnDemand interface (https://ondemand.hpc.rutgers.edu). The version of the GPFS software running on the OnDemand server is different from that of our storage systems and that difference is preventing OnDemand from accessing the storage servers.

Note: this problem only impacts access to the cluster using OnDemand – regular command-line access is not affected.

Our infrastructure administration team is working with IBM to get this issue resolved ASAP.

In the meantime, if you must have GUI-based access to Amarel or Perceval to proceed with your work, our FastX system may serve as a viable alternative. The desktop interface is very similar to that provided through OnDemand. Here are some basic instructions for getting started:

(1) Use a browser to access https://amarel.rutgers.edu:3443
(2) Confirm/accept the security exception
(3) Log-in
(4) Choose ‘Launch Session’ and select an XFCE Desktop, then click the Launch button. You’ll be launching a desktop session on login node using FastX.
(5) At this point, since you’re using a login node, you’ll want to get access to a compute node to run your applications.
(6) Right-click and choose ‘Open Terminal Here’
(7) In that terminal window, launch an interactive session on a compute node. For example, ‘srun –time=1:00:00 –pty bash’
(8) From there, you can load modules and run GUI-based applications.

Please let me know if you have questions or need help.

Galen

Link to this item

Galen Collier
Friday, January 10, 2020 at 7:51 pm

Amarel and Perceval maintenance work is complete and both systems have been returned to service.

Please let us know if you spot any problems.

Galen

Link to this item

Paul Arias
Monday, January 6, 2020 at 1:30 pm

Good afternoon,

We wanted to take this time to inform you of the plan for Caliburn’s return to production.

Between August and the end of September, the technical team at OARC took time to research the system and to prepare a plan of action that would return Caliburn to production. The Module Data Center (MDC) preparation should be complete by mid-January to allow Caliburn to return to full production by January 29, 2020. The high level task schedule appears in the attached png file.

A more detailed view of the MDC Caliburn Readiness Plan can be viewed here. On or around January 29, 2020, Caliburn will be available to previous users. Further updates can be viewed at https://it.rutgers.edu/oarc/caliburn-system-status/

Attachment:

Best,

Paul Guillermo Arias, Ph.D.

Senior Scientist, Office of Advanced Research Computing

Link to this item

Galen Collier
Friday, December 20, 2019 at 8:14 am

Hi Folks,

Amarel, Perceval, and Didact will be offline for maintenance January 6-10.

All Amarel resources, including those in Camden and Newark, will be offline.

Items to be addressed during the outage include:

— Lenovo Scalable Infrastructure firmware updates (to match that of the storage systems)
— Lenovo Distributed Storage Solution for IBM Spectrum Scale (DSS-G) and Lenovo GPFS Storage Server (GSS) firmware updates
— Rewriting of all filesystems with the GPFS 5.0.x format to enable variable sub-block sizing
— Add GPFS updates to all compute node images
— Implement Spanning Tree Protocol (STP) across Amarel’s internal network
— Make a range of network interface configuration changes for our non-storage infrastructure
— Patching OS images and various service systems
— Moving enclosures (i.e., racks, power, cooling, connectivity) to enable a future storage expansion

Please note:

(1) Submitted jobs with a run time that overlaps this maintenance window will be held in a “pending” state and will be resumed when the maintenance is complete. So, for the next couple of weeks, it might be best to set run times that enable your jobs to end before Jan 6.
(2) The automated purging of files in /scratch that have not been accessed for 90 days will remain in effect through this maintenance period.

Let me know if you have questions or need help.

Galen

Link to this item

Galen Collier
Wednesday, September 25, 2019 at 11:25 am

Caliburn Users,

Just a clarification about how to access the storage system:

Data transfers via SFTP are now allowed via the following endpoint,

username: @caliburn-sftp.rdi2.rutgers.edu

You should be able to access your home folders, and also the project, staging, and scratch filesystems.

project /gpfs/gpfs/project1/
staging /gpfs/gpfs/staging/
scratch /gpfs/gpfs/scratch/

Galen

Link to this item

Galen Collier
Tuesday, September 24, 2019 at 11:39 am

Caliburn Users,

Access to Storage System: We’ve been able to get infrastructure work schedules aligned sufficiently to temporarily enable access to Caliburn’s storage system. That system is now available for access via the usual methods (e.g., SFTP). Our team will keep these systems up and accessible between 12:00pm ET Monday and 3:00pm ET Friday this week and next week.

Of course, let us know if you encounter problems accessing or moving your data.

Galen

Link to this item

Galen Collier
Sunday, September 22, 2019 at 10:09 pm

Caliburn Users,

General information and status details for Caliburn are available here: https://oarc.rutgers.edu/caliburn-system-status/

Access to Storage System: Our infrastructure team and the OIT facilities / data center team are still working to repair and install components needed to stabilize Caliburn. We are optimistically hoping to be able to offer access to data stored in Caliburn’s storage systems in approximately two weeks (approximately mid-October).

Current Round of Submitted Proposals: We are very disappointed to report that, at this time, Caliburn is not expected to be safe to operate in production-mode for months. Much of the infrastructure acquisition and installation work required is similar to that encountered when initially building a system of this kind: it will take time to purchase, install, and configure the required components.

We will continue to update the web page noted above with minor updates, but major updates (important news) will be shared via e-mail.

Of course, let us know if you have questions.

Galen

Galen Collier, PhD
Director, Research Support

Link to this item

Galen Collier
Tuesday, September 10, 2019 at 4:56 pm

OARC Research Computing Users,

This automated process will begin on Wednesday, Sept. 18 at 2:00pm ET.

Once this process begins, files sitting unused for 90 days in the Amarel and Perceval /scratch file systems will be deleted on an approximately weekly basis. Of course, the purging of files will only happen in the /scratch file system. No purging of files will take place anywhere else.

The /scratch filesystem on any shared computing system (not just ours) is a large temporary storage space intended only for actively running jobs. No data stored in /scratch are backed-up, so a hardware failure of the drives, controllers, or other support systems could cause unrecoverable loss of data at any time. That’s why important data must always be stored where it will be backed-up and many funded project data management plans now explicitly address this concern.

There are a variety of backed-up storage options available to Rutgers students, staff, and faculty:
• Unlimited storage in Box.Rutgers.edu, https://box.rutgers.edu for all students, staff, faculty (15 GB file size limit, but larger files can be split before uploading)
• 5 TB of space in Office365 OneDrive, https://it.rutgers.edu/rutgers-connect/knowledgebase/onedrive-for-business for all staff, faculty, and RBHS students (10 GB file size limit, but larger files can be split before uploading)
• Unlimited storage in Google Drive for ScarletApps users, https://it.rutgers.edu/scarletapps (5 TB file size limit, but larger files can be split before uploading)
• Backed-up /projects storage on Amarel, $150/TB for 4 years for compute node owners

Please let us know if you have questions or need help.

Galen

Link to this item

Galen Collier
Thursday, September 5, 2019 at 10:44 am

Former Sirius Users,

As announced previously, the data saved from the retired Sirius server’s /home directories and temporarily transferred to /projects/sirius on Amarel is scheduled for removal.

The /projects/sirius repository will be permanently deleted tomorrow (Friday) at 12:00pm ET.

Galen

Link to this item

Galen Collier
Thursday, August 22, 2019 at 4:23 pm

OARC Research Computing Users,

As announced previously, files sitting unused for 90 days in the Amarel and Perceval /scratch file systems will be automatically deleted. This process will begin in September.

Some basic information about transferring data from Amarel is available here: https://sites.google.com/view/cluster-user-guide/amarel#h.t618y9lhk7u9

In addition, if you are interested in learning about steps for setting-up rclone for transferring data from Amarel to box.rutgers.edu (free unlimited storage for the Rutgers community), let us know.

Please note that transferring data across the campus network from any source can be a very slow process. Don’t wait to start moving your data.

Galen

Link to this item

Ehud (Udi) Zelzion
Friday, August 9, 2019 at 10:02 am

OARC Research Computing Users,

Due to maintenance work that will take place between 9pm Thursday Aug 22 to 5pm Friday Aug 23 at the Newark Data Center, all Newark-based systems will be powered-down and unavailable.

If you plane on submitting jobs targeting Newark-based resources (e.g., Amarel-N and ‘nm3’ partitions) please make sure that your job’s run-time doesn’t overlap with the maintenance window, which will cause your jobs to stay “Pending” and won’t start.

Please let us know if you have questions or need help.

Ehud (Udi) Zelzion,
Senior Scientist
Office of Advanced Research Computing (OARC) and
Genome Cooperative @SEBS

Link to this item

Galen Collier
Thursday, July 11, 2019 at 12:13 am

OARC Research Computing Users,

Maintenance is complete. All Amarel and Perceval systems have been returned to service and jobs pending due to maintenance have resumed.

The /scratch filesystem and client software used to access it are entirely new.

Let us know if you spot any problems.

Galen

Link to this item

Galen Collier
Tuesday, July 2, 2019 at 12:43 pm

OARC Research Computing Users,

(1) As announced on June 10, files sitting unused for 90 days in the Amarel and Perceval /scratch file systems will be automatically deleted. This process will begin in September.

(2) We are shifting to a new /scratch filesystem on July 10 and all nodes must be updated with new GPFS software. Doing so will require a 6-hour downtime for the entire system (July 10, 11:00am to 5:00pm ET). Data currently stored in /scratch will be moved to the new /scratch filesystem, so there is no need for users to move data. Jobs whose run time intersects with this maintenance window will remain in a “pending” state until the system is returned to service.

Please let us know if you have questions or concerns.

Galen

Link to this item

Bill Abbott
Friday, June 28, 2019 at 11:14 am

All,

I’ve been informed that conley3 will not need to be shut down next week.
I’ve removed the reservation and conley3 is online as normal.

Regards,

Bill

Link to this item

Bill Abbott
Monday, June 24, 2019 at 4:03 pm

Conley3 users,

There will be a power maintenance in the machine room housing conley3
requiring the nodes and master to be powered off. I will be shutting
the machines down Friday, June 28 at 4 PM and bringing them back up on
Friday, July 5 by 5 PM.

Regards,
Bill

Link to this item

Galen Collier
Monday, June 10, 2019 at 1:03 pm

OARC Research Computing Users,

The Institutional Planning & Operations team has completed their work in the Hill Data Center, so our maintenance outage is now complete. The compute nodes in Piscataway have been returned to service.

The work completed includes:
– Modifications to the data center HVAC cooling water loop
– Upgraded power monitoring, diagnostics, and fault detection
– Maintenance for the UPS and power distribution systems
– Replacement of building power switches and feeder cables
– Repairs to the building’s domestic water supply

Important notes:

(1) Files residing in /scratch for 90 days without being accessed (i.e., used or read) will be automatically purged. Everyone’s /scratch directory is considered temporary storage intended only for actively running jobs. Nothing in /scratch is backed-up. We would like to remove limits on /scratch storage to enable researchers to import, use, or generate very large data sets, but doing so will require adherence to this policy of purging of old files. Information about managing or moving data is available in our user guide, here: https://sites.google.com/view/cluster-user-guide/amarel#h.t618y9lhk7u9

(2) Reminder: the quota or limit on storage in /home is 100 GB. If you have more than 100 GB of data in your /home directory, you may be prevented from writing additional data until you remove items stored there.

(3) Lmod (the ‘module’ command) has been updated to version 8.1.6. Saved modules may need to be updated/refreshed.

Please let us know if you have questions or need help.

Galen

Link to this item

Galen Collier
Monday, May 13, 2019 at 11:07 am

OARC Research Computing Users,

Amarel, Perceval, and associated systems will be offline for maintenance around June 1-2. The exact hours/days for this maintenance window will be determined and announced as soon as possible.

The Hill Data Center in Piscataway hosts many core components of OARC’s computing infrastructure. That facility is undergoing significant maintenance and repair work on mechanical and electrical systems on June 1 and 2. Our system administration team will be using this facilities outage as an opportunity to get additional maintenance work completed on OARC’s clusters, storage systems, and networks. That additional work will take place in the days immediately before or after the Hill Data Center outage and we’re working quickly to get the schedule for that work scheduled. Systems in Camden and Newark will also undergo maintenance, but the work schedule at those sites may be shorter.

Please let me know if you have questions or concerns.

Galen

Link to this item

Kristin Lepping
Thursday, April 25, 2019 at 11:09 am

Rutgers Faculty and Research Computing Community,

The Office of Advanced Research Computing (OARC) is producing a plan for the upcoming Amarel Phase III compute and storage equipment purchase. We are asking for your input to gauge researcher interest in becoming or adding to Amarel ownership. If you believe you will benefit from becoming an Amarel owner within the next fiscal year, or wish to increase your current owner allocation, please help us determine the best quantity of equipment to buy on behalf of the greater community by letting us know.

While details and exact pricing are yet to be determined, current expectation of Phase III standard node configuration is:
2X Intel Xeon Cascade 6230 Processors: 2.666MHz DDR4 memory, 22 MB L3 cache, 20-core processors (40 cores/node), 192GB RAM (12 x 16GB DIMMS), 960GB SSD on-board drive, GigE and Infiniband EDR (100Gb/s) adapters

With regard to Phase III compute performance, we’d like to bring your attention to the VNNI (vector neural network instructions) which Intel has added as part of the Cascade Lake generation. VNNI may significantly improve performance in AI/machine learning applications.

FY20 pricing for the above is expected to be approximately $7,100 per node with a four-year warranty (subject to change based on vendor selection).

In addition, as announced in the attached March email, we will soon be adding a number of consumer GPU servers to the Amarel cluster. Information regarding the GPU servers is attached.

About Ownership:
– Amarel owners (e.g., individuals, departments, institutes) always have dedicated, immediate access to purchased resources
– Buying-in to Amarel with our standard 4-year ownership plan is considered an equipment purchase with no indirect costs
– Extremely low pricing is achieved through multi-vendor negotiations and OARC provides administration, power, cooling, rack space, network access

For more information on both Phase III and the consumer GPUs, or to discuss your interest in becoming an Amarel owner, contact us at help@oarc.rutgers.edu. Thank you!
– Kristin

Kristin Lepping, MIS
Director, Business Operations, Grants, and Workforce Development

Link to this item

Galen Collier
Wednesday, March 13, 2019 at 11:40 pm

Research Computing Users,

The Office of Advanced Research Computing (OARC, http://it.rutgers.edu/oarc) is finalizing plans to purchase a number of consumer GPU servers that will be added to the Amarel cluster. We’re sharing this information with the research community in case anyone is interested in taking advantage of this unusual opportunity to buy-in to the cluster (i.e., become an Amarel “owner”).

For more information, contact us at help@oarc.rutgers.edu

The servers we are currently purchasing are configured as follows:

4U rackmount enclosure
8x NVIDIA RTX 2080Ti Graphics Cards (GPUs)
2x Intel Xeon Silver 4116 processors (12x 2.1 GHz cores, 16.5 MB cache)
12x 16 GB DDR4-2666MHz REG/ECC Memory Modules (192GB Total)
2x Samsung PM983 960 GB NVMe SSDs (OS Drives)
4x Segate 2 TB Enterprise HDDs (Storage Drives)
4 year warranty
Expected price: $ 28,500

About Ownership:
– Amarel owners (e.g., individuals, departments, institutes) always have dedicated, immediate access to purchased resources
– Buying-in to Amarel with our standard 4-year ownership plan is considered an equipment purchase with no indirect costs
– Extremely low pricing is achieved through multi-vendor negotiations and OARC provides administration, power, cooling, rack space, network access

Please let us know if you have questions.

Galen

Link to this item

Galen Collier
Wednesday, March 6, 2019

Former Sirius Users,

1. The transfer of your old Sirius /home data to /sirius/[NetID] on Amarel is now complete. Be sure to copy data you wish to save because the entire /sirius directory will be removed in September.

2. Just a reminder that, as former Sirius users, you have exclusive access to the ‘Sirius3 Desktop’ option when connecting to Amarel using Open OnDemand (one of our graphical desktop environment solutions). Here are some basic instructions:

(a) Connect to our Open OnDemand web portal to the Amarel cluster: https://ondemand.hpc.rutgers.edu
(b) log-in with your NetID and standard NetID password,
(c) choose ‘Interactive Apps,’
(d) choose ‘Sirius3 Desktop’ to launch a desktop environment,
(e) and then
(f) click on the ‘Launch noVNC in New Tab’ button when your job is ready to start.

Your Sirius3 Desktop will open in a new browser tab, and you can then run applications and access your storage on Amarel.

Please let us know if you have questions or if you run into problems.

Galen

Link to this item

Galen Collier
Wednesday, February 13, 2019 at 2:15 pm

Perceval users,

Since some recently experienced the undesirable effect of jobs being preempted, we are making some adjustments to our partitions. As a legitimate (NIH-funded) Perceval user, your job will not be preempted on Perceval if that job is submitted to one of the privileged Perceval partitions. Jobs may be preemptable if they are submitted to general priority partitions.

Please let us know if you encounter problems and we will be happy to revisit and adjust our settings as needed.

Privileged partitions, only available for Perceval users:

1. “perceval” includes only Perceval compute nodes and is only available for Perceval users; not preemptable
2. “gpu_p” includes only Perceval cuda (gpu) nodes and is only available for Perceval users; not preemptable
3. “mem_p” includes only Perceval largemem node and is only available for Perceval users; not preemptable

General partitions, available for federated jobs from different clusters (Amarel)

4. “main” includes all nodes on Perceval and includes federated “main” partitions from other clusters, preemptable by jobs submitted to any of Perceval privileged partitions.
5. “gpu” includes Perceval cuda nodes and federated Amarel “gpu” nodes and requires to have a legitimate gpu job request; is preemptable by “gpu_p”; it will preempt jobs submitted to “main“
6. “mem” includes Perceval largemem node and federated Amarel “mem” nodes and requires to have a legit large memory job request; is preemptable by “mem_p”; it will preempt jobs submitted to “main“

Link to this item

Galen Collier
Tuesday, February 12, 2019

OARC Research Computing Users,

The storage quota for all Amarel and Perceval /home directories has been temporarily set to 200 GB. During our next maintenance outage (when no jobs will be running), the original 100 GB quota will be restored.

When our filesystems underwent maintenance recently, the standard 100 GB quota in-place for all user directories in /home was disabled. This change was not evident upon completion of the maintenance work and it enabled a large number of users to push their /home storage beyond the standard quota. Simply enforcing the standard quota would immediately put that very large number of users over quota and would have impacted a large number of actively running jobs. We certainly never want to see productive work interrupted. A compromise to help maintain the health of our storage system (i.e., performance and snapshot capability) would be to temporarily implement a larger-than-normal quota which would have minimal impact on actively running jobs. Some analytical work showed that a 200 GB quota would be about as close to the original quota as we could get, for now. So, that quota has been implemented.

A date for reverting back to the standard 100 GB quota has not yet been set. Keeping Amarel and Perceval running without interruption is a top priority, so that minor quota change will be postponed until it becomes necessary.

Reminder: there are detailed instructions for transferring files to/from Amarel and Perceval in our user documentation: https://sites.google.com/view/cluster-user-guide#h.3jyxmx35y8y2

Please let us know if you have questions or concerns.

Galen

Link to this item

Vlad Kholodovych
Tuesday, January 29, 2019 at 11:58 pm

OARC Research Computing Users,

as you may notice around 6.50pm on Tuesday, Jan 29 the login node of amarel cluster became partially unresponsive due to a high load on the login node.
The server did not shut down completely and was in a process of slow recovery. Our tech team was aware of the situation and monitored the recovery process. We hoped that the server would fully recover in a reasonably short time with a minimal or no impact on the users activity. However, the recovery progressed much slower than was anticipated and to avoid further negative impact on our users we decided to reboot the server at 11pm.

The amarel server is back up now and accepting new jobs.

Unfortunately, all open fastX and ssh sessions got lost as a result of rebooting.
All jobs that were submitted to the queue and running on the server before the reboot should not be affected.

We apologize for inconvenience and for interruption of your research.

—
Vlad Kholodovych, PhD
Senior Scientist
vlad.khol@rutgers.edu

Link to this item

Stan Kolasa (OIT) – not an OARC-generated communication
Sunday, January 13, 2019 at 11:59 am

Fellow IT Staff:

As many of you know, OIT Enterprise Messaging manages several legacy systems providing IT services to the university community. Over the years, many of the services provided by these systems are no longer used or needed and have moved to other systems. Enterprise Messaging is planning to decommission these legacy services and systems in 2019, some of them as early as June 1.

This message is intended to provide advance notice about our plans for decommissioning these systems. The ITLC has already been recently informed.
The systems involved are:
RCI: This system is primarily used by faculty and staff. The three major services provided by this system are email, storage and website hosting.
* RCI email: Email services on RCI were already turned off when we migrated to Rutgers Connect.
* RCI storage: Some people still have files/data stored in the accounts on RCI. If anyone still needs those files/data and doesn’t want anything to be erased, they should move it to OneDrive, Google Drive, Box (once it is available to the Rutgers community), or some other local storage. We hope to have all storage (except those dealing with webpages) moved off RCI by June 1 and therefore will be disabling those accounts on RCI. We will be contacting users with active accounts about this service decommissioning.
* RCI website hosting: We are already in the process of contacting users and departments with websites on RCI to see if they are still needed or not. If still needed, they need to be moved elsewhere, such as Sites@Rutgers (a service provided by Enterprise Application Services) or other web-hosting vendors, or they will be decommissioned. The owners of the web sites will need to decide how they want to handle their current web sites on RCI and handle the migration of them if needed We hope to have all websites off of RCI sometime in 2019, depending on the number of active sites.

Eden: This is a system primarily used by students and student organizations. The email services have already been decommissioned. We plan to handle storage and websites on Eden in a similar fashion to RCI and have the storage service decommissioned by June 1 and websites decommissioned sometime in 2019. Notices will be sent to users with active accounts who use the above mentioned services.

Apps.rutgers.edu: This is a special system that is used by about 300-400 faculty/staff for classroom initiatives or for research purposes. Students also use these systems at times as part of course work or working with a researcher. This system handles long-running jobs that have been submitted by the user. The jobs are typically scientific/mathematical type of applications, such as SAS, Maple and others. Luckily, there is a similar—and better—service being offered by Office of Advanced Research Computing (OARC). We have already communicated this to the faculty using this system, and we let them know we plan to fully decommission all services and remove all accounts on June 1. More info on OARC and the Amarel Cluster can be found at http://it.rutgers.edu/oarc (see note at the top for “Apps & Sirius Users”).

Once the services on the systems mentioned above are migrated elsewhere, we will be in a position to make sure all accounts on these systems are fully disabled and then turned off. We will disable user accounts for several weeks to ensure we have not missed anyone. Then as part of the final decommissioning of a system, we will remove all of the remaining accounts and turn off the system completely.

We will be sending out targeted communications to end users who still have accounts on the above systems/services. Please note you may get questions from your user community when we send out communications, and they may ask you for assistance in moving off their data.

If you have any questions or concerns, please email kolasa@rutgers.edu .

Thank you for your assistance with the above.

Stan Kolasa
Office of Information Technology – EAS, Enterprise Messaging

—————————————————————————————————————————————————
Stan Kolasa
Office of Information Technology – EAS, Enterprise Messaging
Rutgers University
kolasa@rutgers.edu

Link to this item

Galen Collier
Tuesday, October 23, 2018 at 8:32 pm

OARC Research Computing Users,

On Thursday, Oct 25 at noon (ET), we will upgrade the version of Lmod used on Amarel and Perceval from version 6.0 to 7.8. In case that name sounds unfamiliar, Lmod is the tool we use to manage software modules (listing and loading modules). This change will be completed within about 10 minutes and it should go unnoticed by most users.

Here is a link to the documentation for Lmod: https://lmod.readthedocs.io/en/latest

Some notable changes:

— The default version of java (i.e., the module loaded with the ‘module load java’ command) will be changed to 1.8.0_171
— The Lmod “family” directive will be added to the modulefile for some modules to more strictly prevent potentially conflicting modules from being loaded

Of course, if you notice any problems with the loading of modules, please let us know.

Galen

Link to this item

Galen Collier
Friday, October 12, 2018 at 2:20 pm

OARC Research Computing Users,

Maintenance for the Amarel, Perceval, and NM3 systems is now complete. Thanks for your patience!

The maintenance ran a few hours longer than expected because it comprised an unusually complex set of tasks spanning multiple campuses (more of an overhaul to improve performance for a widely distributed, rapidly growing system). Some debugging is still underway.

Details about the maintenance work that was completed and further updates can be found at https://oarc.rutgers.edu/systems-status

** Important notes:
(1) The /scratch filesystem has been redesigned to improve performance across our various campuses. Files that were previously stored in /scratch can be found in /oldscratch. The contents of /oldscratch will be available until December 14, so please move any data you wish to save.

(2) The contents of /home/projects can now be found in /projects (i.e., the link to /projects under /home has been removed).

As always, please let us know if you have questions or if you encounter problems: help@oarc.rutgers.edu

Galen

Link to this item

Eric Marshall
Thursday, October 4, 2018 at 10:18 am

Dear Newton User,

After almost nine years of service, we are shutting down the Newton Cluster on October 29th, 2018. OARC has two options to provide continuity of service for you. If you are currently NIH funded, you eligible to use the NIH-funded Perceval cluster. You may also move your computing to the Amarel cluster as an open access user or become an Amarel owner with resources dedicated and immediately available to you. Perceval and Amarel are state-of-the art machines significantly more capable than Newton. Information for both clusters can be found at https://sites.google.com/view/cluster-user-guide.

Please do not hesitate contacting me or the rest of the OARC team (help@hpc.rutgers.edu) to discuss data migrating and any concerns regarding the end of Newton’s service life and transition to a new environment. OARC trusts that if you have a continuing requirement for high performance computing, you will find these machines more than suitable.

Regards,
Eric and the rest of the OARC Team

Link to this item

Galen Collier
Wednesday, October 3, 2018 at 3:55 pm

OARC Research Computing Users,

This is a reminder of the upcoming maintenance window scheduled to begin Friday, Oct 5 @ 6pm and end Thursday, Oct 11 @ 6pm.

The Amarel, Perceval, and NM3 systems will be offline with no access to their associated filesystems.

This will be a relatively long maintenance window because there are many major tasks to be completed:

— Connecting the new research network across all campuses (Camden, Newark, RBHS, New Brunswick)
— Network configuration adjustments to enable a consistent VLAN across all campuses
— Configuration changes for our GPFS filesystems across all campuses
— GPFS Storage Server updates to support the new distributed configuration
— Cloud federation-related configuration changes
— SLURM updates and other related software updates

Jobs scheduled for a timeframe that overlaps with this maintenance window will be held in ‘pending’ status. We anticipate having those jobs resume queuing and running as submitted when the maintenance is complete.

Please let us know if you have questions or concerns.

Galen

Link to this item

Bill Abbott
Wednesday, August 1, 2018 at 4:29 pm

All,

When requesting to be taken off of the OARC cluster mailing list, please do not “Reply All”; that sends email to all users. Instead, please
email help@hpc.rutgers.edu separately.

All replies requesting this have been received; you do not need to resend.

Regards,

Bill

Link to this item

Eric Marshall
Friday, July 27, 2018 at 11:52 am

Our filesystem is now operational again.
My apologies for the inconvenience.

We have a path to reduce or eliminate the cause of this outage, which should be in place in the next few weeks.

Regards and again my apologies,
Eric

Link to this item

Eric Marshall
Friday, July 27, 2018 at 11:50 am

As of 11:30 am, we are experiencing a problem with our distributed file system. We are working to resolve the issue and will send an announcement once service is restored.

Regards,
Eric Marshall
Director of Advanced Computing Infrastructure

Link to this item

Eric Marshall
Friday, July 27, 2018 at 11:50 am

As of 11:30 am, we are experiencing a problem with our distributed file system. We are working to resolve the issue and will send an announcement once service is restored.

Regards,
Eric Marshall
Director of Advanced Computing Infrastructure

Link to this item

Galen Collier
Monday, July 9, 2018 at 10:55 am

OARC Research Computing Users,

The URL for our user guides, training resources, etc. is now https://sites.google.com/view/cluster-user-guide.

This site is under active development and it will serve as a single, unified resource comprising all of our training materials (user guides, workshop info, miscellaneous reference materials, and much more).

Our existing Amarel, Perceval, and Community Resources user guides will be retired soon.

Galen

Link to this item

Eric Marshall
April 3, 2018 at 11:37 am

Dear Newton User,

After almost nine years of service, we are planning to retire Newton on April 16, 2018. OARC has two options to provide continuity of service for you. If you are currently NIH funded, you eligible to use the NIH-funded Perceval cluster. You may also move your computing to the Amarel cluster as an open access user or become an Amarel owner with resources dedicated and immediately available to you. Perceval and Amarel are state-of-the art machines significantly more capable than Newton. Information for both clusters can be found at oarc.rutgers.edu/resources. Please do not hesitate contacting me to discuss data migrating and any concerns regarding the end of Newton’s service life and transition to a new environment. OARC trusts that if you have a continuing requirement for high performance computing, you will find these machines more than suitable.

Regards,

Eric Marshall and the rest of the Systems Administrator’s Team

Link to this item

Galen Collier
Tuesday, March 27, 2018 at 4:39 pm

Hi Folks,

OARC has setup a new community-contributed software repository and user guide: https://rutgers-oarc.github.io/community/ (still under active development).

Please let me know if you have questions – this is an experiment and we’re anxious to see if it turns out to be a valuable resource.

Galen

Galen Collier, PhD

Link to this item

Eric Marshall
Thursday, December 21, 2017 at 4:50 pm

The Office of Advanced Research Computing will be closed for most normal business during the week of 12/25 – 1/1 in accordance with the University holiday calendar. All remote services will be operational and monitored along with our support line, help@hpc.rutgers.edu .

As a result of reduced workforce during this week, effort will be focused if necessary on problems associated with our technical infrastructure. For this reason, you should expect that hardware and access problems will be given a higher priority. Requests for new application and scientific support should be deferred to the following week whenever possible as a timely response cannot be guaranteed during the holiday week. Thank you for your cooperation and best wishes for a happy and healthy holiday season.

The OARC System Administration and Scientific Support Teams
—
Leslie P. Michelson and the rest of the OARC

Link to this item

Galen Collier
Wednesday, November 15, 2017 at 3:17 pm

Good Afternoon All,

We’ve encountered a problem with the load on our SLURM job scheduling system and we’re currently tuning some settings to resolve it. This issue may already be resolved, but we are monitoring the system to ensure that it’s stable.

Please note: if you are submitting very large numbers of jobs, using a job array is a much more efficient option than scripting the submission of a large collection of standalone jobs. More information about using job arrays can be found here: https://slurm.schedmd.com/job_array.html and there is a simple example in /projects/oarc/examples/array_job.

Of course, let us know if you encounter problems or if you need help.

Galen

Galen Collier, PhD

Link to this item

Bill Abbott
Saturday, October 28, 2017 at 1:37 am

Amarel users,

The maintenance window has concluded and Amarel is back to normal operation. There are test jobs running on most of the nodes still but as they finish the nodes will start running user jobs already in the queue.

The HPCC nodes have been merged into Amarel and are available in the main partition. HPCC members are now Amarel owners and will receive owner partitions and project directories, and should be using amarel.hpc.rutgers.edu as the login node instead of fen.hpc. OARC
personnel will be in touch with PIs to coordinate the transition.

Thanks for your patience as we expand our compute capabilities.

Regards,

Bill Abbott
Office of Advanced Research Computing

Link to this item

Les Michelson
Monday, October 23, 2017 at 10:42 pm

Dear HPCC Owner/User:

In April of this year we communicated to you regarding the merger of HPCC into the Amarel environment. This merger will take place during the Amarel maintenance window scheduled for October 25-27 (Wednesday thru Friday of this week). As such HPCC will adopt the Amarel
scheduling policy.

The Amarel model and attendant scheduling policy will provide HPCC owners much faster turn-around time for most jobs and access to a far larger number of nodes. There are, however, several salient differences between the way jobs are scheduled on HPCC now and how they
will be handled post-merger.

Unit now, HPCC jobs have been scheduled using a fairshare algorithm in which owner job priority is proportional to resource purchased. When HPCC is fully utilized, or busy to an extent in which insufficient resources are available to run new jobs, job submissions are queued.
All jobs will eventually run to completion without interruption.

In the Amarel model, jobs requesting resources that do not exceed those purchased, e.g., nodes/cores and memory, are guaranteed to run immediately and will run to completion. This is an important distinction. Jobs requesting physical compute resources in excess of
ownership will run as “open access” in a partition called “main” and will be subject to preemption if the resources allocated are needed to satisfy an owner job submission. Jobs submitted to the “main partition” can be optionally flagged to be re-queued if preempted. Each owner has an assigned queue providing access to owned resources.

Amarel users with a requirement for non-preemptible access for an urgent or very large job, can request a reservation for non-preemptible access. Reservations requests will be reviewed internally by OARC staff or referred to the OARC Faculty Advisory Committee for review as
appropriate. Such requests will be subject to the availability of sufficient non-owner resources

HPCC members will become owners in the Amarel environment with the same resources they currently own. Ownership will be maintained for no less than two years, predicated on purchased node installation date and useful service life, currently estimated to be four years. At the end of the ownership period, HPCC owners can purchase new nodes at a
significantly subsidized cost or become general, open access users, subject to job preemption with optional re-queue. OARC, of course, hopes that HPCC owners will continue to invest in Amarel as their requirements and funding permit.

Access to storage for HPCC owners remains unchanged under the Amarel umbrella and will remain available through at least the service life of the nodes owned and can be renewed at the then current pricing for storage. Currently, parallel, clustered storage w/backup is priced at
$150/TB.

HPCC owners are asked to contact us via help@hpc.rutgers.edu beginning Monday October 30th, for further information on partition names and additional options for managing storage. Some HPCC owners may not have been migrated to the Amarel environment by October 30, so it is import that the help line be contacted prior to first access.

More details are available at:
https://oarc.rutgers.edu/resources/amarel/

https:/oarc.rutgers.edu/services/condo-model/
or by emailing help@hpc.rutgers.edu.

Welcome to Amarel. We are ready to help you optimize the transition to Amarel.

Regards,

Leslie P. Michelson, Ph.D.
Director for Technical and Scientific Support
Office of Advanced Research Computing

Link to this item

Janae Baker (sent university-wide)
Monday, September 25, 2017 at 8:10 pm

Open access to the Amarel Advanced Computing Cluster is now available
for all Rutgers researchers

The Office of Advanced Research Computing (OARC) is pleased to announce the commencement of open access to the Amarel compute cluster, effective immediately. Rutgers researchers who have purchased nodes through the condo model buy-in program will continue to have priority access, but the cluster is now open to the entire Rutgers research community on a first-come, first-served basis.

OARC anticipates that the greatest growth will occur within the condominium cluster as it will demonstrate the sustainability qualities that are essential for Rutgers to continue to provide its investigators with state-of-the-art resources for computationally demanding problems.

All interested researchers and students are encouraged to browse the Amarel User Guide. This contains both general information as well as specific guidance for developing and running jobs, types of hardware partitions and other useful tips.

For further information on gaining free general cluster access and specific details for current Perceval and HPCC account holders, please visit the General Access page on OARC’s website.

For any additional information, please contact:
Kristin Lepping, Assistant Director at: klepping@rutgers.edu
Janae Baker, Program Coordinator at: janae.baker@rutgers.edu

Link to this item

Bill Abbott
Wednesday, September 13, 2017 at 9:34 am

OARC cluster users,

The legacy HPCC cluster is scheduled to be merged into the new Amarel cluster, with HPCC membership translating into Amarel ownership. Now that Amarel has entered normal production and is available University-wide we’d like to proceed with the merge.

HPCC owners will receive owner queues containing purchased compute resources and group project directories containing purchased disk space.

Details of Amarel ownership can be viewed here.

The main differences between the cluster models are that HPCC relied on “fairshare” to guarantee owner resources, whereas Amarel uses dedicated owner queues with preemption of general access resources in favor of owners.

Fairshare is still a factor but owner queues guarantee immediate, non-preemptible access to owner resources.

For issues and questions regarding this merge or any other OARC clusters contact <help@hpc.rutgers.edu>.

Regards,

Bill Abbott
Office of Advanced Research Computing

Link to this item

Eric Marshall
Tuesday, June 6, 2017 at 1:55 pm

Dear users,
The OARC clusters, Perceval, HPCC, and Hadoop are will remain unavailable for a little while longer while we complete problem remediation. As you may know, we are in the process of transitioning the recently installed Amarel cluster to regular production status. In order to provide an efficient and coherent environment for OARC’s user community, Amarel uses much of the same infrastructure associated with Perceval, HPCC, and Hadoop.

During the integration of Amarel into the OARC environment, we encountered a few problems with the high speed Infiniband network that connects all of the compute and storage elements. This was likely compounded by a problem that developed with the General Parallel File System (GPFS) which supports all /home /scratch and shared storage. We believe we have resolved the GPFS problem and are now verifying the health of the overall file system. At the same time, we are working to resolve any remaining Infiniband issues. We expect the system to be available sometime tomorrow (June 7th). We will advise you if there any changes to this schedule.

Thank you for your patience and apologize for the inconvenience this has caused.

Regards,
OARC technical and scientific team

Link to this item

Kevin Abbey
Sunday, June 4, 2017 at 12:06 pm

Dear HPCC and Perceval users,

There is a technical issue affecting both systems.
We are working to resolve the issue and will update you when it is resolved.

Thank you for your patience,
OARC Help

—
Kevin Abbey
Systems Administrator
Office of Advanced Research Computing

Link to this item

Bill Abbott
Thursday, April 13, 2017 at 11:20 am

HPCC and Perceval cluster users,

Within the next month or so the HPCC cluster will evolve into the larger Amarel condo cluster run by OARC. This requires a brief maintenance window to modify the network and ib fabric and to add a new pool of nodes. This maintenance will also affect the Perceval cluster, which
shares the OARC environment.

We’ve scheduled the maintenance for Monday, April 24. We don’t anticipate it lasting beyond one day.

Jobs that won’t complete by that day will be queued and can run following the maintenance.

For questions regarding Amarel policies (slightly different than HPCC policies) send email to help@hpc.rutgers.edu. Information can be found here:
https://oarc.rutgers.edu/resources/amarel/

Regards,

Bill

Link to this item

Ryan Novosielski
Friday, March 17, 2017 at 4:33 pm

HPCC Users,

A maintenance period has been scheduled for the HPCC and perceval clusters from 10:00a on Wednesday, 3/22 to 6:00p on Friday, 3/24. This maintenance is necessary to correct a problem we have been experiencing with our storage system, as well as to install some scheduler upgrades and add a redundant connection to the Rutgers network for improved performance and reliability as well as improve the layout of our Infiniband fabric. If the maintenance is concluded sooner, the clusters will be returned to service before the end of the maintenance period and an announcement will be sent to the user community. A maintenance reservation has been put into place that will delay any job that will not complete by 10:00a on Wednesday, 3/22, and such jobs will run after the maintenance has completed. Please note that there will be no access to either HPCC or perceval while maintenance is in progress. Please plan accordingly.

Thank you for your patience as we work to improve our systems. Please respond to this message if you have any questions or need additional information about this maintenance period.

=The OARC Sysadmin Team

Link to this item

Les Michelson
Monday, February 20, 2017 at 2:57 pm

Dear Perceval user,

We are truly delighted to tell you that the first full year of production on the NIH-funded Perceval cluster has exceeded expectation in the amount of work that has been successfully completed by investigators across the University. In many ways, Perceval has been one of the transformative elements driving the significant progress underway at Rutgers in developing state-of-the-art computational environments for the research community.

As many of you know, Perceval is a high-demand resource, often running at 100% utilization. This a very good thing considering the enormous cost of facilities of this type. At the same time, we are cognizant of the diversity in job size, job number, frequency of submission, or, simply stated, the numerous ways labs use shared, advanced computing to support their particular research needs. A single job scheduler policy, however complex, cannot optimally satisfy all requirements all of the time or, for that matter, predict future patterns of job submission. Consequently, problems regarding the timeliness of job execution have been reported to us, and we are actively taking that feedback into account.

In an effort to provide a better experience for investigators who use Perceval on a less than regular basis or have significant job requirements with lengthy intervening intervals, OARC has introduced a change to scheduler policy which limits any one SLURM (the job scheduler) group to a maximum of 40% of all available standard node cores. A SLURM group is typically the members of a PI/lab. Jobs submitted in excess of this limit will be queued and prevented from running until the resources used by that SLURM group allow a new job to become eligible for dispatch without exceeding the system resource limit. Jobs, once queued by this mechanism, will age and compete with others according to SLURM fairshare policy. We will observe the impact of this change on the accessibility and utilization of Perceval.

It should be remembered that Perceval is a batch machine and that jobs, once dispatched, are allowed to run to completion (no pre-emption). As a result, a job will not run immediately upon submission if currently idled resources cannot support the requested allocation. We encourage users to submit jobs on a regular basis whenever possible. Investigators who will need a significant resource at a pre-determined future time, should contact us with as much lead time as possible to discuss a guaranteed reservation. We also intend to bring the general question of scheduler policy to the attention of the OARC Faculty Advisory Committee at its next scheduled meeting.

Our aim is to make the NIH’s investment in this instrument as accessible as possible to the community and insure that it remains an important resource in your research arsenal.

The OARC Technical Administration Team

—
Leslie.Michelson@rutgers.edu

Link to this item

Bill Abbott
Wednesday, January 4, 2017 at 10:30 am

BioMaPS cluster users,

The tyr cluster has been decommissioned and the nodes will be merged into the Amarel condo cluster.

There are home directories local to tyr that aren’t part of other clusters. If you need to retrieve user data from the tyr cluster, please email help@hpc.rutgers.edu to coordinate access.

If you don’t need that data, please let us know also so we can remove it.

Thanks,

Bill

Link to this item

Office of Advanced Research Computing (OARC) announces buy-in to Rutgers’ next generation of high-performance computing

OARC is currently looking for researchers, departments, and schools interested in the first buy-in of the Amarel system.

The Office of Advanced Research Computing (OARC) is excited to announce the next phase in Rutgers advanced computing. In early 2017, OARC is unveiling Amarel, a “condominium” style computing environment developed to serve the university’s wide ranging research needs.

Named in honor of Dr. Saul Amarel, a Rutgers pioneer in artificial intelligence and research computing, Amarel is a shared, community-owned advanced computing environment available to any investigator or student with projects requiring research computing resources on any Rutgers campus.

Read the full announcement on the OARC website: https://oarc.rutgers.edu/resources/amarel/

Amarel offers the Rutgers community:

• Owner access for investigators through a University-sponsored matching program
• Owners have highest priority and are guaranteed access commensurate with investment
• Allocations feel as close to a locally maintained and managed system as possible, but with unused cycles available to the general pool
• Free basic-level access for the entire RU community on a first come, first served basis
• Non-owner jobs can be preempted immediately by higher priority owner jobs
• Racks, cabling, network interconnect, system administration, maintenance and spare nodes are all covered by OARC, allowing for highly competitive pricing
• Uniform technical and policy elements which are sufficiently flexible to support the heterogeneity in the Rutgers environment, yet standardize and simplify management procedures, troubleshooting, and provide a seamless user experience
• Uniform technical and policy elements which are sufficiently flexible to support the heterogeneity in the Rutgers environment, yet standardize and simplify management procedures, troubleshooting, and provide a seamless user experience
• Connections to facilities on national research networks, cloud-based resources, and cooperative facilities

More information on node configurations and estimated costs can be found on the OARC website: oarc.rutgers.edu/services/condo-model/

Link to this item

Ryan Novosielski
Sunday, October 23, 2016 at 1:21 am

The Office of Information Technology (OIT) upgrades in the Hill Center Data Center, which required electricity to be shut down on Saturday, October 22nd, have been completed as of about 3:15p. The power shutdown was required to make infrastructure changes that will accommodate greater research computing capacity and to improve reliability.

As of about 1:00a Sunday morning, HPCC is available for use, and jobs that were held before the maintenance period are now running. However, there will be brief periods where HPCC will not be reachable on Monday morning, 10/24, between 9:00a and 10:00a as we make changes to increase network performance and reliability. All running jobs will continue to run, but connections to the HPCC login nodes may be interrupted.

Additionally, during this maintenance window, we migrated the majority of the legacy HPCC compute nodes to our new HPCC environment (accessible now via login node fen2.hpc.rutgers.edu), which is running more modern system and application software, uses the Lmod environment modules software to make it easy to configure your user environment for OARC-installed software packages, and contains 20 new compute nodes with 24-core Intel Haswell-generation processors, among other benefits. You will find additional information in our user guide:

https://sites.google.com/view/cluster-user-guide

A small number of nodes will remain in the existing/legacy HPCC environment to allow current user jobs to finish and to allow users who are not able to migrate immediately to continue working. OARC staff will be available to assist with the new environment if needed; please contact  help@hpc.rutgers.edu  if you have having any difficulty.

*NOTE: you will need to copy any data you will require from your legacy HPCC /scratch directory to the /scratch directory on the new environment. Please only copy the data that you will need, and keep in mind that this storage location is only for temporary storage during the life of a project and is not backed up. /home directories are shared between the legacy and new environments so you will not need to copy these files. Contact us at help@hpc.rutgers.edu if you need any assistance.

We apologize for any inconvenience these changes may cause, but these improvements are necessary in order to support the expansion of our research resources and to improve our computing resources. If you have any questions or need any assistance, please respond to this e-mail.

=The OARC Sysadmin Team

Link to this item

Ryan Novosielski
Friday, October 21, 2016 at 10:19 am

The Office of Information Technology (OIT) will be performing upgrades in the Hill Center Data Center, which will require electricity to be shut down on Saturday, October 22nd, from 6:30a to 11:00p. This is required to accommodate greater research computing capacity and to improve reliability. As a result of these upgrades, per the previous announcement sent out by OIT, OARC equipment housed in Hill Center will not be available this weekend. We will begin a shutdown for the weekend at 5:00p, -TODAY- Friday, October 21st.

The following systems will not be available:

* HPCC (both legacy fen and new environment fen2)
* NIH-funded perceval cluster (all compute nodes and infrastructure)

…and related storage systems. A maintenance reservation has been put into place on all systems that will delay any job that will not complete by 5:00p -TODAY- until after the maintenance has completed.

Additionally, during this maintenance window, we will be migrating the majority of the legacy HPCC compute nodes to our new HPCC environment (accessible now via login node fen2.hpc.rutgers.edu), which is running more modern system and application software, uses the Lmod environment modules software to make it easy to configure your user environment for OARC-installed software packages, and contains 20 new compute nodes with 24-core Intel Haswell-generation processors, among other benefits. You will find additional information in our user guide:

https://sites.google.com/view/cluster-user-guide/home

As mentioned above, this new environment is already available, so we encourage you to login with your existing credentials at fen2.hpc.rutgers.edu and take a look. A small number of nodes will remain in the existing/legacy HPCC environment to allow current user jobs to finish and to allow users who are not able to migrate immediately to continue working. OARC staff will be available to assist with the new environment if needed; please contact help@hpc.rutgers.edu if you have having any difficulty.

*NOTE: you will need to copy any data you will require from your legacy HPCC /scratch directory to the /scratch directory on the new environment. Please only copy the data that you will need, and keep in mind that this storage location is only for temporary storage during the life of a project and is not backed up. /home directories are shared between the legacy and new environments so you will not need to copy these files. Contact us at help@hpc.rutgers.edu if you need any assistance.

We will begin to restore all equipment to service as soon as possible after the power upgrades and system changes, and will send out updates as information becomes available. We expect have full service restored by 12:00 noon on Monday, October 24th. We apologize for any inconvenience this may cause, but these improvements are necessary in order to support the expansion of our research resources and to improve our computing resources. If you have any questions or need any assistance, please respond to this e-mail.

=The OARC Sysadmin Team

Link to this item