Using SCC OnDemand for Interactive Data Transfer The scc-globus node is dedicated to file transfers and interactive usage through SSH is subject to the same policies as apply on the login nodes. For interactive use, researchers can SSH directly to scc-globus.bu.edu from within the cluster or use SCC OnDemand. Note that the data transfer node does not accept interactive jobs from the scheduler neither qsh nor qrsh requests will work with the -l download command line option. #$ -l download module load awscli aws s3 sync s3://mybucket scc-directory A researcher should change the module and commands for their own purposes. This example script would use the data transfer node to synchronize files from Amazon Web Services S3 Service. This can be done for standard qsub batch job submissions either on the command line or inside of a batch script.Įxample qsub command line: scc1$ qsub –P projectname -l download download.qsub The -l download command line option for qsub will place your job on the data transfer node. The data transfer node can be requested with the “ -l download” command line option with a maximum runtime of 24 hours and limited to a single processor core. In comparison, the SCC compute nodes offer large computing capacity, but reside on a private network and share a connection to the outside world which limits the data transfer speeds. The data transfer node is intended only for data transfer tasks and not for compute-intensive workloads. The data transfer node is a single server named “ scc-globus.bu.edu” that provides a direct 10GbE connection from the SCC to the internet for data transfer. Suggested Software and Modules for Transferring Data from Cloud Storage. Using SCC OnDemand for Interactive Data Transfer.This page contains the following sections: The Data Transfer Node has its own high bandwidth internet connection and its only intended to be used for data transfer tasks. Accessing these sources require an internet connection and often an application to interact with the remote servers hosting the data.Ī dedicated node, called the Data Transfer Node, is available on the SCC for these types of data transfer tasks. Alternatively, data may also be made available via Application Programming Interface (API) by the organization hosting the data. Data is stored in cloud storage services such as AWS S3 buckets, Google Drive, or One Drive.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |