Changes between Version 2 and Version 3 of ComputeResources/UMCGCluster


Ignore:
Timestamp:
Aug 19, 2011 5:27:16 PM (13 years ago)
Author:
laurent
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ComputeResources/UMCGCluster

    v2 v3  
    1111
    1212The 10 nodes are dedicated for the GCC group at UMCG. GoNL being the most compute intensive project at GCC, most of the cluster can be used for it. The storage is shared by different groups in Groningen but there is currently no "hard limit" on how much space GoNL can use on the storage; this of course will only work as long as there is sufficient space for everyone.
     13
    1314== Access ==
    1415Access to the UMCG cluster is done via SFTP (data access only, see [wiki:DataManagement/SftpServer SFTP page] about this) or SSH. There is no public access to the UMCG cluster. Additional personal ssh or sftp accounts can be requested via Morris who keeps the list of all users that have full data access.
     
    1819
    1920 * '''Storage''': The block size on the storage is 6MB, which means that each file -regardless of its real size- will occupy at least 6MB on the file system. This means that data should rather be kept in big files rather than a multitude of small files whenever possible. Typically things like logs, old submit scripts, etc. should be compressed into 1 file for archiving.
    20  * '''I/O''': While 10Gb network connection per node is fast, typical GoNL jobs use large files and consumes lots of I/O. Therefore, I/O should be kept minimal and if a job can be parallelized on multiple cores (i.e. load data once in memory, process it on multiple cores, push it back), it is typically preferred as having separate processes all loading the same data in memory. 
     21 * '''I/O''': While 10Gb network connection per node is fast, typical GoNL jobs use large files and consumes lots of I/O. Therefore, I/O should be kept minimal and if a job can be parallelized on multiple cores (i.e. load data once in memory, process it on multiple cores, push it back), it is typically preferred as having separate processes all loading the same data in memory.
    2122 * '''Local Storage''': In order to reduce I/O, temporary files (and eventually other heavily used resources) should be stored directly on the local node; the local storage on each node is mounted in /local. Note that:
    2223   * Any data on the local storage that you want to keep after the job terminates should be copied to the general storage as the local storage is periodically cleaned and any data that is not in use by currently running job will be deleted.
     
    4445'''`Example script header:`'''
    4546
    46 `-----------------------`
     47{{{
     48#!/bin/bash
     49#PBS -N JobName
     50#PBS -q gcc
     51#PBS -l nodes=1:ppn=1
     52#PBS -l mem=4gb
     53#PBS -l walltime=12:00:00
     54#PBS -o /target/gpfs2/gcc/home/lfrancioli/output.log
     55#PBS -e /target/gpfs2/gcc/home/lfrancioli/error.log
    4756
    48 `#!/bin/bash #PBS -N JobName#PBS -q gcc#PBS -l nodes=1:ppn=1#PBS -l mem=4gb#PBS -l walltime=12:00:00#PBS -o /target/gpfs2/gcc/home/lfrancioli/output.log#PBS -e /target/gpfs2/gcc/home/lfrancioli/error.log`
     57#Here comes your bash script commands
    4958
    50 `#Here comes your bash script commands`
    51 
    52 `echo "Hello World!"`
    53 
    54 `----------------------------`
    55 
     59echo "Hello World!"
     60}}}
    5661'''Commonly used flags:'''
    5762