- Lustre best practice
- Transferring Data
- Compressing/uncompressing files and directories
- Scientific Data Libraries
The global filesystem on Vilje is Lustre. This is the file system for all home areas, /home=113T, and a larger work area, /work=395T. Files not used in the last 21 days are periodicly removed from /work.
Disk Quota and Backup
The disk usage are limited with disk quotas. Default quota:
Check your file system quota using the “dusage” command, specifying the username and filesystem. The listed values are reported in kbytes.
For the home directory:
The home areas are backed up daily incrementally, that is all files modified since last full backup are copied. A full backup is carried out monthly. There is no backup of files at the /work area.
Lustre best practice
- Avoid doing metadata operations (i.e. traverse directries with thousands of files)
- Always set a stripecount > 1 on files larger than 1GB
- Always set small stripecount for small files (stripecount=1)
- keep small and large files i separate directories with appropriate stripecount. 1 for smaller files and > 1 for larger files
- Applications which does parallell writes to files, will benefit from striping.
|How to set stripecount|
Get striping information
See this section
Binary Data (Endianess)
See this section
Compressing/uncompressing files and directories
The available UNIX-commands for compression and uncompression are:
gzip/gunzip– for description see ‘man gzip‘
bzip2/bunzip2– for a description see ‘man bzip2′
To compress a file, follow the example:
To uncompress a gzipped file, follow the example:
To compress a directory, the compression tool needs to be used in combination with a archiving tool, like
To unpack a archived directory:
Scientific Data Libraries
See the NetCDF page.
See the HDF5 page.