Transferring Data

The transfer of files to and from Idun can be accomplished using network drives, scp, or sftp. The following sections exemplify the usage of these methods.

Network Drives – Mounting Idun’s home and work directory to your local machine

Idun has two samba servers ({idun-samba1,idun-samba2}.hpc.ntnu.no) that enable the mounting of a user’s home and work directory directly to its local machine. The respective paths to the shares are:

  • Home directory: //<samba-server>/<username>
  • Work directory: //<samba-server>/work

The following sections illustrate the mounting of a user’s home directory for Windows, Linux, and macOS with idun-samba1.hpc.ntnu.no. For mounting the work directory, simply replace the home directory path with the work directory path in the examples.

Windows

In Windows, the File Explorer can be used to attach the home directory as a network drive. Navigate to File Explorer -> Map network drive and enter the details as shown below:

Replace <username> with your Idun user. This mounts the user’s home directory as drive Z into the File Explorer.

Linux

In Linux, the mount.cifs command can be used to attach a user’s home directory to a local directory:

$ sudo mount.cifs //idun-samba1.hpc.ntnu.no/<username> /tmp/mntpt -o username=<username>,domain=WIN-NTNU-NO

In the above example, the home directory of <username> is attached to the local directory /tmp/mntpt. The original content of /tmp/mntpt is overshadowed with the files of the user’s home directory from Idun. Files can now directly transferred to Idun by simply copying them to /tmp/mntpt.

The umount command can be used to detach the home directory:

$ sudo umount /tmp/mntpt

This reveals the original content of /tmp/mntpt again. A script to simplify this process can be found here.

macOS

In macOS, Finder can be used to attach the home directory as a network drive. Navigate to Finder -> Go -> Connect to Server and enter the details as shown below:

Replace <username> with your Idun user.

Smbclient – Accessing NTNU’s network drives directly from Idun

Copying data to Idun by mounting the home and work directory from Idun directly to a local machine is acceptable if the data is located on the local machine, but it creates a lot of overhead if the data resides on another machine. For example, a user would need to perform the following steps to transfer data from NTNU’s home directory to Idun’s home directory:

  1. Mount NTNU’s home directory to the local machine.
  2. Transfer the data from NTNU’s home directory to the local machine.
  3. Mount Idun’s home directory to the local machine.
  4. Transfer the data from the local machine to Idun’s home directory.

This transfers the data twice, once from NTNU’s home directory to the local machine and once from the local machine to Idun. A better approach would be to directly transfer the data from NTNU’s home directory to Idun’s home directory, cutting the network traffic and data transfer time in half. The smbclient tool is an interactive program that enables the transferal of data directly between Idun and any of NTNU’s samba shares, such as //home.ansatt.ntnu.no/<username>, //home.stud.ntnu.no/<username>, or //forskning.it.ntnu.no/ntnu/<share>.

It can be invoked as follows:

smbclient
[<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> ls
  .                                   D        0  Mon Jan 14 10:26:49 2019
  ..                                  D        0  Wed Oct  9 17:52:59 2019
  public_html                         N       26  Fri Mar 22 17:35:01 2013
  .bash_profile                       A       31  Tue Jan  8 11:37:52 2013
  .bashrc                             A       25  Tue Jan  8 11:37:53 2013
  .emacs                              A       37  Tue Jan  8 11:37:53 2013
  .cache                              D        0  Tue Aug 27 12:54:05 2013
  .bash_history                       A     5265  Mon Jan 14 10:26:49 2019
  .ssh                                D        0  Wed Nov  8 10:29:47 2017
  .viminfo                            A     6639  Wed Nov  8 10:21:51 2017

smb: \> quit
[<username>@idun-login1 ~]$

The above example connects to an employee’s NTNU home directory, lists its content, and exits again.

The help command lists all supported commands and can provide information about the usage of each command:

smbclient-help
[<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> help
?              allinfo        altname        archive        backup
blocksize      cancel         case_sensitive cd             chmod
chown          close          del            dir            du
echo           exit           get            getfacl        geteas
hardlink       help           history        iosize         lcd
link           lock           lowercase      ls             l
mask           md             mget           mkdir          more
mput           newer          notify         open           posix
posix_encrypt  posix_open     posix_mkdir    posix_rmdir    posix_unlink
posix_whoami   print          prompt         put            pwd
q              queue          quit           readlink       rd
recurse        reget          rename         reput          rm
rmdir          showacls       setea          setmode        scopy
stat           symlink        tar            tarmode        timeout
translate      unlock         volume         vuid           wdel
logon          listconnect    showconnect    tcon           tdis
tid            logoff         ..             !

smb: \> help cd
HELP cd:
        [directory] change/report the remote directory

smb: \>

Useful commands for transferring data are:

  • ls: list the contents of the current remote directory
  • pwd: show current remote directory
  • cd <directory-name>: change the current remote directory to <directory-name>
  • lcd <directory-name>: change the current local directory to <directory-name>
  • get <remote-file> <local-file>: Copy remote file <remote-file> to <local-file>
  • mget <mask>: Copy all files that match <mask> from the share
  • put <local-file> <remote-file>: Copy local file <local-file> to <remote-file>
  • mput <mask>: Copy all files that match <mask> to the share
  • prompt [on|off]: Turn prompting for filenames for mget and mput on or off
  • recurse [on|off]: Turn directory recursion for mget and mput on or off

The following example shows how to copy a remote folder to Idun:

smbclient-example
[<username>@idun-login1 ~]$ smbclient //home.ansatt.ntnu.no/<username> -U <username> -W WIN.NTNU.NO
Enter WIN.NTNU.NO\<username>'s password:
Domain=[WIN-NTNU-NO] OS=[SpinStream2] Server=[Windows 2000 Lan Manager]
smb: \> ls
  .                                   D        0  Fri Oct 11 12:04:05 2019
  ..                                  D        0  Wed Oct  9 17:52:59 2019
  copy-test                           D        0  Fri Oct 11 12:04:18 2019

smb: \> prompt off
smb: \> recurse on
smb: \> mget copy-test

The above example shows the copying of the copy-test folder to the user’s home folder on Idun. The user needs to turn off prompting in order to avoid being questioned for each file and turn on recurse in order to recursively copy the entire folder.

Secure Copy

Secure Copy (scp) is a program to securely transfer files between computers. It uses the Secure Shell (ssh) for data transfer, providing the same authentication and security. For example, copying a file named myfile from a local system to Idun:

$ scp myfile <username>@idun.hpc.ntnu.no:.
Password:

Secure File Transfer Protocol

Secure File Transfer Protocol (sftp) is a file transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. Example, put file from local system to Idun:

$ sftp idun.hpc.ntnu.no
Password:
Connected to idun.hpc.ntnu.no
sftp> put myfile

For Windows users WinSCP is a free graphical SCP and SFTP client.