Skip to content

Data Transfers

Created by Angenent, Holger, last modified by Kuhlmann, Justus Theodor on 24. May 2024

Data Transfers via the command line

There are the usual ways, how you can transfer your data to the cluster:

  • scp: The regular way of transferring data in Linux
  • rsync: Also a standard tool in Linux like scp, but has the advantage that it can resume broken transfers as it normally keeps the metadata of the files
  • rclone: A mighty tool for data transfers, normally to cloud storage. May also be utilized for data transfers to Palma and might be faster than the former solutions when transferring many small files
  • WinSCP: The standard tool when using Windows

Data Transfers via the web interface

This part is still in a testing phase! Please be careful when using it!

If you have access to Palma, you can use a Nextcloud for data transfers. To make use of it, do the following:

  • Create a directory called "transfer" at /scratch/tmp/$USER: mkdir /scratch/tmp/$USER/transfer
  • Log in at https://palma-web.uni-muenster.de
  • Click on the "scratch" directory. (You might have to click multiple times in the first run). Data you put in here will be stored in the scratch directory. Also, data that you put in /scratch/tmp/$USER/transfer will be visible here
  • The recommended way to use this is to upload your data via the web interface and move it to another location on the cluster afterwards. The speed of the nextcloud will probably decrease, if the directory becomes too crowded.
  • In future iterations, this is intended to be used for realizing data transfers between different HPC clusters.

Transferring data to the Nextcloud via rclone

If you want to copy data to and from Palma from the outside and have rclone available, it can be used for a performant data transfer:

Setting up rclone:

  1. rclone config
    • n) New remote
    • name> palma-web
    • Storage> 47 (this is WebDAV)
    • url> https://palma-web.uni-muenster.de/remote.php/dav/files/$username (replace $username with your user id)
    • vendor> 2
    • user> $username (replace $username with your user id)
    • y/g/n> y (y) Yes, type in my own password)
    • password: Create an app password in the web interface at Settings → Security (Einstellungen → Sicherheit) and put it here.
    • bearer_token> (Leave empty)
    • n) No (default)
    • y) Yes this is OK (default)
    • q) Quit config

Data transfer with rclone:

  • From remote to palma: rclone folder palma-web:scratch/ --progress --verbose
  • From palma to remote: rclone palma-web:scratch/_folder_ . --progress --verbose

See also: https://linuxpip.org/rclone-examples/#rclone-two-way-sync

Mounting webdav (e.g. sciebo or Nextcloud) as a filesystem with rclone

  1. Load necessary modules: module load palma/2023a rclone
  2. Setup the remote host with rclone config interactive command line tool, as shown above:
    • type in the name (for example sciebo)
    • select the storage type: WebDAV (option 47)
    • type in the URL: https://uni-muenster.sciebo.de/remote.php/webdav/
    • select the vendor: Owncloud (option 3)
    • type in the username: your sciebo username (@uni-muenster.de)
    • type in the password: your sciebo password (it will be stored in a config file encrypted)
  3. Create a directory where you want to mount sciebo, for example mkdir ~/sciebo
  4. Mount sciebo using rclone mount command: rclone mount sciebo: ~/sciebo
    • The mount stays active while the command runs, when you interrupt it with ctrl+c, the mount will be unmounted
  5. All content on sciebo will appear in the ~/sciebo directory as local files and you can copy them, edit them, create new files, etc.
  6. Unmount by interrupting the call from step 4
  7. In case unmounting fails for any reason (like for example you still using the directory when rclone is stopped, remounting will not work (not even with --allow-non-empty) and you will have to unmount manually using: fusermount -u ~/sciebo