Title : Port of the week: rclone
Author: Solène
Date  : 28 October 2020
Tags  : portoftheweek

New **Port of the Week** after 3 years! I never thought it was so long
since last blog post about slrn.

This post is about the awesome **rclone** program, written in Go and
available on most popular platforms (including OpenBSD!). I will
explain how to configure it from the interactive command, from file
and what you can do with rclone.

**rclone** can be see as a rsync on steroids which supports lot of
Cloud backend and also support creating an encrypted data repository
over any backend (local file, ftp, sftp, webdav, Dropbox, AWS S3,
etc...).

It's **not** a automatic synchronization tool or a **backup**
software. It can copy files from A to B, synchronize two places
(can be harmful if you don't pay attention).

Let's see how to use it with an ssh server on which we will
create an encrypted repository to store important data.

[Official documentation](https://rclone.org/)


## Installation

Most of the time, run your package manager to install `rclone`.
It's a single binary.


## Interactive configuration

*You can skip this LONG section if you want to learn what rclone
can do and how to configure it in a 10 lines files.*

There is a parameter to have a question / answer interface to
configure your repository, using `rclone config`.

I'll make a full walkthrough to enable an encrypted repository
because I struggled to understand the logic behind rclone when I
started using it.

Let's start. I'll create an encrypted destination on my local NAS
which doesn't have full disk encryption, so anyone who access the
system won't be able to read my data. First, this will require to
set up an sftp repository and then an encrypted repository using the
previous one as a backend.


Let's create a new config named `home_nas`.

    $ rclone config
    2020/10/27 21:30:48 NOTICE: Config file "/home/solene/.config/rclone/rclone.conf" not found - using defaults
    No remotes found - make a new one
    n) New remote
    s) Set configuration password
    q) Quit config
    n/s/q> n
    name> home_nas


We want the storage type 29, "SSH/SFTP" (I removed all 50+ others
storages for readability).

    Type of storage to configure.
    Enter a string value. Press Enter for the default ("").
    Choose a number from below, or type in your own value
    [...]
    29 / SSH/SFTP Connection
       \ "sftp"
    [...]
    Storage> 29


My host is 192.168.1.200

    ** See help for sftp backend at: https://rclone.org/sftp/ **
    
    SSH host to connect to
    Enter a string value. Press Enter for the default ("").
    Choose a number from below, or type in your own value
     1 / Connect to example.com
       \ "example.com"
    host> 192.168.1.200

I will connect with the username `solene`.

    SSH username, leave blank for current username, solene
    Enter a string value. Press Enter for the default ("").
    user> solene

Standard port 22, which is the default

    SSH port, leave blank to use default (22)
    Enter a string value. Press Enter for the default ("").
    port> 

I answer **n** because I want rclone to use ssh agent, this could
be the ssh password to the remote user, but I highly discourage
everyone from using password authentication on SSH!

    SSH password, leave blank to use ssh-agent.
    y) Yes type in my own password
    g) Generate random password
    n) No leave this optional password blank (default)
    y/g/n> n


Leave this except if you want to provide a private key.

    Raw PEM-encoded private key, If specified, will override key_file parameter.
    Enter a string value. Press Enter for the default ("").
    key_pem> 

Leave this except if you want to provide a PEM-encoded private key.

    Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.
    
    Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
    
    Enter a string value. Press Enter for the default ("").
    key_file> 


Leave this except if you need to use a password to unlock your
private key. I use ssh agent so I don't need it.

    The passphrase to decrypt the PEM-encoded private key file.
    
    Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys
    in the new OpenSSH format can't be used.
    y) Yes type in my own password
    g) Generate random password
    n) No leave this optional password blank (default)
    y/g/n> n


If your user agent manage multiples keys, you should enter the
correct value here, I only have one key so I leave it empty.

    When set forces the usage of the ssh-agent.
    
    When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key is
    requested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errors
    when the ssh-agent contains many keys.
    Enter a boolean value (true or false). Press Enter for the default ("false").
    key_use_agent> 


This is a question about crypto, accept the default except if you
have to connect to old servers.

    Enable the use of insecure ciphers and key exchange methods. 
    
    This enables the use of the following insecure ciphers and key exchange methods:
    
    - aes128-cbc
    - aes192-cbc
    - aes256-cbc
    - 3des-cbc
    - diffie-hellman-group-exchange-sha256
    - diffie-hellman-group-exchange-sha1
    
    Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.
    Enter a boolean value (true or false). Press Enter for the default ("false").
    Choose a number from below, or type in your own value
     1 / Use default Cipher list.
       \ "false"
     2 / Enables the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange.
       \ "true"
    use_insecure_cipher> 


We want to keep hashcheck feature so just skip the answer to keep
the default value.

    Disable the execution of SSH commands to determine if remote file hashing is available.
    Leave blank or set to false to enable hashing (recommended), set to true to disable hashing.
    Enter a boolean value (true or false). Press Enter for the default ("false").
    disable_hashcheck> 

We are at the end of the configuration, we are proposed to change
more parameters but we don't need to.

    Edit advanced config? (y/n)
    y) Yes
    n) No (default)
    y/n> n


Now we can see the output of the configuration file of rclone in
regards to my `home_nas` destination. I agree with the configuration
to continue.

    Remote config
    --------------------
    [home_nas]
    type = sftp
    host = 192.168.1.200
    user = solene
    --------------------
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d> y


Here is a summary of the configuration, we have only one remote
here.

    Current remotes:
    
    Name                 Type
    ====                 ====
    home_nas             sftp


In the menu, I will choose to add another remote.  Let's name it
`home_nas_encrypted`

    e) Edit existing remote
    n) New remote
    d) Delete remote
    r) Rename remote
    c) Copy remote
    s) Set configuration password
    q) Quit config
    e/n/d/r/c/s/q> n
    name> home_nas_encrypted

We will choose the special storage `crypt` which work on an existing
backend.

    Type of storage to configure.
    Enter a string value. Press Enter for the default ("").
    Choose a number from below, or type in your own value
    10 / Encrypt/Decrypt a remote
       \ "crypt"
    Storage> 10

To this question, we will define we want the data stored to
`home_nas_encrypted` being saved in `home_nas` remote in the
`encrypted_repo` directory.

    ** See help for crypt backend at: https://rclone.org/crypt/ **
    
    Remote to encrypt/decrypt.
    Normally should contain a ':' and a path, eg "myremote:path/to/dir",
    "myremote:bucket" or maybe "myremote:" (not recommended).
    Enter a string value. Press Enter for the default ("").
    remote> home_nas:encrypted_repo

Depending on the level of obfuscation your choice may vary.  The
simple filename obfuscation is fine for me.

    How to encrypt the filenames.
    Enter a string value. Press Enter for the default ("standard").
    Choose a number from below, or type in your own value
     1 / Encrypt the filenames see the docs for the details.
       \ "standard"
     2 / Very simple filename obfuscation.
       \ "obfuscate"
     3 / Don't encrypt the file names.  Adds a ".bin" extension only.
       \ "off"
    filename_encryption> 2

As for the directory names obfuscation, I recommend to enable it,
otherwise that leave the whole directory tree readable!

    Option to either encrypt directory names or leave them intact.
    
    NB If filename_encryption is "off" then this option will do nothing.
    Enter a boolean value (true or false). Press Enter for the default ("true").
    Choose a number from below, or type in your own value
     1 / Encrypt directory names.
       \ "true"
     2 / Don't encrypt directory names, leave them intact.
       \ "false"
    directory_name_encryption> 1

Type the password that will be used to encrypt the data.

    Password or pass phrase for encryption.
    y) Yes type in my own password
    g) Generate random password
    y/g> y
    Enter the password:
    password:
    Confirm the password:
    password:

You can add a salt to the passphrase, I choose not too.

    Password or pass phrase for salt. Optional but recommended.
    Should be different to the previous password.
    y) Yes type in my own password
    g) Generate random password
    n) No leave this optional password blank (default)
    y/g/n> 


No need to change advanced parameters.

    Edit advanced config? (y/n)
    y) Yes
    n) No (default)
    y/n> n

Here is a summary of the configuration of this remote backend.
I'm fine with it.

    Remote config
    --------------------
    [home_nas_encrypted]
    type = crypt
    remote = home_nas:encrypted_repo
    directory_name_encryption = true
    password = *** ENCRYPTED ***
    --------------------
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d> y


We see we have now two remote backends, one with the crypt type.

    Current remotes:
    
    Name                 Type
    ====                 ====
    home_nas             sftp
    home_nas_encrypted   crypt

Quit rclone, the configuration is done.

    e) Edit existing remote
    n) New remote
    d) Delete remote
    r) Rename remote
    c) Copy remote
    s) Set configuration password
    q) Quit config
    e/n/d/r/c/s/q> q


## Configuration file

The previous configuration process only produced this short
configuration file, so you may copy/paste from it and adapt to add
more backends if you want, instead of doing the tedious `config`
process.

Here is my file `~/.config/rclone/rclone.conf` on my desktop.

    [home_nas]
    type = sftp
    host = 192.168.1.200
    user = solene
    
    [home_nas_encrypted]
    type = crypt
    remote = home_nas:encrypted_repo
    directory_name_encryption = true
    password = GDS9B1B1LrBa3ltQrSbLf1Vq5C6VbaA1AJVlSZ8


## First usage

Now we defined our configuration, we need to create the remote
directory that will be used as a backend, this is important to avoid
errors when using rclone, this is a simple step required only once.

    $ rclone mkdir home_nas_encrypted:

On the remote server, I can see a `/home/solene/encryted_repo`
directory. It's now ready to use!


## A few commands

**rclone** has a LOT of commands available, I will present a few
of them.


### Copying files to/from backend

Let's say I want to copy files to the encrypted repository.  There
is a `copy` command.

    $ rclone copy /home/solene/log/templates home_nas_encrypted:blog_template  

There are no output by default when the program runs fine.  You can
use `-v` flag to have some verbose output (I prefer it).


### List files on a remote backend

Now, we want to see if the files were copied correctly, we will use
the `ls` command.

    $ rclone ls home_nas_encrypted:
          299 blog_template/article.tpl
          700 blog_template/gopher_head.tpl
         2505 blog_template/layout.tpl
          295 blog_template/more.tpl
          236 blog_template/navigation.tpl
           57 blog_template/one-tag.tpl
           34 blog_template/page.tpl
          189 blog_template/rss-item.tpl
          326 blog_template/rss.tpl

We can also use `ncdu` to mimic the **ncdu** program displaying a
curses interfaces to visualize disk usage in a nice browsing tree.

    $ rclone ncdu home_nas_encrypted
    -- home_nas_encrypted: ------------------
      6.379k [##########] /blog_template


### The sync command

Files and directories can also be copied with the `sync` command,
but this must be used with care because it makes sure the destination
matches exactly the origin when you use it. It's the equivalent of
`rsync -a --delete origin/ destination/`, so any extra files will
be removed! Note that you can use `--dry-run` to see what will
happen.


### Filters

When you copy files using the various available method, instead of
using a path, you can provide a filter file or a list of paths to
transfers. This can be very efficient when you want to recover
specifics data.

The documentation about
[filtering is available here](https://rclone.org/filtering/)


### Parameters

**rclone** supports a lot of parameters, like to limit upload
bandwidth, copy multiples files at once, enable an interactive mode
in case of file deletion/overwriting.


### Mount

On Linux, FreeBSD and MacOS, **rclone** can use a FUSE filesystem
to mount the remote repository on the filesystem, making its uses
totally transparent.

This is extremely useful, avoiding the tediousness of the get/put
paradigm of **rclone**.

**This can even be used to make an encrypted repository on the local
filesystem! :)**


### Create a webdav/sftp/ftp server

**rclone** has the capability of act as a server and expose a
configured remote backend on various network protocol like webdav,
sftp, ftp, s3 (minio) !

The
[serv document is available here](https://rclone.org/commands/rclone_serve/)

Example running a simple webdav server with hardcoded login/password:

    $ rclone serv webdav --user solene --password ANicePassword home_nas_encrypted: