How to grab from multiple sources ?

You can add multiple sources in the sources field of the configuration file:

ftp|sftp:
  ...
  sources:
    - /path1
    - /path2/folder

What kind of CRON expression can I use for scheduling ?

A CRON expression represents a set of times, using 6 space-separated fields.

  • 0 */30 * * * * will launch a job every 30 minutes.
  • 15 * * * * * will launch a job every 15 seconds.

More examples can be found on the official library documentation.

What Regexp semantic is used to filter inclusions/exclusions ?

FTPGrab uses Compile to parse regular expressions. This means the regexp returns a match that begins as early as possible in the input (leftmost) like Perl, Python, and other implementations use. You can test your regular expression on regex101.com and select Golang flavor. Check this quick example.

What logs look like ?

Here is a sample output:

Tue, 29 Jan 2019 22:23:58 CET INF Starting FTPGrab 5.0.0
Tue, 29 Jan 2019 22:23:58 CET INF ########
Tue, 29 Jan 2019 22:23:58 CET INF Connecting to 198.51.100.0:21...
Tue, 29 Jan 2019 22:23:58 CET INF Grabbing from /complete/
Tue, 29 Jan 2019 22:23:59 CET INF --------
Tue, 29 Jan 2019 22:23:59 CET INF Checking /complete/Burn.Notice.S06E16.VOSTFR.HDTV.XviD.avi
Tue, 29 Jan 2019 22:23:59 CET INF Never downloaded
Tue, 29 Jan 2019 22:23:59 CET INF Downloading file (184.18MB) to /tmp/seedbox/Burn.Notice.S06E16.VOSTFR.HDTV.XviD.avi...
Tue, 29 Jan 2019 22:24:47 CET INF File successfully downloaded!
Tue, 29 Jan 2019 22:24:47 CET INF Time spent: 48 seconds
Tue, 29 Jan 2019 22:24:47 CET INF --------
Tue, 29 Jan 2019 22:24:47 CET INF Checking /complete/Burn.Notice.S06E17.VOSTFR.HDTV.XviD.avi
Tue, 29 Jan 2019 22:24:47 CET INF Never downloaded
Tue, 29 Jan 2019 22:24:47 CET INF Downloading file (186.27MB) to /tmp/seedbox/Burn.Notice.S06E17.VOSTFR.HDTV.XviD.avi...
Tue, 29 Jan 2019 22:25:40 CET INF File successfully downloaded!
Tue, 29 Jan 2019 22:25:40 CET INF Time spent: 50 seconds
Tue, 29 Jan 2019 22:25:40 CET INF --------
Tue, 29 Jan 2019 22:25:40 CET INF Checking /complete/test_special_chars/123.bin
Tue, 29 Jan 2019 22:25:40 CET INF Never downloaded
Tue, 29 Jan 2019 22:25:40 CET INF Downloading file (33.27kB) to /tmp/seedbox/test/test_special_chars/123.bin...
Tue, 29 Jan 2019 22:25:42 CET ERR Error downloading, retry 1/3 error="dial tcp 198.51.100.0:21: connect: connection refused"
Tue, 29 Jan 2019 22:25:42 CET INF --------
Tue, 29 Jan 2019 22:25:42 CET INF Checking /complete/test_special_chars/123.bin
Tue, 29 Jan 2019 22:25:42 CET INF Exists but size is different
Tue, 29 Jan 2019 22:25:42 CET INF Downloading file (33.27kB) to /tmp/seedbox/test/test_special_chars/123.bin...
Tue, 29 Jan 2019 22:25:44 CET ERR Error downloading, retry 2/3 error="dial tcp 198.51.100.0:21: connect: connection refused"
Tue, 29 Jan 2019 22:25:44 CET INF --------
Tue, 29 Jan 2019 22:25:44 CET INF Checking /complete/test_special_chars/123.bin
Tue, 29 Jan 2019 22:25:44 CET INF Exists but size is different
Tue, 29 Jan 2019 22:25:44 CET INF Downloading file (33.27kB) to /tmp/seedbox/test/test_special_chars/123.bin...
Tue, 29 Jan 2019 22:25:46 CET ERR Error downloading, retry 3/3 error="dial tcp 198.51.100.0:21: connect: connection refused"
Tue, 29 Jan 2019 22:25:46 CET ERR Cannot download file error="dial tcp 198.51.100.0:21: connect: connection refused"
Tue, 29 Jan 2019 22:25:46 CET INF Time spent: 6 seconds
Tue, 29 Jan 2019 22:25:46 CET INF --------
Tue, 29 Jan 2019 22:25:46 CET INF Checking /complete/exlcuded_file.txt
Tue, 29 Jan 2019 22:25:46 CET INF Not included
Tue, 29 Jan 2019 22:25:46 CET WRN Skipped: Not included
Tue, 29 Jan 2019 22:25:46 CET INF ########
Tue, 29 Jan 2019 22:25:51 CET INF Finished, total time spent: 1 minute 56 seconds

What’s the structure of the webhook notification ?

An HTTP request is sent with a JSON format response that looks like :

{
  "ftpgrab_version": "5.2.0",
  "server_ip": "10.0.0.1",
  "dest_hostname": "my-computer",
  "journal": {
    "entries": [
      {
        "file": "/test/test_changed/1GB.bin",
        "status_type": "skip",
        "status_text": "Not included"
      },
      {
        "file": "/test/test_changed/56a42b12df8d27baa163536e7b10d3c7.png",
        "status_type": "skip",
        "status_text": "Not included"
      },
      {
        "file": "/test/test_special_chars/1024.rnd",
        "status_type": "success",
        "status_text": "1.049MB successfully downloaded in 513 milliseconds"
      }
    ],
    "count": {
      "success": 1,
      "skip": 2
    },
    "duration": "12 seconds"
  }
}

What type of report is sent when it is completed ?

Here is a mail sample:

How can I edit/remove some entries in the database ?

FTPGrab currently uses the embedded key/value database bbolt.
You can use boltBrowser which is a GUI web-based explorer and editor or this CLI browser to remove some entries.