GMbackup -- Gentle Music's backup tools

GMbackup consists of scripts that support the making of local incremental backups, local backups to a separate disk and remote backups. The scripts herein are written the Groovy programming language run in a Gradle environment. Groovy and Gradle rely on the Java framework.

The GMbackup tools do not cover backing up Windows. The recommendation is to let these backup scripts handle the home directory's important subdirectories like OneDrive and Documents. When backing up Windows, remove these directories subject to backups by GMbackup before starting the Windows backup to minimize the size of the Windows backup.

Still to be documented

• wrapper Bash scripts
• other Bash scripts

Assumptions

Directory creation or modification dates do not have to be considered.

A file is considered to be moved if

• it has a certain size and a certain date and time in the source tree AND
• has the same size as well as the same date and time on the local backup disk but has a different location or name on the local backup disk.

File and directory names are in UTF-8.

A large USB disk is required to create local backups, probably 50 to 100 percent larger than the amount of data to be backed up

Another large USB disk is required to create remote backups, as large as the local backup disk.

If running Windows, neither the Cygwin tree nor the MSYS2 tree must be subject to GMbackup, due to bad symlinks causing problems for programs like rsync.

The remote system must run the same Unix system as the local system, because the remote system runs scripts generated by the local system.

The reader understands that the file name syntax in this document is Unix-like. If stating a file named /e/gmbackup, it implies that if run on Windows, this file's native name is E:\gmbackup and if run in the Cygwin environment, the name is /cygdrive/e/gmbackup.

Introductory View

The four columns of disks represent:

1. the source trees subject to backup
2. the GMbackup data store including incremental backups
3. local backups
4. remote backups

Concepts

This section introduces concepts used documenting GMbackup.

Source files are files subject to be backed up.

Installation directory is an arbitrary directory where all the GMbackup scripts are installed and executed. If run on Windows, most scripts require administrator privileges.

Incremental backups are tar backup archives holding the modified and new files since the last complete backup or the last incremental backup. The scripts working with incremental backups mark files as being incrementally backed up by creating empty marker files in a shadow directory with today's date. The tar backup archives are stored in the GMbackup data store's top directory, like /e/gmbackup.

The GMbackup directory is this tool's data store, holding information required by GMbackup and incremental backups. Example of such a directory is /e/gmbackup. It has the following structure:

• subdirectories keeping track of various disks and file trees to be part of backups. Subdirectory name example: /e/gmbackup/host1-documents. In each subdirectory the following files may reside:
  • backup.file.exclusions listing files to be excluded from backups.
  • backup.directory.exclusions enumerating directories to be excluded.
  • backup.dirs-with-dupes.exclusions enumerating directories to be excluded. The directory names enumerated herein are to be avoided when detecting moved files as these directories may contain identical copies of files.
  • directory trees keeping track of the incremental backups, also called shadow trees.
  • empty files making up the incremental backup marking name and date of source tree files
  • empty <filename>,i files marking that and when a source file has been incrementally backed up.
• incremental tar files. Example: /e/gmbackup/host1-documents-<date>-<index>.tar, where index is that day's incremental backup number. Upon a regular backup to the local backup disk, these tar files are removed.

The layout of the GMbackup data store is detailed here.

Local backups are backups made to the local backup disk. Carrying out a local backup, the incremental empty marker files and the incremental tar files are removed. After a local backup, the file trees on the local backup disk are identical to the source file trees.

Remote backup sets are tar backup archives and some scripts to be delivered to a remote site to be remotely unpacked. After the unpacking of the tar archives of such backup sets, the remote disk will be a copy of the local backup disk.

List files -- Essential to backing up only new and modified files, GMbackup creates list files containing detailed information about a file tree or a disk. Here's a fragment of such a list file.

# L:\hostx
# /l/hostx

.:

  2025-06-15 22:00.279              76 nmdsdcid

cmd:

  2023-01-09 10:11.297              45 scripts.txt
  2023-04-24 15:49.214            1582 pvr-backup.sh

documents/old/Site_files:

  2020-05-15 13:19.422            6977 ajax_view.js.download

NOTE: File creation or modification times have a granularity in milliseconds. Some older files or files not created in Windows or Linux may have a second as the smallest unit, making a time represented as 1998-05-23 13:14.000.

Once having complete backups by GMbackup, these list files are handy when looking for files in a computer's various disks and files trees backed up by GMbackup, particularly using the Bash script find.sh.

These lists files are crucial when comparing file structures carried out by GMbackup's scripts.

Scripts calling scripts -- All logic is implemented using Gradle/Groovy and Bash scripts. In the usual case, a Bash script calls a Gradle script. The Gradle script produces a Bash script with backup actions as regular Bash commands. This implies that backup actions can be reviewed and checked for consistency before being executed.

Prerequisites

For a working GMbackup environment, the following software and disks are expected:

• A Java 11 runtime environment
  • Newer Java versions may work but have not been tested.
• If running Windows:
  • Cygwin or
  • MSYS2.
  • Git for Windows is not supported.
  • WSL is not supported.
• A large local backup disk
• A large remote backup disk, if the mirroring of the local backup disk on another site is desired.

Included in the GMbackup distribution is the Gradle wrapper. The Gradle wrapper is a set of Java archives capable of downloading a minimal Gradle runtime environment required to run the GMbackup Groovy scripts.

Recommended MSYS2 additional packages

• mingw-w64-x86_64-emacs
• rsync
• openssh
• git
• tree

Recommended Cygwin packages

• Base
• Doc
  • man-pages-linux
• Editors
  • nano
  • emacs
• Net
  • curl
  • openssh
  • rsync
  • tftp
  • wget
  • wget2
• Utils
• Text

The Bash command executable must be version 4.4.12(3)-release or higher. The GMbackup package has been successfully tested with Bash version 5.2.21(1)-release.

Environment

Locally, there are four directories in use and a large backup disk. The local directories used are:

1. The script directory -- This is the working directory where all GMbackup scripts are located.

   • lists -- This subdirectory holds list files in detailing the contents of the local backup disk and the remote disk. Scripts that create remote backup sets use these list files for comparison as sources for making a remote backup. List files have the name syntax <location>-after-backup-<backup-no>-<os>.list, location being local or remote.

2. The GMbackup directory -- This is where incremental backups are stored and where the GMbackup exception files are stored. The directory contains a full tree copy of the source file tree, but all files are empty and are are called marker files. More on the GMbackup directory tree is found here.

3. The local backup disk or directory -- Where complete local backups are stored. Normally this is a separate and detachable disk, but it can be a directory on a local disk, too.

4. The directory for remote backup sets -- This is where incremental remote backup archives and scripts are to be stored before being transferred to the remote site.

On the remote computer being the host of the remote disk, there are two directory trees of importance:

1. The working directory for remote backup increments -- Where incremental tar backup archives are received during transfer.

2. The backup disk or directory -- Where a shadow of the local backup disk or directory is mirrored.

On Windows, most scripts require Administrator privileges.

GMbackup properties

In the working script directory, a properties file is expected describing some locations required by GMbackup called gmbackup-<hostname>-<os>.properties. Here's an example:

    gmbackupDirectory=/e/gmbackup
    localBackupRoot=/l/hostx
    localRemoteTarget=/e/not-to-be-backed-up

Backup operations files

Backup operations files are text files where each line describes a disk or file tree to be backed up by scripts. Scripts invoke this file line by line. These files reside in the working (script) directory, too. The syntax of such files is described by the following example named hostx-operations.trees:

    # to: a relative path, possibly under E:/gmbackup and L:/hostx
    from=~/cmd to=cmd
    from=~/Desktop to=desktop
    from=~/Documents to=documents
    from=~/Pictures to=pictures
    from=~/Videos to=videos
    from=/e/dev to=e-dev
    from=/w to=w-wd12tb

An instruction line has the following syntax:

• from= is placeholder for where to state a directory tree to be backed up.

• The directory top or disk root to be backed up in relaxed Windows or Cygwin syntax.

• A space delimiter

• to= is a placeholder where to state the name of the top backup directory to hold the incremental backup

• This field has two purposes: 1. The name of the subdirectory in the GMbackup directory where to hold incremental backup information and 2. the top directory on the backup disk or backup top directory. Here are two directory location examples emanating from this name, if set to documents:

  1. /e/gmbackup/documents -- the GMbackup data store directory holding incremental backup and exclusion descriptions
  2. /l/hostx/documents -- the location of the local disk backup of /c/Users/userx/Documents

Example:

from=/e/dev to=e-dev

This line tells the scripts that the directory tree /e/dev is subject to backups.

The example's to= parameter (a relative path) depicts two top directories, here set to hostx-e-dev:

1. /e/gmbackup/e-dev: the location of GMbackup data store, set by the property gmbackupDirectory

2. /l/hostx/e-dev: the local backup disk set by the property localBackupRoot

In detail:

1. The GMbackup directory, possibly /e/gmbackup, will have a subdirectory e-dev where all files from /e/dev will be shadowed as empty files dated the day of an incremental backup. The directory /e/gmbackup/e-dev may also contain the files backup.file.exclusions and backup.directory.exclusions describing directories (by relative paths) and files to be excluded from the backup. The root, /e/gmbackup, was stated as the gmbackupDirectory property in the properties file above.

2. The backup disk, here /l (or possibly a backup disk tree, /e/local-backups), will hold the directory e-dev where copies of files from /e/dev will be backed up and mirrored. The root, /l was stated as the localBackupRoot in the properties file above. Following this example, the backup disk's top directory will be /l/hostx/e-dev.

Using this example, let's assume there is a file called /e/dev/expenses-2024.docx, dated 2024-10-31 11:45.

• When an incremental backup is run on November 1, an empty marker file /e/gmbackup/e-dev/expenses-2024.docx,i, dated 2024-11-01 (some time) will be created if there is no incremental empty file there. The file is also added to a tar file in the same top directory. Please note the marker file's suffix ,i.

• When a complete local backup is run, the source file will be copied to to /l/hostx/e-dev, with the same date, if there is not already there or if the backup copy is older. Additionally, a marker file being an empty file with exactly the same name is created in the same relative directory in the GMbackup data store. In this example that empty marker file is created in /e/gmbackup/e-dev as expenses-2024.docx. The incremental marker file with the ,i suffix is removed during the local backup procedure.

NOTE: The from= and to= strings are used in subsequent script commands.

Scripts supporting incremental backups

• create-incremental-backup.sh -- By calling make-incr-backup-script.gradle, this script creates two Bash scripts:
  1. backup-incrementally-<date>.sh
  2. purge-incrementally-<date>.sh

to be run manually.

Main script for local backups and remote backup preparation

All required steps to make a local backup and to prepare for a remote backup are compiled into one script called full-backup-cycle.sh. The script facilitates the use of the GMbackup set of scripts.

Here's how to read the script's internal documentation:

$ ./full-backup-circle.sh hostx-operations.trees ?
##
## The steps are:
##
##01  1. Compare source directories (the "from" parameter) to the local disk's
##01     corresponding directories (the "to" parameter). looking for directory
##01     and file moves, creating the detected move scripts
##
##02  2. Review the move script created in phase 1.
##
##03  3. Execute the detected moves script, that is, make the corresponding
##03     moves on the local backup disk
##
##04  4. Create the script that backs up the source file trees from the input
##04     file in $1.
##
##05  5. Run the backup by executing the backup and purge scripts from phase 4.
##
##06  6. Create this backup's local backup disk list file,
##06     local-after-backup-<backup-no>-<os>.list
##
##07  7. Apply the moves found in the detected moves script to last remote
##07     disk's list file
##
##08  8. Create the remote backup incremental kit script. The created script
##08     will include the same moves to be made on the remote site.
##
##09  9. Create the remote backup incremental kit.
##
##10 10. Increment the backup number.
##

Steps 1 and 4 iterate over the lines in $1 (the first parameter), hostx-operations.trees used as an example in this document.

These are the details of each step:

1. This first step looks for files and directories that have been moved within each set of file trees to be backed up. The analysis is implemented in detect-moves-in-disk-tree.gradle and is carried out once per line in hostx-operations.trees, that is, once per source file tree. The script detected-moves-<backup-no>-<os>.sh is generated. At its first invocation, the output script move-files-on-target-<backup-no>.sh is created. During the remaining invocations of the Gradle script, the file is appended to.

2. Step 2 is implemented to review the suggested moves on the local backup disk determined in step 1. Simply, the generated script for moved files, detected-moves-<backup-no>-<os>.sh, is displayed.

3. Step 3 executes the script detected-moves-<backup-no>-<os>.sh, hence moving files and directories on the local backup disk, just like they once were at the source tree or disk. If using the remote backup scheme, the same files and directories later will be moved on the remote disk, too.

4. Step 4 creates the scripts backup-to-local-backup-disk-<backup-no>-<os>.sh and purge-local-backup-disk-after-backup-<backup-no>-<os>.sh, by iterating over the lines and file trees in hostx-operations.trees. The analysis is implemented in make-backup-to-back-disk-scripts.gradle. At the first invocation, the output script is created. At the remaining invocations of the Gradle script, the file is appended to.

5. Step 5 runs the newly generated scripts backup-to-local-backup-disk-<backup-no>-<os>.sh and purge-local-backup-disk-after-backup-<backup-no>-<os>.sh.

6. Now at step 6, the local backup disk reflects the various source file trees and the local backup is complete. This step is the first step to enable the comparison between the local backup disk and the remote disk. A complete listing of the local backup disk's contents is run calling reflect-tree.gradle, creating local-after-backup-<backup-no>-<os>.list. The resulting .list file will be used in step 8.

7. In step 7, the former remote backup disk's last list file (backup number - 1) is read and the moves of directories and files detected during step 1 are applied to the list file, as if the moves in the remote backup disk had already been carried out. The Gradle script making this modification to the list file of last remote backup is called apply-moves-to-list-file.gradle and the output list file is called remote-after-backup-<backup-no - 1>-with-moves.list and is used in step 8.

8. In step 8, the list files from the current local backup disk and the list file from step 7, that is, a list of all files on the remote backup disk but reflecting the moves as they had already been made. The script make-remote-backup.gradle is called creating the shell script backup-to-remote-kit-<backup-no>-<os>.sh and one or many tar lists to describe the incremental tar files to be created in step 9.

9. Running step 9, the script created in step 8, backup-to-remote-kit-<backup-no>-<os>.sh is run. The remote backup kit is created in the directory depicted by the property localRemoteTarget. The created kit consists of one or more tar files and generated scripts to be run at the remote site. Also, the text file holding the current backup number, next-backup is added to the remote kit. If using the configuration example above, the remote backup kit will be stored in the directory /e/not-to-be-backed-up/remote-backup-<backup-no>.

10. Step 10 is the final step. Here all generated scripts are moved to the older subdirectory and the backup number is incremented by one in the file next-backup, hence completing the local backup cycle and preparing for the next backup, whenever it is carried out.

Here follows an enumeration of all Gradle scripts.

Script supporting remote backups

Scripts on the remote backup site

The remote kit created by make-remote-backup.gradle has the following contents:

1. next-backup -- Holds the current backup number

2. local-after-backup-<backup-no>-<os>.list -- The complete list of files from the local backup disk after the local backup had completed and for later comparison.

3. remote-site-actions.sh -- This generic script implements four actions:

   1. Unpacking of the kit's scripting environment.
   2. Running the remote site backup script synch-on-remote-<backup-no>.sh script, that optionally moves files moved on the local backup disk, "untars" files that were new to the local backup disk and finally, deletes files that no longer exist on the local backup disk
   3. Making a list of all files on the remote backup disk for comparison: remote-after-backup-<backup-no>-<os>.list
   4. Comparing the two list files as a consistency check. The file remote-after-backup-<backup-no>-<os>.list must be copied back to the local system's working directory and is subject to comparison when the next backup is to be run.

4. remote-toolkit.tar -- The scripting environment required on the remote site:

   1. The Gradle wrapper
   2. GMbackup's Gradle scripts
   3. detected-moves-<backup-no>-<os>.sh -- The moves of files and directories carried out on the local backup disk
   4. synch-on-remote-<backup-no>-<os>.sh -- The generated main script to be run on the remote site, calling tar and removing files that have already been removed on the local backup disk.
   5. tarlists of the backup tar savesets featured

5. to-remote-<backup-no>-<nn>-<os>.tar.gz -- One or many tar savesets containing files to copied to the remote backup disk.

Wrapper scripts

• create-backup-to-backup-disk-scripts.sh -- This Bash script simplifies the use of make-backup-to-backup-disk-scripts.gradle. Parameter:

  • The backup description text file describing what to back up and where to store the backup by naming a root directory. This script is normally not used and replaced by full-backup-cycle.sh.

• create-disk-tree-list.sh -- This Bash script wraps reflect-tree.sh, creating a list file detailing the contents of a disk or a directory tree. Parameters:

  • The disk root path or a directory root path to be reflected in a list file
  • Optional: the name of the list file, default is lists/<device>-yyMMdd.list.

• create-incremental-backup.sh -- Wraps make-incremental-backup.sh. Parameters:

  • The backup description text file describing what to backup and where to store the backup by naming a root directory.

• create-remote-backup.sh -- Simplifies the use of the make-remote-backup.gradle script. Parameters:

  1. The list file describing the current state of the local backup disk.
  2. The list file describing the last known state of the remote backup disk.
  3. The target directory ("the working directory for remote increments") for the remote backup, later to be copied to the remote site.

• create-target-disk-moves.sh -- This Bash script wraps make-move-script-for-target-disk.gradle. Parameters:

  • source disk's or source root's list file to analyze.
  • target disk's list file to analyze and to be prepared for corresponding moves.

• find-moves-on-local-backup-disk.sh -- Wraps detect-moves-in-disk-tree. Parameters:

  • source tree or disk to investigate
  • target tree to investigate

• impose-moves-into-list-file.sh -- This script calls reflect-moves-in-list-file.gradle. Implicit input: move-files-on-target-<date>-<os>.sh. Parameter:

  • list file reflecting the state of the remote host's backup disk

Common Groovy files

The following Groovy files are common object definitions used by multiple Gradle scripts. They are located in the buildSrc/src/main/groovy/se/gentlemusic/common subdirectory.

• CaseInsensitiveDirName.groovy -- Compares directory names alphabetically and case insensitively.
• DirectoryEntry.groovy -- Internal object for directories
• FileEntry.groovy -- Internal object for files

Layout of the GMbackup data store

The GMbackup data store contains:

1. directory trees keeping track of what has been backed up incrementally and locally, as well as
2. descriptions of files and directories to be omitted during backups
3. incremental backup tar savesets

Here's an example of a GMbackup data store:

1. /e/gmbackup -- top directory defined in gmbackup-<hostname>-<os>.properties
   1. documents -- backup directory for C:/userx/Documents named in hostx-operatations.trees.
      1. backup.directory.exclusions -- Enumerates directories to be excluded from being backed up, one directory per line
      2. backup.file.exclusions -- Enumerates files to be excluded from the backup, one filename per line
      3. backup.dirs-with-dupes.exclusions -- Enumerates directories to be excluded from duplicates lookup, one directory per line
      4. aaa -- subdirectory matching C:/userx/Documents/aaa to store empty marker files mapping to the source aaa directory
         1. payroll.xlsx -- empty marker file matching `C:/userx/Documents/aaa/payroll.xslx'
      5. letter.docx -- empty marker file matching C:/userx/Documents/letter.docx. GMbackup uses this file's date to see if the source file has been updated or not when creating a local backup.
      6. letter.docx,i -- empty marker file stating that the source file has been incrementally backed up. The date depicts when it was incrementally backed up.

Getting started

This section describes the steps required to establish a working environment.

Unpack the distribution to the designated working directory.

cd workingDir
tar -xvf gmbackup-1.0.tar

Rename gmbackup-template.properties to gmbackup-<hostname>-<os>.properties.

Currently, os is

• g for GNU/Linux
• c for Windows with Cygwin
• m for Windows with MSYS2

(The letter is the first letter from the uname command in lowercase.)

mv gmbackup-hostname-os.properties gmbackup-hostx-c.properties

Carefully edit the properties file (gmbackup-<hostname>-<os>.properties):

1. Set the gmbackupDirectory property to the directory where to place the GMbackup data store.
2. Set the localBackupRoot property to the local backup disk's root directory. The property can be disk or directory.
3. Set the localRemoteTarget property to the directory where to (locally) store remote intremental backups.

Create the file that enumerates the backup operations, that is, the list of source trees to be backed up and their target backup disk's top name. A typical name for his file is "hostx-operations.trees".

Source directories under the login directory can be specified with a tilde (~).

Here's an example line, based on that the hostname is hostx, the source tree is OneDrive and the target directory is set to a string that can depict the origin and to be used as top directory on in the GMbackup data store and on the local backup disk:

from=~/OneDrive to=onedrive

Remarks:

1. The hostname is optional but nice to have if the local and remote disks are to hold backups from different computers.
2. The from= and to= strings are mandatory
3. Comments can be written to lines starting with a hashmark (#).

The GMbackup data store has to be created as well as the local backup disk roots. The script setup-gmbackup.sh:

1. creates the GMbackup data store root directory, like /e/gmbackup.
2. creates each target root directory based on the to= properties (one per line) in the source operations file. Example: /e/gmbackup/onedrive
3. creates three empty exclusion description files. Example: /e/gmbackup/onedrive/backup.dir.exclusions
4. creates each target root directory on the local backup disk. Example: /l/hostx/onedrive

Handling of moved files

This section details how files moved (or renamed) in a source tree are to be moved/renamed on the local backup disk and remote disk, too.

In the first step of full-backup-cycle.sh, the script detect-moves-in-disk-tree.gradle is called looking for moves in a source disk tree, by comparing the source tree to last backup's list file (local-after-backup-<backup-no>-<os>.list) listing files from the backup disk.

If a file with a certain size and certain last modification modification date appears in different directories or with a different name in the source tree compared to the list file, it is considered to have been moved. Example:

The file payload.docx was moved from the source tree's root to a directory called docs.

$ mv payload.docx docs

The "detect moves" script will create a script snippet looking like this:

safelyMove "payload.docx" "docs/payload.docx"

safelyMove is a Bash procedure moving a file with additional error handling.

Imagine that this move was found in the Documents directory. Following the example from the Backup operations files section above, the local backup disk's target directory is /<backup-drive>/hostx/documents. The safelyMove function calls will be preceeded by a change directory (cd) command to /<backup-drive>/hostx/documents:

cd /<backup-drive>/hostx/documents
safelyMove "payload.docx" "docs/payload.docx"
safelyMove ...

This generated Bash code will be used both backing up the local and remote backup disks.

In order to tell the script make-incremental-backup.gradle to understand that moves will be carried out before actually creating the remote backup kit, the script apply-moves-to-list-file.gradle is called making the changes to the list file (remote-after-backup-<n-1>-<os>.list) representing the current state of the remote disk.

The following edit matches this move example:

Before:

documents:

  2024-11-16 16:28.029            3459 payload.doc
  2025-12-11 23:04.121             433 example.txt
  ...

After:

documents:

  2025-12-11 23:04.121             433 example.txt
  ...

documents/docs:

  2024-11-16 16:28.029            3459 payload.doc

The changes are written to remote-after-backup-<n-1>-<os>-with-moves.list.

This file is then compared to the list file representing the local backup disk after the backup, local-after-backup-<n>-<os>.list when producing the remote backup kit.

Thanks to the edits of the remote disk's last list file, files that have been moved do not need to be added to the remote backup kit.

Recommended file structure on the backup disks

In order to be able to use the local and remote backup disks from different computers, the following structure is recommended:

• <local-disk-root>/hostx
• <local-disk-root>/hosty

and

• <remote-disk-root>/hostx
• <remote-disk-root>/hosty

as used in this document's examples.

[2025-08-25/gm]