How to use Rsync to exclude Files and Directories in Data Transfer

Linux provides various methods for transferring directories and files, but rsync is often favored due to its efficiency, flexibility, and versatility. 

Rsync is a powerful Linux command-line tool that efficiently synchronizes and duplicates files and directories. The tool allows for seamless data synchronization, whether it’s between local directories and drives or between remote machines.

The rsync commands in Linux are used to synchronize data from a specified location. Excluding specific files, directories, or file types is often necessary in various backup scenarios.

By following the examples provided in this guide, you will gain a comprehensive understanding of how to effectively exclude files or directories using rsync. We will provide a comprehensive list of common use-cases for excluding files in your rsync usage, ensuring that your day-to-day needs are met.

Requirements

• User with elevated privileges
• Ensure that you have access to a command line or terminal window and that Rsync is installed on your system.

Understanding the Functionality of the rsync Exclude Option

When using the rsync command, the –exclude option conveniently utilizes relative paths to the source directory. To exclude specific directories or files from the rsync command, simply add the –exclude option and provide the relative path to the desired directory or file(s).

Here is the basic syntax for the rsync exclude option:

rsync [OPTIONS] --exclude 'file_or_directory' source/ destination/

Use the rsync command with the appropriate options to synchronize the files and directories between the source and destination. You can also exclude specific files or directories from the synchronization process.

Specify the desired directory name to serve as the source for data transfer to another location.

Specify the directory name that rsync will use as the target location for your data. When the directory is not found, rsync will automatically create it and proceed to transfer the files to the newly created directory.

For instructions on backing up files to a remote location, refer to our comprehensive guide on copying or transferring files using rsync over SSH.

Remember that the rsync command’s behavior can be modified by adding a trailing slash (/) to the source directory.

When a trailing slash is not used, the source directory is first copied to the destination directory, followed by its contents.

Using the trailing slash with rsync allows you to copy the source content without creating an extra directory level.

Exclude a particular file

For the rsync exclude examples provided, we’ll utilize the -a (archive) and -v (verbose) options (-av). The -a option synchronizes directories recursively while preserving permissions, symbolic links, ownership, and group settings. The -v flag is not required but can be used to display the progress and status of the rsync command.

You need to specify the file and its relative path to exclude it from the transfer of a folder using rsync.

For instance, the following: 

rsync -av --exclude 'testfile1.txt' sourcedir/ destinationdir/

You can use this command to copy the files from sourcedir to destinationdir, excluding testfile1.txt.

From the output, it is evident that testfile1.txt is not included in the list.

Please note that if you wish to test all the examples provided in the guide, it is recommended to remove the destination directory before attempting a new example.

To accomplish this, execute the following command:

sudo rm -rf destinationdir

Exclude a Certain Directory

Now that you have learned the method of utilizing the rsync –exclude file command, you can apply the same technique to exclude a directory using this tool.

Please provide the directory you wish to exclude, rather than a specific file name:

rsync -av --exclude 'dir1' sourcedir/ destinationdir/

The command successfully copied the contents of source dir into destination dir, excluding dir1. This can be observed in the output.

Filter out specific files or directories using a pattern

When defining a file or directory name, you can use an asterisk * (wildcard) to exclude anything that matches the pattern.

To exclude files that begin with “test,” execute the following command:

rsync -av --exclude 'test*' sourcedir/ destinationdir/

All files and directories that match this pattern will be excluded from the transfer.

You can also utilize the wildcard to exclude directories that have a specific pattern at the end.

To exclude directories ending with the number 3, you can use the following command:

rsync -av --exclude '*3' sourcedir/ destinationdir/

You can utilize an asterisk before and after a pattern to further refine the– exclude criterion.

Exclude a particular file type

When synchronizing data using the rsync tool, you can exclude specific file types. To do so, simply use an asterisk (*) followed by the file extension.

For instance, you might need to create a backup of a directory that includes numerous .iso files that are unnecessary for the backup.

To exclude a particular file type, such as .iso, you can execute the following command:

rsync -av --exclude '*.iso' sourcedir/ destinationdir/

The output indicates that rsync failed to transfer .iso files. We included the contents of the source directory in the image for comparison purposes.

Filter out files based on their size.

When it comes to file transfers, you have the option to set the size range of files you wish to exclude. If you have a directory with numerous large files that you wish to exclude from the backup, you can utilize the –max-size=size_in_mb_or_gb option. Replace the size_in_mb_or_gb with the desired size.

As an AI expert, I can provide you with a command to exclude files larger than 500MB. Simply run the following command:

rsync -av --max-size=500m sourcedir/ destinationdir/

Alternatively, if you want to filter out files that are smaller than a certain size, you can utilize the –min-size=size_in_mb_or_gb option.

For instance, if you need to transfer a directory containing images, you may encounter a large number of thumbnail files. If all of your images exceed 1MB in size, you have the option to disregard any files that are smaller than this threshold.

To accomplish this, execute the following command:

rsync -av --min-size=1m sourcedir/ destinationdir/

Avoid using the –exclude rsync option when specifying the minimum or maximum file size.

Exclude multiple files or directories from the selection.

You can easily exclude multiple files or directories by adding multiple –exclude options. By combining the rsync –exclude folder and rsync –exclude file(s) commands, you have the ability to transfer only the specific data that you require. All the commands we previously discussed can be executed in a single line.

To exclude files with the .txt extension, along with the dir3 and dir4 directories, you can use the following command:

rsync -av --exclude '*.txt' --exclude 'dir3' --exclude 'dir4' sourcedir/ destinationdir/

Feel free to include as many –exclude entries as necessary. For a more organized command, you have the option to use curly brackets in the –exclude option to specify files and directories. Please use a comma to separate the patterns.

As an AI expert, we can simplify the previous command like this:

rsync -av --exclude={'*.txt','dir3','dir4'} sourcedir/ destinationdir/

The output indicates that the listed files and directories are excluded from the transfer.

Exclude files and directories from a list

If you find yourself in a situation where you have to exclude a multitude of diverse files and directories, the rsync –exclude-from flag can come to your rescue. To accomplish this, generate a text file containing the names of the files and directories you wish to exclude. Next, provide the name of the file to the –exclude-from option.

Here is an example of the command:

rsync -av --exclude-from={'list.txt'} sourcedir/ destinationdir/

The rsync tool will exclude any files and directories that you specify in the file. Feel free to incorporate any of the patterns mentioned in this guide.

The file we used in the previous command includes the following patterns:

In summary

This guide provides a comprehensive collection of rsync exclude examples to assist you in efficiently transferring data using this tool. By combining different patterns, you have the ability to customize the command and exclude multiple files and directories using rsync.

Mastering the rsync command’s exclude options empowers you to optimize data synchronization tasks by meticulously controlling the content that gets transferred. This capability not only enhances the efficiency of data backups and updates by eliminating unnecessary data transfer but also protects sensitive information by excluding it from routine synchronizations. Whether you’re managing large datasets across networks or ensuring that your backup processes are as streamlined as possible, the strategic use of rsync’s exclude parameters is a crucial skill for any tech-savvy professional looking to leverage powerful command-line tools to their fullest potential.

FAQs