The WANFast file copy program, wcp, provides fast file copy, backup, and synchronization over the WAN at speeds up to 100X faster than ftp, sftp, rsync, and other solutions.  It supports both Windows and Linux, and can copy files between them without losing any file attributes or settings.  WANFast's wftp program provides the same functionality but with an interactive command line API similar to the well-known ftp program.  It allows you to manage local and remote file systems and copy files between them.  WANFast's wgui program provides this functionality with an easy-to-user graphical interface.

• Set Up
• Command Syntax
• Basic Options
• Selecting Files to Copy
• File Backup Mode
• Block Level File Synchronization Mode
• Copying and Syncing Block Devices
• Copying Slow Disks and Network File Systems
• Controlling Encyption, Compression, and Other Basic WANFast Options
• Copying Windows Encrypted Files/Directories
• Advanced Options
• Setting wcp Options on the Server
• Auditing
• Integrating with Other Applications
• Changing File System Types -- Support for File System Metadata -- Mapping User IDs
• Exit Codes
• Examples

Set Up

See Getting Started for detailed instructions on how to set up your WANFast environment and run a WANFast command.  Be sure to first set up your WANFast private/public keys and add your public key to the authorized_keys file for an account on the remote computer.  Use wping to test your WANFast session with an account on the target machine.  If wping successfully logs into the target computer but is unable to transfer data, a firewall somewhere between the two machines is probably blocking UDP traffic.  You will either need to configure the firewall to allow WANFast traffic, or use the -w option to transfer file data over a WANFast tunnel.  See wcp Client Command Syntax and Configuration Options for a summary of all wcp syntax and options.

Command Syntax

To copy files between your computer and a second computer running the WANFast software, use one of the commands below:

wcp [OPTIONS] SourcePath [SourcePath ...] TargetPath
wftp [OPTIONS] [[User@]RemoteHost]    wftp Command Syntax 
wgui                                  wgui Command Syntax

For wcp, you must specify at least one source path and a single target path.  For details on wftp or wgui see the links above.  Each source path can be either a file or directory.  The single target path must be a directory.   If a path starts with "/", it specifies an absolute path.  It it doesn't start with "/", then it is a relative path.  On the client side (the computer on which you run the wcp command), a relative path is taken relative to your current working directory.  On Linux only you can begin a path with "~/" to refer to your home directory.  On the remote computer, paths are relative to the home directory of the user account that you use to run the operation. 

Two things to remember when running on Windows:  1) use the forward slash, "/", as the directory separator; 2) you can include a drive designator in an absolute path, e.g., "D:/MyPath".

Depending upon which direction you wish to copy the files, either the TargetPath or the first SourcePath must include the remote host specifier:

[User@]RemoteHost:Path

User optionally specifies the account to use for accessing files on the remote computer.   If you do not specify User, your current account ID will be used.  RemoteHost must specify either the host name or IP address of the remote computer.  In order for the connection to complete,

1. The remote machine must be running the wfstd daemon process.
2. The port used by the wfstd process cannot be blocked by a firewall.
3. If the wfstd process has been configured with a port other than 4900, then you must specify this non-standard port on the wftp command line using the --server-port=PORT option.
4. You must have a valid WANFast key pair configured on the local machine.  This key pair must either be in the default location for your current account, or must be specified on the command line using the -i KeyPath option.
5. The security critical WANFast files on the local machine must have the proper security attributes set.  If you get an error message telling you that you have insecure file permissions, run the wfstPerms command to set them correctly.
6. The target account (UserName) on the remote machine must have a valid authorized_keys file containing your WANFast public key.
7. The security critical WANFast files on the remote machine must have the proper security attributes set.

On the command line, you must always include the colon, ":", after the RemoteHost, even if using a default path.   Thus, in a put operation to copy files from your computer to a 2nd computer, you would use the command:

wcp [OPTIONS] LocalPath LocalPath ... [User@]RemoteHost:RemotePath

For a get operation to copy files from the 2nd computer to your computer, you would use the command:

wcp [OPTIONS] [User@]RemoteHost:RemotePath RemotePath ... LocalPath

Note that in the 2nd form of the command, only the first SourcePath should include the remote host specifier.  All paths are considered remote except for the final target path on the local computer.  Also note that the target path must always be a directory, even when transferring a single file.  If the directory does not exist on the target computer, it will be created.  Thus there is no way to change the name of a file when copying with wcp.  The command "wcp ~/xx RemoteHost:yy" will copy the file "xx" from your home directory to the file "/YourHomeDirectory/yy/xx" on the remote computer.

You can specify multiple operations with a single command using the --op-list=PATH option to specify a file containing additional operations.  Each line of the file should specify a separate operation using the same format as the command line:

SrcPath SrcPath ... SrcPath TargetPath

The orientation of the additional operations must match the command line -- that is, if the command line is a put operation that specifies a set of local paths to copy to a remote host, then SrcPath in the operations file must specify a path on the local computer and TargetPath must specify a path on the remote computer.  The reverse would be true for a get operation.  Do not include the username and remote host address for the additional operations.  They will run using the same connection as created by the command line.  

When additional operations are specified, they will run sequentially in order, starting with the command line and then the first operation in the file.

(back to top)

Basic Options

wcp, wftp, and wgui all support the same options that are described in the next several sections.  The option format shown in this document is correct for both wcp and wftp.  For wgui options are specified in the various options menus as described in the wgui User's Manual.  The follow functional descriptions are valid for all three.

• -V, --verbose controls the amount of information that wcp will print to standard output.  By default (without -V) wcp prints a status line with the total number of bytes currently copied so you can track its operation.  It will also print , with a summary at the end to provide totals for the number of files copied, errors, etc.  to print the file names being transfered and useful status information as it works.  You can repeat this option up to three times to get increasing levels of status.
• -Q, --quiet turns on quiet operation.  Only errors will be printed to standard output.
• -r, --recurse  causes wcp to recurse into any directories that it encounters, sending the contents of the directory. 
• -a, --archive  causes wcp to copy all file attributes from the source to the target along with the file data, provided that the the user accounts on the source and target computers are sufficiently privileged to read and write the attributes, respectively.
• -T, --preserve-times  If you do not want to preserve all file attributes from source to target (perhaps due to missing user IDs on the target), then you can use this option to preserve just the file modification time to aid in later sync operations.
• -t, --test-drive wcp tells you what changes it would make without actually making them.
• -A, --absolute-paths  The files created on the target computer will use the full absolute path of the original file on the source computer.   Without this option, the command "wcp /etc/hosts RemoteHost:/tmp" would result in the file "/tmp/hosts" on the target computer.  When using this option you would wind up with "/tmp/etc/hosts".
• -K, --overwrite By default wcp will refuse to overwrite existing files on the target.  Using this option will force the overwrite of any existing files.
• --newer  Only overwrite a file if the file on the source is newer than the file on the target.
• -S, --follow-links By default, when wcp encounters a symlink (or reparse point on Windows), it will recreate the link on the target.  When using this option wcp will follow the link and copy the linked file rather than the link.
• -o, --preallocate-files  When using this option, wcp will allocate all of the space for a file on the target before copying any of the file data.  This ensures that all of the data blocks used for the file on the target are allocated as sequentially as possible.  It is also useful if the target computer is low on disk space so that the operation does not leave any partially copied files should the target disk become full.
• -w, --tunnel-transport  By default wcp will use WANFast's proprietary UDP-based file transfer protocol.  This option causes wcp to transfer files over an acceleration tunnel, instead.  This is useful if you are copying files over a high-speed LAN environment, as the TCP-based tunnel will be faster than UDP.  It is also useful when copying files over a WAN if the target computer sits within a firewalled environment that does not permit incoming UDP packets.
• --cross-mount  By default wcp will not cross a mount point when recursing through a file system.  When this option is set, it will cross the mount point and continue operation within the mounted file system.
• --vss On Windows a VSS snapshot is typically referenced using a symbolic link.  If the source of a wcp copy is a VSS snapshot, you will need to add the --vss option so that wcp will follow the symlink to the snapshot.

(back to top)

Selecting Files to Copy

The source paths that you specify for an operation can use Linux shell globbing to select more than one file or directory.  For example, specifying the source path "/myDirectory/*.txt" will select all files in /myDirectory that end with the ".txt" extension.   Please note that on Windows, file globbing can only be used in the final path component.  Thus on Linux, specifying the path "/myDirectory/*/myFile" will include all files named "myFile" in all subdirectories of "myDirectory".  But it will fail on Windows.

Wcp also permits the use of regex compatible pattern matching to select or exclude files from a copy operation.   Use the options:

• -X pattern, --exclude=pattern to specify patterns that exclude files from the operation.
• -I pattern, --include=pattern to specify patterns that include files in the operation.
• -P path, --pattern-file=path to specify the path of a file containing patterns to apply to the operation.  Each line in the pattern file is considered a single pattern, unless the line begins with '#', in which case the line is skipped.  Each pattern line should begin with the single character 'E' for an exclude pattern or 'I' for an include pattern.  There should be a single space between the pattern type and the actual pattern.

The patterns are processed sequentially in the same order as they are received in the options or pattern file.  The first pattern that matches a path determines whether the path will be included or excluded from the operation.  Note that if a directory matches an exclude rule, then the directory and all of its children will be excluded.  Thus the patterns file below will copy all of source files (files ending in ".c" or ".h") to the target.

# A patterns file to copy all of source files to the target
I .*/$
I .*\.[ch]$
E .*

The first line includes all directories. When wcp internally evaluates a path, directories always end with "/", except for any directory that you provide from the command line, so you should be sure to end any directory path with "/".  The second line includes all files ending in ".c" or ".h".  The third line rejects all other paths.  A side effect of this command is that in addition to the source files that we want, all directories will be created on the target, even if they are empty. This set of patterns would be used with a command such as:

wcp -rV -P PatternFile MySoureDirectory/ MyBackupServer:/MyBackupDirectory

Note that for this set of patterns to work, you must specify the --recurse (or -r) option and include a trailing "/" on all directories specified in the SourcePath list. 

There are many web sites that explain the syntax of regex pattern matching, including http://www.rexegg.com/ .  As it can be easy to make mistakes with regex syntax, you might want to run your commands first with the --test-drive (or -t) option to be sure of what wcp will do.  

Note that on systems running with non-ASCII character sets, the regex expressions and content of any patterns files must be UTF-8 encoded.

Options to Control Regex Matching

The pattern file format supports several options that can control the behavior of regex pattern matching.  Options are specified as single characters immediately after the I or E.  Available options include:

• c, force case sensitive matching.  By default Linux matching is case sensitive and Windows is case insensitive.
• C, force case insensitive matcching.
• w, do not ignore white space (default).
• W, ignore white space.
• a, do not force pattern anchoring (default).
• A, force pattern anchoring

Thus the pattern file entry:

ECW .*program files/$

would match any of the following:

• c:/PROGRAM FILES/
• e:ProgramFiles/
• c:/Users/test/Program files/

(back to top)

File Backup Mode

When backing up files to a remote computer, it is essential to retain all of the file's attributes along with the data.  These attributes might include the file's owner, access permissions, access control list (ACL), creation and modification time, etc.  The specific attributes vary depending upon the type of operating system and file system, but maintaining them is critical to ensuring the safety of your data.

When used with the --archive (or -a) option, wcp will fully preserve all file attributes when copying files, provided that the accounts used on the source and target server are sufficiently privileged to read and set the attributes, respectively.  When you copy a file between different operating systems, such as Windows to Linux, the target system might not understand or be able to enforce all of the original file attributes.  But it will archive them so that they will be restored when the file is copied back to its original operation system.  When the target system cannot understand the source systems security attributes, it sets its own attributes to be as restrictive as possible so as not to compromise the security of your data.  Thus a Linux file server can be used to backup and restore a Windows computer (or vice versa), without loss of vital security settings.

By default, archived attributes are stored on the target in a file with the same path as the copied file/directory with the ".wcpA" extension added to the end.  You can specify an alternate extension, if necessary, as described in Advanced Options.

Some additional options that are useful in file backup mode include:

• --save-attrs causes wcp to always create an attribute archive file, even when the source and target are running the same OS.  This is useful when the account on the target system is not sufficiently privileged to set all of the attributes, allowing your users to backup files to a computer on which they have limited access.  It is also useful when the two computers do not share the same user accounts, allowing the target to retain the original file ownership settings.
• --save-attrs-src tells wcp to use the archived attributes on the source computer, but to not archive the attributes on the target.  This is useful when restoring a backup that you created using --save-attrs option above.
• -A, --absolute-paths The files created on the target computer will use the full absolute path of the original file on the source computer.   For example, the command: "wcp -arAv /Users RemoteHost:/backups/5-31-2017" will back up the entire Users file system from the local computer into the /backups/4-31-2017 subdirectory on the remote computer.

wcp handles Windows file attributes in a transportable manner.  Windows account names and groups are exported along with the domain in which they are defined.  The target system first attempts to locate a matching domain/name.  If not found, the domain is dropped and Windows attempts to find a match for the name among all other defined domains.  Thus account names transfer correctly if the two computers belong to the same domain.  They will also transfer correctly even without a common domain as long as the same accounts are defined on the two computers.  See Changing File System Types -- Support for File System Metadata -- Mapping User IDs for details on how to map specific IDs between the source and target machine.

(back to top)

Block Level File Synchronization Mode

In synchronization mode, wcp uses the well-known rsync algorithm to rapidly determine which blocks within a file are different between source and target, and only send the differences over the wire.   You turn on sync mode with the --sync (or -s) option.  When syncing, you should also specify either the --archive (or -a) or --times-only (or -T) option so that the target file receives the same last modification date as the source file.  This will greatly speed up subsequent sync operations.  The first option copies all file attributes from the source to the target.  The second only copies the creation and modification times.

By default wcp will copy the files that exist on the source but not the target to the target.  It will synchronize the files that exist on both the source and target to ensure that the target matches the source.  And it will leave untouched any files/directories on the target that do not exist on the source.  By using the --delete-files (or -d) option in synchronization mode, you can optionally delete files on the target system that do not exist on the source to make the target file system identical to the source.  Any exclude/include patterns that you have specified will be used to filter the paths that get deleted -- i.e., to be deleted a file on the target must not exist on the source and must not be excluded by a pattern.   Warning!  It can be confusing if you use patterns with a sync operation with the delete operation because the meaning of the patterns is slightly different for sending a file versus deleting a file.  If you need to use patterns for a delete, we recommend that you first perform a sync operation without delete to ensure that the target matches the source except for its extra files/directories.  Then run a sync with delete and your patterns to delete the selected files on the target.  We also recommend using the --test-drive (or -t) option with a delete prior to actually running the command.

By default wcp will skip any file that has the same size and last modification date on both the source and target.  You can override this and force a block-by-block file comparison using the --always-hash (or -H) option.

(back to top)

Copying and Syncing Block Devices

To wcp, a block device looks like any other file. This means that you can copy a block device from one machine directly to a block device on another machine.  Copying the entire device will copy whatever file system(s) are on the block device.  All file data and attributes will be preserved exactly. When the file system that you wish to copy contains nothing but many small files, this can be a quicker way to make the copy.  But note that when copying a volume, wcp cannot differentiate between file data and unused disk.  So if the disk is mostly empty, you will be copying a lot of extra data.  Once you have copied a block device, you can also perform a block level sync between the source and the target.

You can also use this capability to create a volume backup -- or disk image -- copying a block device on the source to a file on the target.  You can reverse this at any time, copying the file to a block device, whether restoring the original or creating a new disk.

When copying a block image, wcp can only handle a single source item on the command line, whether a block device or file.  You must also specify the --vol-copy command option to inform wcp that you want to copy the contents of the device rather that recreate the device file system entry on the target.  If you need to specify more than one volume copy, use the --op-list parameter to specify an operation list file.

On linux:

wcp -KVw --vol-copy /dev/sda2 User@TargetIP:/dev/sdc2

copies device /dev/sda2 on the source to /dev/sdc2 on the target, while

wcp -KVw --vol-copy /dev/sda2 User@TargetIP:/tmp/ImageFile

copies the same device to a file on the target.

On Windows the device name path looks like:

• //./H:/    -- the volume mounted as H:/
• //?/Volume{f68c410b-fc6d-11e7-80d3-c4af03b2f494}/  -- A volume referenced using its UUID.

You can display the available UUIDs for your volumes using the mountvol.exe command in Windows.  Note that you will need to change all backslashes '\' to forward slashes '/' to accommodate wcp.  See https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file for details on Windows device name spaces.

Once your block device has been copied, you can do a block level synchronization of the two devices using the same commands as above, but substituting the -s option for the -K.  For the linux example above, that would look like:

wcp -sVw --vol-copy /dev/sda2 User@TargetIP:/dev/sdc2

Note that because wcp sees the block device as a single file, when performing the sync operation it will read entire contents of the block device on both machines, comparing the contents on a block-by-block basis.   Only the blocks that have changed will be copied over the network.  

Different Volume Size

When copying a block device, the target volume size must be at least as large as the source, even if the source is not fully used.  If it isn't, the copy operation will fail with ERR R 368, No space left on device.  If the target volume is larger than the source, not all of the target will be used.  The file systems copied to the target will be unaware of the additional space and appear to be the same size as the source.  Most file system types supported by Windows and Linux allow you to resize the file system to take advantage of the additional space.  But this will require commands outside of wcp.  Note that if you wish to reduce the size of a file system on the target, you will need to use the a file copy operation.

Copying a Volume with Contained Partitions

Both Windows and Linux have separate path names for a storage volume and any partitions of that volume.  That makes it possible to copy the entire volume, with all contained partitions, in one operation.  For example, on Linux a volume might use the device path /dev/vdb and have partitions /dev/vdb1, vdb2, and vdb3.  Running the operation:

 wcp -KVw --vol-copy /dev/vdb User@TargetIP:/dev/vdb

will copy the entire volume, along with its partition table and all partitions.  But note that the operating system will probably not immediately recognized the new partitions as these are typically scanned as part of system boot.   In order to access the partitions you will first need to either reboot, or issue the operating system dependent command to rescan the disks.  On Linux this would be:

partprobe /dev/vdb

(back to top)

Copying Slow Disks and Network File Systems

By default wcp copies file data in blocks of approximately 16KB.  When working with slower file systems, such as those on older mechanical disk drives or mounted network file systems, copy and sync times can often be greatly improved by reading/writing in much larger blocks.  For disk drives, this minimizes seek time. For network file systems, it minimizes the number of commands (round trip operations) needed to access the data.  wcp has an internal file buffering capability that reads/writes data in 1 MB blocks and employs a dedicated I/O thread so the main process is not slowed waiting for the disk.  By default this is only turned on when copying raw block devices, which are not  buffered by the operating system.  

When working with a slower file system, you can turn on wcp's internal disk buffering and take advantage of more efficient I/O by adding the following lines to your wfst.cfg file.

command-opt = wcp:--rd-buffer
command-opt = wcp:--wr-buffer

If the slow device is on the client end of your connection, add this to the wfst.cfg file in your home WANFast directory.  See Create or Import Your Key Pair for help finding your home directory.  If the slow device is on the server end of your connection, add this to the wfst.cfg file in the global configuration directory.   On Linux this will be /etc/wfst.  On Windows this will be C:/ProgramData/WANFast.

 

Special Notes for Accessing Windows Shares

The wcp application does not automatically authenticate to a Windows share.  This means that if you try to access files on a share without taking the steps below, you will receive an access error.  In order to successfully access the share:

If the Share is on the Client

• Bring up a command prompt window
• Mount the remote share using the command: net use X: \\ShareHost\share, where X is an used drive on which to mount the share, ShareHost is the name or IP address of the host and share is the share name.  The command will likely ask you for a valid username and password for accessing the share. 
• From this same command prompt windows, run your wcp command, such as:

wcp -rKVw //ShareHost/share/path/FileToCopy User@TargetHost:C:/Path

Note that /ShareHost/share must match what you used above when mounting the share.

If the Share is on the Server

• The wfst server process does not likely have access to the network share. Nor can you start it directly from the command line as we did the client program above.  Thus, you must first stop the WANFast service so you can reuse the network port.
• Log into the server machine using the Account ID that you would normally use to copy files.  This account must have Administrative privileges.
• Bring up a command prompt window, using the right mouse button to start it with Administrative privileges. 
• Make sure the wfstd service is stopped.
• Mount the share as described above.
• If adding the file buffering options described in the previous section, do so to the personal wfst.cfg file of the account to which you've logged in.
• Run the following command:

wfst --target-mode

This will start the WANFast client in a special "perform like the server" mode.  Like the server, it will listen for incoming client connections and service any client requests.  However, because it was not started as a Windows service, it cannot change the user ID under which it runs.  This means that the client must specify this account when connecting to the client -- e.g., if you logged into the server as user "JoeTest" to start wfst, then on the client machine you must run something like:

 wcp -rKVw JoeTest@ServerHost://ShareHost/share/path/FileToCopy c:/tmp

(back to top)

Controlling Encryption and Compression and Other Base WANFast Options

By default encryption is turned on for all data transfers and compression is turned off.  You can change this behavior with the following options:

• -E or --no-encrypt turns off encryption.
• -n alg or --encrypt-alg=alg specifies the encryption algorithm that is used for WANFast tunnels and wcp control traffic.  By default this is set to hc-128, which is the fastest 128-bit stream cipher  supported by WANFast that has significant international backing.
• -b alg or --block-encrypt-alg=alg specifies the algorithm to use for encrypting data with WANFast's UDP file transfer.  Because this transfer mechanism processes data in random order on the target side, it must use a separate encryption context and initialization vector for each data segment.  By default this algorithm is set to sosemanuk-256, another fast 128-bit stream cipher with significant international backing.  It has very fast initialization, making it the fastest algorithm for use with the UDP transport.
• -q e or --query e prints the list of supported encryption algorithms.
• -c or --compress turns on data compression.  This speeds up the transfer of data over slow WAN connections, but consumes significant additional CPU time on the sending machine.  On faster WAN connections (> 400Mbps) the use of compression will actually slow down your data transfer due to CPU constraints.  The exact WAN speed at which compression becomes beneficial depends upon the ratio of CPU power / WAN speed as well as the compressibility of your data.  If data transfers are slow, try compressing.
• -z alg or --compress-alg=alg specifies the compression level to use when compressing. It can take the values lz4, zlib-1, zlib-2 through zlib-9.  By default wftp compresses your data on the fly using lz4 compression while sending. Lz4 is currently the fastest compression algorithm, but provides the least amount of compression.  Overall it is a good balance for transfer rates up to about 400Mbps.  If your WAN transfer rate is faster than this, then you should turn off compression as it will only slow you down.  If your transfer rates are significantly less than this, perhaps in the 50Mbps range or lower, you should consider trying one of the zlib compression levels.  These are slower, but provide significantly improved compression.  In slower network environments they can dramatically increase your overall throughput.  The slower your transfer speed, the more you can potentially gain.  The zlib compression values are numeric from 1 to 9.  Level 1 provides the least compression, but runs the fastest.  Level 9 provides the most compression, but is the slowest.  Level 5 is a decent default value. 
• -k or --auto-compress turns on WANFast's adaptive compression algorithm, which continuously monitors your data stream, compressing data that can be compressed and ignoring data that is already compressed to best optimize your network bandwidth and CPU utilization.

(back to top)

Copying Windows Encrypted Files/Directories

The Windows Encrypted File System (EFS) allows you to encrypt individual files and directories while they reside on disk.  It also allows you to set a flag on directories so that any file created in them will be encrypted. Files are automatically decrypted for you when you read them from disk. The keys and certificates used for this process are managed using the Windows Certificate Manager (run certmgr.msc) or the command line program cipher.

Wcp and wftp will copy your encrypted files for you, keeping them safe in transit.   There are two separate methods for copying encrypted files, designed for different use cases.  Make sure you are using the correct method or you might not be able to read your files after they have been transferred.

Method 1, retaining the original encryption keys.  If you run the wcp or wftp client with elevated privileges from an account that has the Backup privilege, the command will use Microsoft's special encrypted file backup API to transfer the files in their encrypted state.  This has the advantage that you will be able to copy all encrypted files, not just your own.  But you will be unable to access your files once copied unless you also transfer your keys to the target machine.  This can be done in two ways:

1. If the encrypted files are being copied as part of a full system migration, then your keys are also being moved and you are all set.
2. Otherwise you will need to export your keys from the source machine and import them on the target.  On the source, use the certificate manager or cipher utility to export your keys into a PFX file (e.g. cipher /X FileName).  Do this before you copy the files so your keys are copied with them.  Be sure to set a secure password on the PFX file.  Once on the target, find the PFX file in Windows explorer and double-click on the file.  This will import them.

Method 2, rekeying the file.  If you run the wcp or wftp client without the Backup privilege, then the command will decrypt the file using the encryption key of your login account.  You will only be able to transfer files that are encrypted with your key.  You will get an access error for all other encrypted files.  WANFast will encrypt the data while it is in transit to the target machine.  On the target machine Windows will re-encrypt the data using the  encryption key of the account that you logged into on the target. 

So if you log into the source as "Joe", and run the command "wcp -arKVw EncryptedFile Sue@TargetMachine":

• the transfer will only work if "Joe" has the necessary key to decrypt the file.  Otherwise you will get an access error.
• on the target the file will be encrypted using Sue's key.

(back to top)

Advanced Options

There are a few other options available for use with wcp, but they are used less often.

• -x MBPS, --max-bandwidth=MBPS sets the maximum bandwidth to be consumed by a WANFast session, should you wish to limit WANFast's transfer rates in order to provide more network bandwidth for other applications.
• -p N, --port=N sets the UDP port number used on your computer for transferring file data.  N should be the integer port number.  By default wcp will use any unused port within the UDP port range that you specified during installation.
• --save-attrs-tgt tells wcp to ignore any archived attributes on the source and use the native source attributes instead, archiving these attributes on the target.  This is similar to the --save-attrs (or -C) option, except that it ignores any archived attributes on the source.
• -F, --no-overwrite is only used on the wfstd server, telling it to prevent the overwrite of any files by wcp, even if run with the --overwrite options.  See below for details on how to set wcp options on the server.
• --block-size=N sets the block size used by wcp for transferring data.  By default wcp sizes its transfer blocks to fit within a standard, 1518 byte, ethernet frame to avoid packet fragmentation over the WAN.  But if your network uses a lot of IP options, it is possible that WANFast's default packet size is too large and becomes fragmented, causing slower transfer rates.  You can then use this option to specify a smaller packet size, such as 1432. 
• --archive-extension=str specifies the file extension used by wcp when archiving file attributes.  By default this is ".wcpA".  If that conflicts with existing files on your computer, then you can set this to some other, unique value.
• -J, --ignore-unk-ftype  Tells wcp to ignore unsupported file types rather than print errors.
• --no-attr-errs  Do not print error messages if unable to set all file attributes on the target.
• --delete-first  When using the --delete option in sync mode, this will cause wcp to make two passes through the file system.  On the first pass it will delete any files that need to be deleted to free up disk space prior to adding new files in the 2nd pass. Use this option if you are low on disk space on the target.
• --log-changes  When set, wcp will write a line to its log file on the target machine for each change that is made to the target's file systems.
• -N, --numeric-ids  If the source machine has user IDs that are not recognized on the target, you can use this option along with the --archive option to preserve file attributes in their numeric format.  On Linux this would preserve the numeric user and group IDs even though they are not configured on the target.  On Windows this would preserve the user, group, or account SIDs even though the are not configured on the target.  In either case you can use the tools native to the target to either create the necessary accounts or map the numeric IDs to some other value supported on the target.
• --no-sparse  By default wcp will attempt to preserve a sparse file intact when copying.  If this option is specified, the holes in the sparse file will be transferred as blocks of zeroes.
• --check-attrs  When specified with a sync operation with --archive set, this causes wcp to update the attributes on files and directories on the target even if wcp does not modify them.  This is useful on Windows, which does not update the file modification time when you change file attributes.  Thus you would need to specify this option to pick up changes in file attributes (such as security settings) on otherwise unmodified files.
• --quick-calc  By default wcp will first estimate the size of the data to be transferred so it can give you an estimate of the size and time remaining.  When you are transferring an entire file system, calculating this estimate can take some time.  Using the --quick-calc option tells wcp to estimate the transfer size from the consumed space on the volume.  The estimate will typically run slightly high, but can speed up the operation.

See wcp Client Command Syntax and Configuration Options for a list of all command options.

(back to top)

Setting wcp Options on the Server

The wfstd daemon or service provides the server-side functionality for all WANFast applications including wcp.   To specify a wcp option for the server, you must edit the WANFast master configuration file, see wfstd Daemon Command Syntax and Options.  The following wcp options can be added to your server configuration file: 

• copy = false                    Prevent file copies/syncs in/out of the server
• overwrite = false             Prevent overwriting of any existing files on the server. Overrides client request.
• max-bandwidth = int      Set the maximum bandwidth used for a WANFast session. 
• encrypt-alg = alg             Specify the stream encryption algorithm.  If different than the client-specified algorithm, the stronger algorithm is used. 
• block-encrypt-alg = alg  Specify the block encryption algorithm. If different than the client-specified algorithm, the stronger algorithm is used.  
• min-udp-port = int          Start of UDP port range. 
• max-udp-port = int         End of UDP port range.

(back to top)

 Auditing

The use of wcp is audited on both the client and server as a command execution event.  Audit records are not written for each file that is copied using wcp.  If you wish to record this information on the server, add the following line to your server configuration file and a record of each copied or modified file will be written to the server log file.

command-opt = wcp: -V

(back to top)

Integration with Other Applications

The wcp application includes features to make it easier to integrate with other applications.  You turn these features on through a combination of the options -Q -y --rm=Path.  The -Q option quiets the output to stdout such that it only includes file transfer errors. The -y option bypasses the client check for unrecognized remote hosts, supply a "yes" answer.  This is useful when running wcp from a script. The --rm option specifies the path to a status file that wcp will update at roughly 15 second intervals.  The status file will include a sequence of operation reports, with each report formatted as:

Volume: VolumeID, Status N
Size: N, Scanned: N, Copied: N, PP.P% complete, BW N, TM N, ETA N
Copied: N directories, N files, N symlinks, N hard links
Skipped: N directories, N files, N symlinks, N hard links
Errors: N directories, N files, N symlinks, N hard links, N unknown, N setattr
Deleted: N directories, N files, N dir errors, N file errors

Where:

• VolumeID is the path of the target specified for the operation 
• Status is the exit code for that one operation. 
• N is in integer. 
• In the first line, Size, Scanned, and Copied are in bytes.
• PP.P is the percentage complete
• BW is the average rate that the operation has progressed in bits per second 
• TM is the elapsed time in seconds that the operation has been running, or took to complete.
• ETA is the estimated time to completion for the operation.

The first entry in the status file has a VolumeID = "SUMMARY".  It provides a summary of the entire session.  It is then followed by entries for each of the specified operations.

(back to top)

Changing File System Types -- Support for File System Metadata -- Mapping User IDs

To wcp all file systems look like a simple directory tree.  Thus, it will happily copy files from one file system type on the source to a different file system type on the target, and will even copy files between Windows and Linux.

Please note, however, that different file system types support different forms of metadata, including security attributes.  On Linux and all Linux file systems, wcp supports the standard Linux attributes:  owner, group, mode, and times.  On Windows wcp supports all of the NTFS metadata, including ACLs.  Not all Windows file systems support the full set of NTFS metadata.  When wcp is unable to copy the full set of metadata from source to target, it will print (and log) a warning.  You can disable the warnings using the --no-attr-errs option. 

Mapping User IDs

One of the common issues that will cause attribute warnings is the presence of users or groups on the source machine that are not present on the target machine.  These are handled differently depending upon the options that you use.

If you do not specify the archive option, -a, then file metadata is not copied and all files and directories created on the target will be owned by the user identity being used on the target for the operation.

If you specify -a and -N, the IDs will be copied and applied using their numeric values.  No warnings will be issued, but the resulting file ownership might not be what you desire.  You can then convert the unrecognized numeric IDs on the target into values for an account existing on the target.

If you specify -a without -N, IDs will be mapped from the target to the source by converting the numeric values to/from their human readable values.  For example, if the user, john, has a user ID of 1001 on the source machine and a user ID of 1111 on the target machine, the wcp will map the value from 1001 --> 1111 while transferring the file metadata.  On Windows, wcp will first look for a matching account name using the original Windows domain from the source.  Thus account names will transfer correctly if the two computers belong to the same domain.  They will also transfer correctly without a common domain as long as the same accounts are defined on the two machines.

The ID Map File

Wcp also allows you to map specific accounts between the source and target machines using an ID Map file, id.map.  If you wish to map accounts, this file must exist on the machine that is receiving the files.  On the server -- i.e., when copying files from a client to the server -- the file must be placed in the /etc/wfst on Linux or C:/ProgramData/WANFast on Windows.  On the client -- i.e., when copying files from the server to the client -- the file must be placed in your user directory, ~/.wfst on Linux or C:/Users/YourAccount/AppData/Local/WANFast on Windows.

Each line of the id.map file should contain a mapping of the form:

Type RemoteName,  LocalName

On Windows, the value for "Type" should always be "S", for SID.  On Linux "Type" can either be "U" for a user name, or "G" for a group name.

There must be a comma after the remote name.

The mappings will be searched in order.

You can specify the remote name "*" to indicate a wild card that matches all unrecognized remote account names.  This entry should always go last.  Thus the file:

S TestUser, John
S System Installer, Installer
S *, Administrator

Will map the Windows account TestUser to John on the target machine.  It will also map "System Installer" to Installer on the target machine.  All other unrecognized accounts will be mapped to Administrator for safety.  Note that in order for account mapping to work, the account used on the target machine must be sufficiently privileged to modify file ACL entries.

Whenever an account is mapped, a line similar to the following will be printed to the log file: MAPPED ACCOUNT (S) remote TestUser --> John.  If you do not know which accounts are defined no the source and not the target, you can start with a single default mapping.  After copying the files, search for all MAPPING lines in the log file and set up the desired mappings for those accounts in the id.map file.

(back to top)

Exit Codes

The value returned by wcp is a combination of an exit code, which gives an overall success/failure indication along with a reason for any failure, and a set of exit flags that provide an indication of non-fatal events that occurred during operation.

The exit codes occupy the 4 least significant bits:

WEXIT_OK             0       The Operation ran to completion. 
WEXIT_ERROR          1       The Operation terminated for unknown reason. 
WEXIT_INV_PARM       2       An Invalid parameter was specified on the command line
WEXIT_INV_CONFIG     3       There is an invalid configuration setting 
WEXIT_INV_LICENSE    4       Invalid or expired license on client or server 
WEXIT_LIC_RESTRICT   5       The Operation not allowed due to a license restriction. 
WEXIT_CONN_FAILED    6       Unable to connect to peer. 
WEXIT_CONN_DROPPED   7       Connection dropped after being established. 
WEXIT_USER_CANCELED  8       The user canceled the operation. 
WEXIT_VERSION        9       Incompatible versions between client and server 
WEXIT_INTERNAL       10      Internal, coding error. 
WEXIT_AUTH           11      User authentication issue. 
WEXIT_DISK_FULL      12      Terminated because target disk was full. 

The exit flags include the following bits.  They can be set in addition to the exit codes above.  More than one flag can be present.

WEXIT_FILE_ERROR     0x10    At least one file transfer failed. 
WEXIT_DELETE_ERR     0x20    At least one file on target not deleted. 
WEXIT_ATTR_WARN      0x40    At least one file not set with full attributes 
WEXIT_VOLUME_FULL    0x80    Disk full on at least one volume in multi-volume 

 (back to top)

Examples

Basic operation:

wcp -raV File1 File2 /Users/MyHomeDir /opt/MyDataDir Administrator@BackupServer:d:/backups/5-31-2017

copies two files from the current working directory and two directories to the target Windows server, placing the copied files in the d:/backups/5-31-2017 directory.  If they do not exist, d:/backups and d:/backups/5-31-2017 will be created.  All file attributes will be retained,  We will recurse into the two directories and copy their children as well.  The results on the target computer will include:

d:/backups/5-31-2017/File1
d:/backups/5-31-2017/File2
d:/backups/5-31-2017/MyHomeDir/    and children
d:/backups/5-31-2017/MyDataDir/    and children

If we had included the -A option to use absolute paths, the results on the target computer would have included:

d:/backups/5-31-2017/File1
d:/backups/5-31-2017/File2
d:/backups/5-31-2017/Users/MyHomeDir/  and children
d:/backups/5-31-2017/opt/MyDataDir/    and children

If we had used the -A option above, then the command:

wcp -rsV Administrator@BackupServer:d:/backups/5-31-2017/* /

would restore our local computer by syncing the files back to their state when we made the backup.  It would also leave copies of File1 and File2 in the root directory, which is probably not what we want.  Best not to mix absolute and relative path names in one backup.

(back to top)

See How to Accelerate Your Applications with NO Changes to Your Code

Running Remote Commands or a Remote Shell

WANFast Security