I recently ran into an problem where I needed a remote directory to exist before rsyncing data over to it. Rsync will only create the remote directory for one level, meaning that the parent path must exist:
Works:
rsync file user@remote:/tmp/
Doesn’t work:
rsync file user@remote:/tmp/imaginary/
There’s a few StackOverflow questions about this (OK, the last one is from SuperUser), but none of them solve the problem above. The man page for rsync has the answer tucked away under the —-rsync-path parameter:
Use this to specify what program is to be run on the remote machine to start-up rsync. Often used when rsync is not in the default remote-shell’s path (e.g. –rsync-path=/usr/local/bin/rsync). Note that PROGRAM is run with the help of a shell, so it can be any program, script, or command sequence you’d care to run, so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate.
We can use this knowledge and the example in the man page to make rsync do exactly what we want:
rsync -aq –rsync-path=”mkdir -p /tmp/imaginary/ && rsync” file user@remote:/tmp/imaginary/
This technique is much more efficient than fork-execing an SSH process to run “mkdir -p” first. To test, I compared both versions (rsync only vs ssh, then rsync) 100 times in a for loop. It’s not the most scientific test in the world, but I think it represents some real-world usage:
$ time ./rsync_test.sh
real 1m38.706s
user 0m6.639s
sys 0m1.278s$ time ./ssh-and-rsync.sh
real 2m12.779s
user 0m13.033s
sys 0m2.103s
34 second wall-time decrease, and near half-time decrease in user and sys! Cheers to rsync for making this a feature and for the StackOverflow questions making me refusing to believe the truth.
Don’t mean to nit-pick, and I appreciate your post, but when posting code or command-line examples your quotes should really use fixed-width fonts and disable any “special” characters like long-hyphens or left/right-specific quote marks. This makes it much clearer overall and makes copying/pasting easy as well.
Hi yes Vahab, the module uses a different method of connecting, so this won’t be helpful in that case.
Unfortunately it does not work when I use ‘::’ syntax and specify a module name
From help :
The ‘:’ usages connect via remote shell, while ‘::’ & ‘rsync://’ usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
[ The modules are defined in /etc/rsyncd.conf ]
In this case, –rsync-path does nothing. No errors or anything in the logs, it just gets ignored.
Brilliant!
Super-useful!
This doesn’t seem to be working on newer protocols of rsync? I am more interested in rsync version 3.0.6 protocol version 30
Error which I get is similar to following:
invalid characters in scp command!
here:&& /usr/bin/rsync
If I change “&&” to “;” , error changes accordingly.
I found the
ssh user@remote mkdir -p /tmp/imaginary/
rsync -aq file user@remote:/tmp/imaginary/
is best.
You’re welcome! I was very excited once I finally figured it out! It depends on your use case — a mkdir -p usually isn’t going to explode. For my use, any scenario that was going to make a mkdir fail would have been caught by another process, so the output wasn’t interesting to me.
Finally the solution that works for me! Stackoverflow is full of –relative solutions, that do not cover my scenario.
How about stdin/stdout of the mkdir, should I redirect it?
Glad you liked it 🙂
Real sneaky! Thanks very much