Shuffling and Syncing

Posted on December 2, 2005 3 Comments

One part of my job, when wearing my “Operations guy” hat for my employer, is apparently being the lucky fella who has to keep the various development and production environments for our products in sync. Which is fine as I enjoy the variety and challenge. And, over the last year I’ve learned some nice tricks related to scheduling and “chaining” jobs across multiple servers. So today I thought I’d share a bit about xcopy, robocopy and SyncToy.

A core piece of most of our projects is populating PDFs with data gathered in an interview of some sort. As such, our forms team does a lot of work, daily, in our development environments and then I run nightly jobs to copy the assorted product-related directorys to the production servers to keep ‘em in sync. Traditionally, I’ve done that with good ol’ xcopy and an “exclude” file to be sure I wasn’t copying some stuff over to the server. So, as an example, the xcopy command might look like:

xcopy "W:\projectPDF" "x:\docs\projectPDF" /EXCLUDE:c:\util\pdfexclude.txt /D /S /Y /Z

and that pdfexclude.txt file mentioned looks something like:

\notes\ \samples\ .zip

Pretty straightforward stuff, right? Basically, that’s saying copy everything from w:\projectPDF to x:\docs\ProjectPDF. exclude the directories or file extensions listed in pdfexclude.txt.
/D tells it to compare dates (only copy when the source is newer than the dest).
/S tells it to traverse subdirectories.
/Y gets rid of the “really copy this file?” prompts.
/Z is used to copy networked files in a restartable mode. Frankly, I’m still fuzzy on what exactly it does… but it seems like a good idea based on the description.
When in doubt, just type xcopy /? to get the basic list of switches or options.

Now this all works well enough but I ran into one major snag — as files are renamed on the source side, that new name is copied to the destination. However, the old named file lives on in the dest. Same with directories. And if a file is deleted in source, it is not deleted at the destination. So, over time the destination builds up with some orphans — files and directories that don’t exist in the source any more, but live on in destination.

To work around that I started using microsoft’s SyncToy that was released a few months ago. I guess a good way of describing it would be to say it is like xcopy, but with more options and a very easy to use graphical interface (did I mention xcopy is command line only?). SyncToy works in a similar fashion though. You define a folder “pair” (or specify a source directory and target directory). Then you tell SyncToy what to do with those two directories.

Syncronize - New and updated files are copied both ways. Renames and deletes on either side are repeated on the other.
Echo - New and updated files are copied from left to right. Renames and deletes on the left are repeated on the right.
Subscribe - Updated files on the right are copied to the left if the filename already exists on the left.
Contribute - New and updated files are copied left to right. Renames on left are repeated on right. No deletions.
Combine - New and updated files are copied both ways. Nothing happens with renamed or deleted files.

For my purposes, Echo works quite nicely. However… While SyncToy does let you specify some options related to the copying, there is one critical option that I needed for some directories. You can set an INclude filter (a list of extensions or files), but you can’t specify an EXclude list. Well, in some of my projects that’s a deal breaker. But in general this is a slick tool and can even be run as a scheduled event easily enough. It has a command line argument to run a specific job which is handy. Overall I’d say this is a great tool for folks who want to do quick and dirty backups or file moving and are intimidated by a command line.

As I was searching around to see if there was a way to do excludes with SyncToy, I saw multiple comparisons of it to Robocopy. So, I did a quick search on robocopy to see what the heck it is. Turns out, it is part of assorted Microsoft Windows Server Resource Kits (that links to the 2k3 flavor which has the newest version of robocopy). Now, this is another command line tool with roots to good ol’ xcopy. But it can do SO much more. For instance, it has a /MIR flag. Yup, mirror. As in, “I’ll keep these directories completely in sync. Including deleting any extras over there on the destination side due to renames or deleting on the source side.” Slick! The documentation is a bit overwhelming at first, but there’s one flag that’s definitely a life saver: /L. This guy tells robopy to tell you what it would do with the other information, but to not actually do it. just List what the results would’ve been. Very nice as you define a new job. I’m enamored! And a bit bummed that I had never found this thing before!

I’m still getting a handle on the syntax and flags, but right now I have a command that looks like:

robocopy "X:\projectPDF" "\\myserver\myshare\projectPDF" /XD attachments signatures /XF *.zip Thumbs.db /mir /tee /log:c:\util\pdfs.txt /Z

/XD is a list of directories to ignore.
/XF is a list of files to ignore.
/MIR is mirror (same as /E /PURGE)
/LOG specifies where I’d like the logfile created.
/TEE is nice when you want to log AND watch the job run on screen. It pipes output to both.
/Z copies files in “restartable mode”. Again, I’m fuzzy on what this really does, but it seemed like a good idea…

So, for the next few hours I’ll be doing a lot of replacing xcopy jobs with robocopy jobs. Just changing over one job has freed up over 1GB of “orphans” on a server.