The Wonders of 7-Zip

This past week, I've spent a lot of time going over our nightly job streams and the various relationships between jobs across our many servers. These jobs handle things like code promotion, database syncs (dev/qa/prod), backups, mirrors, disaster recovery, etc. etc. Things have changed a lot since I initially put a lot of the schedule together -- it was time to shuffle and revisit some of the dependencies.

As part of this effort, I wanted to determine how long some of the tasks were taking to run. I was a bit blown away when I realized it was taking almost 3 hours to compress and encrypt our nightly database backups!

That zip is the result of a simple little batch file. Here's the pertinent bits (no, you may not ask about "magicKey"):

I run that in the appropriate directory and it spins through all the .bak files and compresses and encrypts each.

The "-tzip" parameter tells it to save the archive in a zip compatible form, the "-mx=9" compresses it at the highest level -- after they're zipped, I mirror them across a slow VPN link for redundancy, so small is good.

Last night I actually watched this run and made a horrible discovery: It only was using one CPU! Doh. No wonder it was taking so long. The server running this is a dual Hyper-Threaded Xeon, surely I could do some tweaking.

Some quick research turned up the "-mmt=on" clause which turns on multi threaded processing. In experimenting a bit this morning, however, I still wasn't getting multiple CPU usage. Once I dropped the "-tzip" and went 7z native... THEN I got more CPUs in the mix.

That test run, during business hours (and higher server load) ran just over an hour. So probably not quite a 2 hour savings. But wait, there's more!: The compressed archive size used to be over 700MB. Now it's around 380MB. That's a dramatic improvement for going from zip to 7z format.

Result after an hour or so of dinking around: Almost halved the archive sizes (as well as data to send over the VPN) and more than halved the processing time to make the archive. I call that a win.

Possibly Related posts:

1. 7-Zip: Compression vs. Speed
2. More Power
3. Encrypting SQL Server 2005 Backups
4. Using StumbleUpon Firefox Extension with Navigator 9 beta
5. More Dabbling with File Compression Types