This past week, I've spent a lot of time going over our nightly job streams and the various relationships between jobs across our many servers. These jobs handle things like code promotion, database syncs (dev/qa/prod), backups, mirrors, disaster recovery, etc. etc. Things have changed a lot since I initially put a lot of the schedule together -- it was time to shuffle and revisit some of the dependencies.
As part of this effort, I wanted to determine how long some of the tasks were taking to run. I was a bit blown away when I realized it was taking almost 3 hours to compress and encrypt our nightly database backups!
That zip is the result of a simple little batch file. Here's the pertinent bits (no, you may not ask about "magicKey"):
for %%A in (*.bak) do 7z a -tzip -mx=9 -p<magicKey> %%A.zip %%A
I run that in the appropriate directory and it spins through all the .bak files and compresses and encrypts each.
The "-tzip" parameter tells it to save the archive in a zip compatible form, the "-mx=9" compresses it at the highest level -- after they're zipped, I mirror them across a slow VPN link for redundancy, so small is good.
Last night I actually watched this run and made a horrible discovery: It only was using one CPU! Doh. No wonder it was taking so long. The server running this is a dual Hyper-Threaded Xeon, surely I could do some tweaking.
Some quick research turned up the "-mmt=on" clause which turns on multi threaded processing. In experimenting a bit this morning, however, I still wasn't getting multiple CPU usage. Once I dropped the "-tzip" and went 7z native... THEN I got more CPUs in the mix.
for %%A in (*.bak) do 7z a -mx=9 -mmt=on -p<magicKey> %%A.zip %%A
That test run, during business hours (and higher server load) ran just over an hour. So probably not quite a 2 hour savings. But wait, there's more!: The compressed archive size used to be over 700MB. Now it's around 380MB. That's a dramatic improvement for going from zip to 7z format.
Result after an hour or so of dinking around: Almost halved the archive sizes (as well as data to send over the VPN) and more than halved the processing time to make the archive. I call that a win.
Possibly Related posts: