Tag Archive for: network

Read Errors

15 Jul 2010
July 15, 2010

The final (?) note on the ESXi / HP saga (part 1 / part 2). This is too much of a downer to continue documenting!


I’ll start with a quick tip: If moving data to NFS shares seems slow or gives you frequent timeouts look to your network gear.

I was having issues getting ghettoVCB backups to an NFS share on a Windows 2003 server. The VMware ESXi server would sporadically lose connection to the 2k3 server and then kill the backup. I finally replaced the little DLink SOHO 1GB switch with an HP ProCurve and then replaced all the sketchy old network cables with shiny new CAT 6 cables. The backups became noticeably faster and the intermittent connection losses completely disappeared.

Now I can get good backups for 3 out of the 6 Virtual Machines (VMs) on this server. Using any sort of file copy I can get copies of those same 3.

The other three? I’m starting to lose faith – I think we’re hosed. The copies or backups always end with a series of errors. The log errors point to the datastore where the VMs currently reside, not the copy destination. Read errors. Ugh.

Tip: I can’t seem to get “thin” backups to the Windows (or OpenFiler for that mattter) NFS shares. So, regardless of how much data is actually used in that 150 GB virtual disk, I get a full 150 GB backup file. As a workaround, I turned on NTFS compression for the NFS share at the Windows server. Slows the copy speed down by almost half with barely any extra CPU utilization. Worth it though as it took 280 GB of backups down to 16.5 GB!

I have a VMware forum post out there languishing. It did result in me making sure I had the latest/greatest firmware, ESXi updates and HP tools installed though. It also took me down a few unnecessary paths, but that’s OK as it was educational. I’ll probably close that post soon and try a much shorter and summarized version. I may have to figure out how to contact paid support.

I also tried a ServerFault.com post but I think I tried to cover too much territory in it. Face it, many geeks suffer from tl;dr syndrom. Think I’ll close that topic soon as well.

I am trying to get some help from HP now, but this time they’re not so interested in helping. See, at boot time the P400 array controller gives an error 1716 “unrecoverable media error.” HP says, logically enough, that I need to rebuild the array. OK, I’d like to do that but I want image backups first. They say I should’ve had good backups before I did any drive replacements. Well, that’s a good point. I did… but that was almost two weeks ago! *cough* excuse me… I’d just like fresh backups before I toast the array. These machines are all still in use and the company hasn’t been standing still.

There doesn’t appear to be a chkdsk or fsck for vmfs formatted volumes. Seems like that would be useful.


For web searchers dying to share a cure, below I’ve listed some of the error messages.

GhettoVCB errors are:

  • Failed to clone disk : Connection timed out (7208969)
  • Failed to clone disk : Input/output error (327689)

Sample message log errors:

Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4100051614c0) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "mpx.vmhba1:C0:T1:L0" state in doubt; requested fast path state update...
Jul 15 19:18:50 vmkernel: 0:17:59:18.890 cpu4:16218)ScsiDeviceIO: 770: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
Jul 15 19:18:53 vmkernel: 0:17:59:22.500 cpu4:5365)<4>cciss: cmd 0x4100b1402000 has CHECK CONDITION  byte 2 = 0x3

Here’s another set:

Jul 15 19:49:08 vmkernel: 0:18:29:37.553 cpu7:10409)<4>cciss: cmd 0x4100b1402000 has CHECK CONDITION  byte 2 = 0x3
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x4100050fa000) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "mpx.vmhba1:C0:T1:L0" state in doubt; requested fast path state update...
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu7:10409)ScsiDeviceIO: 770: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Jul 15 19:49:08 vmkernel: 0:18:29:37.559 cpu6:312122)Fil3: 5354:  Sync READ error ('EFops_wsus-flat.vmdk') (ioFlags: 8) : Timeout

Unless I can figure out a way to get those last 3 VMs images or copied – or an alternative way to fix the read errors – I see a long weekend rebuilding machines in my future. Fortunately I can still get all the data from the VMs. I just can’t copy the VMs directly!

WhatsUp Gold Engineer’s Toolkit

30 Jun 2010
June 30, 2010

WhatsUp Gold Engineer's Toolkit The folks at Ipswitch released their WhatsUp Gold Engineer’s Toolkit this week. I was in the beta for the past month and find it a pretty handy ‘kit.

What I hadn’t realized was that they’d be giving it away for free when launched. Nice!

So hey, if you do any network work go get yourself a copy. It won’t replace any single specialized tool (like nmap) but it certainly bundles a lot of functionality in one application.

Here’s what you get:

  • Design & Planning
    • Subnet Calculator
  • Discovery
    • Ping Sweep
    • Port Scanner
    • MAC Address Discovery
  • Diagnostics
    • Ping
    • Trace Route
    • WAN Load Generator
    • Spam Blacklist
    • SNMP Grapher
  • DNS Verification
    • DNS Audit
    • DNS & Whois Resolver
    • DNS Analyzer
  • Remote Control
    • Wake On LAN
    • Remote TCP Session Reset

Each tool opens up in a separate tab so you can have multiple things going at the same time which is handy.

As a network engineer, at least part of your average workday is spent diagnosing and troubleshooting existing problems or investigating issues that could cause new problems. These activities often involve the tedious process of accessing individual network elements to gather information on device or subnet configuration and availability; interpreting that data; and then accessing the same elements again to provision or configure devices and services.

Most of the time you don’t need a powerhouse application to support these activities. In fact, many network engineers use a mix of tools they’ve cobbled together and then rely on brainpower and intuition to make up the difference.  Although this strategy works, there is an easier way to get the job done.

VMware Player 3.0 and Network Configuration

29 Oct 2009
October 29, 2009

I updated VMware Player to the latest version (3.0) on my home machine last night[1]. Wanted to check out the latest version and get a first hand look at the new features.

I pointed it to my trusty Xubuntu “security” virtual machine and booted up. I ran into a snag pretty quickly though. Seemed that the client couldn’t get an IP address to get on the network.

I’ve seen and fixed this before. VMware Player seems to like to grab my Hamachi “personal VPN” network adapters instead of the local Ethernet one – it will never get an IP from the Hamachi network!

That’s when I hit the next snag: the new version of VMware Player doesn’t include the vmnetcfg network configuration that I’ve always used to fix this in the past. I poked around a bit but it wasn’t there and I didn’t see anything that looked like a replacement.

I ended up fixing this in a bit more of a low-tech fashion. I simply disabled the Hamachi adapter and then booted the VMware client. Client got an IP and I then re-enabled the Hamachi adapter.

Disabling a network adapter

Of course, after my low-tech solution I went looking for better options… Turns out I’m not the first one to notice that vmnetcfg went missing. I found discussion and the answer in a forum post:

The vmnetcfg.exe is included in the installer, but won’t be installed.
1. Run the installer with /e option. For example:
VMware-player-3.0.0-197124.exe /e .\extract
All contents will be extracted to “extract” folder.
2. Open “network.cab” and copy vmnetcfg.exe to your installation folder,
typically “C:\Program Files\VMware\VMware Player\”.

Well there you go. Even better.

[1] – Past mentions of VMware Player.

Tip on vmnetcfg

18 Aug 2009
August 18, 2009

Tip: vmnetcfg is a really handy tool to tweak VMware network configs. Just don’t forget to run it as admin if on Vista or Win 7 (I always do…).