Mixed EqualLogic and MD3200i design for vSphere storage

Over the years, I have come across several SMB customers running vSphere on iSCSI.  While a majority of them opt for EqualLogic storage, some have existing MD3000i/MD3200i SANs that have been the workhorse Dell storage solution for years.  One question I that comes up often is ‘Can I run both EqualLogic and MD3200i SANs for vSphere?’

There is a simple way to configure both MS3200i and EqualLogic storage in the same iSCSI network while utilizing only 2 physical NICs for SAN traffic on each vSphere host.


iSCSI SAN Switch Configuration

  • Create 5 Vlans on your redundant SAN switches.
    • vlan10- EqualLogic
    • vlan20- MD (A)
    • vlan30- MD (B)
    • vlan40- MD (C)
    • vlan50- MD (D)
    • All ports designated for vSphere host connectivity should be trunk ports with all iSCSI Vlans.
    • Ports designated for EqualLogic should be access ports on VLAN10- EqualLogic.
    • Ports designated for MD3200i should be access ports on the appropriate VLAN.

vSphere vSwitch Configuration

  • Create 6 x vKernel ports for iSCSI connectivity
    • vmk01-vlan10
    • vmk02-vlan20
    • vmk03-vlan30
    • vmk04-vlan10
    • vmk05-vlan40
    • vmk06-vlan50
    • Bind vmk01, vmk02, and vmk03 to pNIC1
    • Bind vmk04, vmk05, and vmk06 to pNIC2

Configure storage as per Dell MD3200i Deployment Guide for ESXi 5 http://dell.to/SYILt6

Configure storage as per EqualLogic Configuration Guide http://bit.ly/16NNR4N

Mixed iSCSI Networking



This will offer the best redundancy available while utilizing only 2 pNICs for iSCSI storage.  If your hosts have additional pNICs available, you can break out the EqualLogic ports (vmk01 and vmk04) onto two pNICs.  On the other 2 pNICs, bind vmk02 and vmk03 to pNIC3 and vmk05 and vmk06 to pNIC4.  Of course, you could always put them on separate pNICs if you have the spare capacity.

Image | Posted on by | Tagged , , , , , , , | Leave a comment

Dell VRTX- First Impressions

On Tuesday, 6/4/13, Dell unveiled its new PowerEdge VRTX converged system during the opening keynote at the 2013 Dell Enterprise Forum in San Jose, CA. The VRTX comes in a Tower or Rack mounted form factor covering 5RU. Inside, you will find slots for 4 of Dell’s M520 or M620 blades, the same servers found in their M1000 Blade Chassis. An onboard CMC provides iDRAC and remote management for the chassis. Each blade maps up to 4 GigE ports on an integrated switch, and there are 8 additional PCIe slots that can be mapped directly to the blades for more IO and accessory options such graphics adapters. In addition to the shared PCIe slots, there is a shared PERC controller (SPERC) that controls access to up to 25 integrated 2.5 inch or 12 3.5 inch drives. Both Howard Marks and Kevin Houston have done great posts on the details and specifications of the VRTX.

Being a vSphere-centric kind of guy, my first thoughts about the VRTX were that it could replace the commonly deployed “3-2-1” vSphere install (3 hosts + 2 iSCSI switches + 1 EqualLogic array) for SMB and ROBO scenarios. If you had up to 4 blades, integrated networking and storage, in a whisper quiet (and it is VERY quiet while running) box with integrated remote management and monitoring, why wouldn’t deploy a VRTX instead of 6 pieces of hardware?

After several inquiries relating to the shared PCIe slots and storage, I came up with the following answers from the Dell engineers onsite:

Networking: While there are only 4 onboard GigE ports per blade, you can easily add a 10Gig, FC, or CNA into the PCIe slots and map the card directly to a blade. Add 4 x dual port CNAs, map them to the blades, attach to an external switch, and carve up your network as needed.

Storage: This became a challenge. Apparently all 25 disks on the SPERC can be either directly mapped to individual blades, joined into RAID groups, and have virtual disks carved out of the RAID groups, which could also be mapped to one or multiple blades. In practice, it is similar to carving up storage in a MD1200 shelf. Where is became challenging was when I asked if there was a way to provide a shared disk for a vSphere installation. I asked several (more than 10) Dell engineers onsite at the #DellEF, and never got a definitive answer on that. Apparently nobody thought to qualify the VRTX as a vSphere platform. The answer provided was that the SPERC could create a shared disk, and present it to multiple blades via a common SCSI bus. Not FC, Not iSCSI, not NFS. The simple question of “Can you install a vSphere cluster on the VRTX” was answered with “I would imagine it would be possible.”

In conclusion, I was very impressed with Dell’s new converged infrastructure foray, the VRTX. The ability to put 4 servers, networking, and storage into a single 5U box will shake up the industry, especially if the price point is as competitive as Dell traditionally leads with. However, the storage subsystem is not much more than a DAS shelf for the blades. I think it was a great first move, but I would have loved to see an integrated MD3200 or EqualLogic controller to give the storage a little bit of intelligence. There are small form factor EqualLogic controllers currently in use on the PS-M4110 array. As far as installing vSphere on the VRTX, I’m not sure if it will work right out of the box. However, with the use of a VSA such as Nexenta to control the storage and make it available to all blades, this could be a huge success in the SMB/ROBO virtualization market. Well played, Dell. Well played indeed.

For more information on the Dell VRTX, head to the Dell TechCenter Blog, where Peter Tsai has aggregated the latest information.

Posted in blades, converged, PowerEdge, storage, virtualization, vmware, VRTX | Tagged , , , , , , | 3 Comments

DIY Stadium Seating for the SuperBowl

Like several people around the country, I host my family and friends for a big SuperBowl party each year.  Being a die-hard Patriots fan, we have recently been blessed with cheering for the home team more often than not.  But regardless of the teams in the big game, we always have a party.

As the event has grown, so had our party attendance.  What started with 9 or 10 people has grown to over 20 most years.  Trying to fit everybody into the living room has been a challenge lately.  This year, I came up with a novel idea to add more seats.  I would add some tiered ‘Stadium Seating’ in my living room with a few cinder blocks (10) and a piece of plywood (5/8 inch)


To start with, I moved my couch forward away from the wall to give me at least 4 feet of working space.


I laid down the 10 cinder blocks in order to support the plywood stage sufficiently.


Once the base was laid properly, I added the 5/8 inch plywood over, bracing it against the wall.


I moved the couch back so it rested against the plywood to help secure it in place.  Finally, I put 4 chairs up on the platform to add the additional seating.


I am looking forward to all of my guests, and now we can all sit comfortably in the living room for the SuperBowl!


Posted in Uncategorized | 1 Comment

VCAP5-DCA Testing Strategy and Tips

This year at VMworld in San Francisco, the #vBrownBag crew was able to host a series of Lightning Tech Talks, a series of 10 minute presentations in the HangSpace Community Stage. I was lucky enough to do two of the presentations, discussing testing strategy for the VCAP5-DCD and the VCAP5-DCA. This post will outline my presentation on the DCA. All of the presentations were streamed live and recorded. They should be available soon on: http://professionalvmware.com

If you are interested in taking the VCAP5-DCA, there are several resources out there that will help you tackle the technical aspects of the official blueprint. A few of them are listed below, and I recommend that you go over them during your exam preparation.

First and foremost is the #vBrownBag training series. These guys support an amazing site, hosting weekly webcasts on vSphere training. They record each presentation which can be downloaded from their website or iTunes. If you haven’t gone there or attended a #vBrownBag presentation, it is well worth your time on a Wednesday evenings at 7:30 CST. You can find all of the recordings here. http://professionalvmware.com/brownbags/

Jason Langer (@jaslanger) and Josh Coen (@joshcoen) have a great study guide at: http://www.virtuallanger.com/vcap-dca-5/

Gregg Robertson (@GreggRobertson5) has a guide here:


Chris Wahl (@chriswahl) has a study sheet here:


And finally, my own testing experience from the VCAP4-DCA


While there are several good site outlining the content, there are not many resources out there that will help you take the exam. The VCAP-DCA is 4 hours in length. While that sounds like a long time, most people that attempt the exam do not complete it in the time allotted. With the right approach to taking the exam, you should be able to efficiently work your way through the questions and wrap up your lab with time to spare.

Let’s look at the structure of the lab and understand the way that it is organized. When you begin the exam, you will be presented with a remote session to the Exam datacenter. You will have your workstation, RDP to a vCenter server, the vSphere Client, Documentation in PDF format, and such. You will be able to access your virtual lab environment and perform any functions or operations that the exam requires. On your test workstation, you will have the Exam interface. This is where all 26 questions will be located. Each question will require you to perform some task within your lab. Don’t worry about writing down any usernames or passwords. They are shown on every desktop and easily accessible.

I mentioned that there are 26 questions on the exam. If you check the DCA blueprint carefully, there are 26 sections that you need to cover. I’m not saying that there is a question on every section… What I AM saying is that one section of the blueprint is no more important or less important than another. It is there for a reason, so my advice is to learn each section to your best ability before scheduling the exam.

When you sit down in the Testing Center, you will be given a plastic coated page and a dry erase marker. In previous exams, I have simply left them unused during the test. This time, they may be your best friend.

When the exam starts, read question 1. Take your dry erase board, and number down the left side, 1 through 26. After 1, write a brief (2-3 words) description of the task described in question 1. Select the ‘NEXT’ button and progress to question 2. Read the question and write a brief description after the 2 on your board. Repeat this process until you reach the final question. After writing the last description, move backwards through the question until you reach question 1 again. You have now outlined all of the tasks involved for your examination, and the process should take no more than 10 minutes.

Your next step should be to evaluate each task and description briefly. Based on the DCA Blueprint, there are two distinct types of tasks involved in this exam, ‘Create’ and ‘Administer’. The ‘Create’ tasks involve building or modifying your lab environment in a manner that makes it possible to perform the Admin tasks. For example, before you could evaluate or optimize the performance of an Air Conditioner, you would probably need to purchase and install the AC unit in your house.

Go down your list of questions and tasks, and mark them with a ‘C’ or an ‘A’, indicating that they are either Create or Build tasks. This should only take about 3-5 minutes at the most. When you are done with this, you are ready to start the actual testing phase of the DCA.

Here is where my recommendations are a bit different from most others out there. I recommend that you start performing the Create tasks, one by one, and skip over the Admin tasks at this time. Why would you do this? Because sometimes the tasks involve processes that run on vSphere hosts, Windows servers, or the shared storage earmarked for your test lab. Some of these processes take time to complete, and if you sit there watching a server reboot, YOU ARE WASTING TIME!!! Additionally, for you to perform ‘Admin’ tasks in your lab, there is a good chance that you will need to have a piece of your lab in place (‘Created’), in order to complete the task. Try to complete all of your ‘Create’ tasks before tackling the bulk of the ‘Admin’ ones. Remember, if the lab doesn’t exist, how can you administer or evaluate it?

The goal of this exam is to test your administrative skills in a vSphere environment. Don’t waste time watching progress bars. If you have launched a process that you know will take 3-5 minutes or longer to complete, leave that process alone and move to another question. If the next ‘Create’ task is one that is dependent on the current task, switch gears and move backward in the question stack to one of the ‘Admin’ tasks. Fill in that empty time with a useful task.

Complete the ‘Admin’ task you have selected, and return to the ‘Create’ task you left earlier. If you are finished with a task, draw a line through the task on your dry erase board to keep a running tally of what you have left and what you still need to accomplish. This is the only time that you should perform ‘Admin’ tasks until you have completed all of your ‘Create’ tasks. During your exam, you may have several of these pauses in the ‘Create’ process in which you can tackle an ‘Admin’ task. However, keep an eye on the timer in the top corner of the window. Don’t get drawn into a task if it is taking too much of your time. Move along to another task and keep working. It is better to complete 24 out of 26 tasks, skipping 2 that took too much time than it would be to only complete 14 tasks because number 15 was not working as you expected. Move along and capture as many completions as you can.

When you are done with your ‘Create’ tasks and the lab is finished, move onto the remaining ‘Admin’ tasks. One advantage to this methodology is that most of the ‘Create’ tasks build upon each other. By the end of the exam, your lab will be completely fleshed out. The ‘Admin’ tasks usually aren’t dependent upon each other. The various ‘Admin’ tasks rely on the ‘Create’ tasks, but not each other. Here is where you can make up some time, based on your own personal strengths and weaknesses. Browse through the ‘Admin’ tasks and select one that is relatively simple for you to accomplish. This may be question 22 or question 3, and it usually won’t matter which order you work on them. My advice is to get as many out of the way that you are comfortable with first, before tackling other ‘Admin’ tasks that you are not as skilled performing. If you aren’t good at scripting or CLI tasks, don’t try to do those first and waste time researching and debugging a script. Move along to a different task. If you have extra time at the end of the exam, return to that difficult task and try to complete it. If time runs out, at least you didn’t miss completing your more familiar tasks.

Hopefully this will give you an advantage when attempting the VCAP5-DCA. Sometimes it isn’t what you know or don’t know that trips you up, but how you approach the game that makes the difference. Good luck!

Posted in certification, VCAP-DCA, VCP, virtualization, vmware | Tagged , , , | 16 Comments

Disaster Recovery and VMware, Pre-SRM

A new customer asked me today if I had any experience with Disaster Recovery, and if I was familiar with VMware’s Site Recovery Manager product.  My Sales Associate said “Of course he knows it, He invented SRM”.  I had to correct him, that I did not invent SRM.  However, I was involved in a co-hosted DR project between my former employer, Bowdoin College in Maine, and Loyola Marymount University in Los Angeles.  Out of curiosity, I searched Google and found an old video that I shot at VMworld 2006 in Los Angeles.  This video outlines our co-hosted DR project.  This video, followed by a lengthy interview at the Palo Alto campus with the DR group was followed up at VMworld 2007 in San Francisco with a mention in Diane Greene’s keynote and subsequent announcement of Site Recovery Manager.

Anyhow, here’s the link to the interview.  I know, it’s pretty cheezy.  Don’t blame me… VMware did all the post-production work!

VMware and Disaster Recovery at Bowdoin College

Posted in DR, Uncategorized, virtualization, vmware | Tagged , , | 3 Comments

VMware ColdClone utility

If you are a SysAdmin using VMware or work with customers in a virtual environment, you have more than likely performed a Physical-to-Virtual (P2V) migration.  In most cases, the free VMware Converter will perform  nicely here by imaging the physical server, adjusting components such as CPUs, memory, or drive sizes, and substituting virtual hardware for the physical components.  This is a very straightforward piece of ‘magic’ that has been the mainstay of virtualizing datacenters for years.

However, there is a piece of the puzzle that has been left out of the SysAdmin’s virtual toolkit for a few years.  If you go back several years, to the VMware Converter v3.03, there was a bootable iso version of the Converter available.  This utility allowed a SysAdmin to boot to the CD, load the Converter into memory, configure the local network, and image the server directly into a VM.

The greatest advantage to this utility was the fact that the server was shut down and offline.  It allowed you to capture the server with the OS completely quiesced, with no open files, or application conflicts.  If the server was on a different network segment than the vSphere hosts, simply connect it to the correct network with the server down and set the IP from the boot image without changing the server OS settings.  There was great advantages to having this CD in your laptop bag.

Alas, all good things come to an end.  VMware stopped supporting and updating the ColdClone image, and eventually dropped it from their website for download.  There is a rumor that it can still be found in the archives and copies are on shelves in the VMware ‘SkunkWorks’ labs.

Fortunately for me, I still have a copy or two stashed away on portable hard drives, burned to CDs, and such.  Sometimes it pays to be a software ‘PackRat’.

Additionally, I have taken other steps, like putting it up on my GoogleDrive so I can access it from anywhere.

Posted in P2V, virtualization, vmware | Tagged , , | 14 Comments

Recover a VM from the vm–flat.vmdk file

Last week I was onsite with a new customer, spending some time familiarizing myself with their existing infrastructure.  They are running vSphere 4.1 on Dell servers and EqualLogic storage.  While the hardware layout was solid and installed well, it quickly became apparent that the SysAdmin was a duck out of water.  There was a knowledge and comfort gap that could cause problems down the road if he became careless.  Little did I know that ‘down the road’ would be 2 hours later.

As the next few hours progressed, I found that he was using BackupExec agents within his critical business applications (2 VMs) and backing up directly.  The rest of his VMs were unprotected other than EqualLogic snapshots.   I pointed out to him that his snapshot schedule created one snapshot per hour, keeping 12 snapshots.  This protected him for less than one day.  He was at significant risk, and I recommended a more aggressive snapshot schedule as well as a more comprehensive backup plan.

We covered several other topics over the next few hours such as vMotion, Storage vMotion, and basic VM operations.  As I prepared to leave, he decided to perform a few Storage vMotion operations on VMs he had renamed, causing mismatched folder names within the datastore .  One of them happened to be his Domain Controller, which I will call ‘MismatchDC”.  When he tried to migrate to another datastore, vCenter threw an error stating that “the file mismatchDC-vmdk could not be found” and shut down the VM.  Within minutes, his phone was exploding with calls.  Nobody could login or get Exchange email.  He had pooched his Domain Controller.

While panic set in around his office and he was scrambling around, I opened up a datastore browser and was shocked.  Apparently the SysAdmin had deleted some ‘unnecessary files’, such as the VMX, VMDK, and everything else except the NVRAM and mismatchDC-flat.VMDK file!  These remained simply because they wouldn’t delete when he tried to delete them.

A quick check of the EqualLogic snapshots determined that he deleted these more than a day ago and they were not available, I also found that the Domain Controller was not on the list of ‘critical servers’ that they were protecting with BackupExec agents.  There was no way to restore this VM!  He started making plans to build a new VM s a DC, run DCPROMO, and sync across the WAN from his second site on the other side of the country.

As he was making his recovery plans and trying to explain to his boss how they would be down for the rest of the day, I recalled that I had read a KB article a while ago about recovering from a flat.vmdk file.  After a quick collaboration with my friend Google, I found it.  While it didn’t exactly fit my needs, it was enough to get rolling.  I took the size of the flat.vmdk file, divided by 1024, and divided by 1024 again to get the size of the VMDK in GB.  Luckily it came out to 25 on the nose.  Assuming that it was Win 2k3 server with a 25 GB drive, and the SysAdmin verified this from memory, I created a new VM called Temp as a Win2k3 server with 1 vCPU, 2 GB RAM, and a 25 GB HDD.

I enabled remote SSH, and logged in via Putty.  I went to the mismatchDC folder to verify contents.

cd /vmfs/volumes/vmfs01/mismatchDC/

I was correct in finding only the mismatchDC.nvram and mismatchDC-flat.vmdk files there.  I then moved over to the Temp VM location.

cd ../temp/

and found the temp-flat.vmdk file.  I renamed this to prevent it from being overwritten.

mv temp-flat.vmdk temp-flat.vmdk.old

Next, I copied over the original flat file and renamed it temp-flat.vmdk

cp /vmfs/volumes/vmfs01/mismatchDC/mismatchDC-flat.vmdk temp-flat.vmdk

When finished, I simply attempted to start the Temp VM.

As it started up, I saw the familiar Windows 2003 server splash screen, and called the SysAdmin over to verify that it was working.

After a few minutes of evaluation, it was proven that the Domain Controller was back up and running with less than an hour of data loss.

Steps to recover a VM from just the flat.vmdk file:

  1. Build new temp VM with EXACTLY identical vmkd file size
  2. Connect via CLI
  3. Rename temp-flat.vmkd file
  4. Copy existing-flat.vmdk file and rename to temp-flat.vmkd
  5. Power on temp VM

Lessons Learned:

  1. Backup your Servers, physical or virtual
  2. Snapshots are NOT backups
  3. Don’t delete things you don’t understand

VMware KB article 1002511


Posted in backups, DR, equallogic, virtualization, vmware | Tagged , , , | 13 Comments