Brents IT Blog

Random thoughts by an IT GOAT


VMWorld 2016

I went to VMWorld 2016.  I have been a few times before and i think there is a previous post here about it.  If you have not been and you run a VMWare environment or wish to set one up, i highly suggest you go to one.  They also have a European conference if you are not located in the U.S, my experiences only reflect the U.S. conference.  

This years was held at Mandalay Bay in Las Vegas.  Technically, in their conference center, which is huge and poorly laid out(imo).  If you did attend, you hope you brought your walking shoes.  I believe the average steps people were reporting when i asked ( cause i didnt log mine lol) was around 20k A DAY.  Yes, a perfect indicator and my proof that this conference was poorly laid out.  I really wish they had gone back to the Venetian, which was VERY well laid out.  At the Venetian, the conference rooms were up or down escalators and the expo area was in the middle, so you could bounce in and out between sessions.  At Mandalay Bay, everything was far apart with the Expo and eating at one end, a small conference area above.  The rest of the conference area was spread out which made getting to sessions a 10-15 minute hike at times.  If you put sessions back to back, you would most likely miss your badge in time walking between rooms, you may even get bumped due to capacity ( due to the stand by line use ).  

They offer meals as part of your conference registration.  This is great as there is generally no capacity at nearby restaurants for 25k hungry people of varying dietary needs.  My complaint, every year i go, is the food.  Its not that the food is bad, its good or great depending on the day.  Its just that the chef seems to be aloof most of the time.  They put out breadstick, but no marina?  Biscuits or bread, but not butter/jelly?  Barbecued meat, but no sauces?  Bread for sandwiches, but not meat?   Tacos, but not cheese?  Condiments, forgetaboutit!  I just DONT GET IT.... i have been four times, its the same thing each time and yes, i fill out the survey and complain, but it seems they only listened to my complaint about their meals being created by a nutrition natzi.  I am a bit of a picky eater, so after my first conference, i complained very loudly about the food being primarily vegan, which was great if you were a vegan ( meat and potatoes please ).  Also, if you love gourmet deserts, they have them, i dont but this is a bonus, so i wont complain and ruin it for everyone else.  One good thing with meals is that they offer snacks in the session areas around 2-3pm, which is awesome.  They have bagged snacks and gourmet snacks, tea/coffee, and sodas ( water is everywhere ).  Again, not big on the gourmet stuff, but they have alternatives thankfully!

Now, as to the content of the conference.  Its really what you make of it and the sessions you can get into.  This year the stand by line was the only way to get into a full session, which usually meant you were standing.  Last one i went to a few years ago had an online version of this which was great because you would be notified of sessions you put yourself on stand by for.  This year, that system broke and they disabled it two weeks before the conference.  Something about too many people wait listing certain sessions.  Why they couldnt just ADD more times for those sessions, i dont know, perhaps logistics with the presenter?  They did repeat several popular sessions, but they appeared to be ones with VMWare staff presenting, so perhaps they just forced them to do another?  The moral of this story, as it has been for many years now, sign up for sessions the second they enable it.  Yes, the second they enable it!!  Review them, if possible, before hand and figure out where you want to go.  Make sure to favorite ones you like or may want to attend just in case you need to switch later.  I sincerely hope they fix this by the time i go in 2018( i try to go every 2 years ).  Also, the general session to me is a waste of valuable time, but i understand why they do it.  Breakout sessions is where its at!

Breakout sessions this year were full of great content.  They made the trip worth the effort ( though talking to expo sales people was also interesting ).  I only had two that i wish i had skipped and one of those still had a little useful information.  The big takeaways for me were:

Wait till they publish the VMWare 6.0 update 2 migration tool before upgrading to vsphere 6
This is the tool that allows you to migrate from a windows based VCenter deployment to the VCenter appliance or VCA.  The VCA in 6.0 is capable of supporting the same type of environment as a VCenter windows server now.  This means a lot for me as currently we have a 2008 R2 windows server running VCenter and a separate database server running SQL 2008 R2.  We have stayed at 5.5 because last time i migrated from 4.1 to 5.5, i had waited until the dust settled and it was completely worth it given the management server issues.  Yes, it was soo bad, the conference i went to on 5.0 actually had a "We are soo sorry, heres what we are doing to fix it" session.  Side note, I have been using VWmare since 3.5.  So vsphere 6 looked to be the same type of situation and the conference and many blog posts have certainly backed that up.  Not as bad as 4.1 to 5, but similar in problems.  The new migration tool automates the tedious process of migrating your VCenter to a VCA ( or many VCAs, future ).  You do need to figure out the redundancy thing for your management nodes, but this tool makes it soo much easier to migrate.  On top of that, VCAs are tuned for VMWare out of the box.  This means your web client will actually be tolerable ( i hate it in 5.5 ), supposedly...  having a domain based windows machine reliant on a MSSQL db is just bad news if things go down in a small cluster.  Using a VCA makes way more sense, so just do it!  Make sure to plan for any scaling before migration and of course wait for the migration tool, which is coming "soon", their words, not mine.

NUMA/vNUMA is your friend until its not
If you are like me, you like having hot plug memory/cpu on your virtual machines enabled.  You also like to migrate virtual machines as needed.  This is generally fine, until you get to large or even monster VMs where NUMA is essential to performance.  NUMA nodes on larger machines are essential to ensuring the machines perform well.  Enabling hot plug cpu disables NUMA!  Holy crap, can i get a warning message for machines with more than 8 vCPUs?  I assume those who read into the docs and are performance nuts would know this, but us general IT guys who focus on more than just VMWare, dont have a red flag to look at until performance issues arise. The increase in performance with NUMA enabled and working is 30-40%, nothing to sneeze at, so this was very enlightening for me.  They also suggest you disable hot plug memory.  If this is a database server, these restrictions are annoying, BUT you need to do it on VMs if they have more than 8 vCPUs.  Also, make sure to be aware of how your hosts are configured.  This play a very important role as to how NUMA is balanced.  If you have 16 vCPUs assigned and your processor has 10 cores with hyperthreading enabled, then NUMA will place the entire load on ONE CPU to ensure the memory is in lock step with the processor.  If assign 24 vCPUS, then you will end up with 2 NUMA nodes, so just keep that in mind.  This also means you need to install memory distributed properly between the CPUs AND setup the VMs so that they dont use more memory than the CPU has assigned to it.  If you have 64GB per CPU, and you assign 128, well you will end up with 2 NUMA nodes because the memory is more than the processor has allocated to it.  All this information is very important to know, especially if you plan to migrate a virtual machine between hardware that has different amounts of capacity!  Once NUMA is established, which is during boot, it keeps the configuration until you power off the machine and power it back on ( not reboot, yes reboot doesnt update the config ).  If you plan to move a machine, move it to another machine with the same config or more capacity, not less or you will regret it as performance will be affected ( possibly severely ).  They are working to deal with this, but its not ready yet.

Performance, use the new hardware and test, test and retest
Rightsizing your VMs in vmware is already essential, more CPU does not mean better performance.  Make sure you baseline your VMs and know what metrics indicate the need for more vCPU and/or memory( CO-STOP and WaitTime for CPU).  There are some great deep dives online in troubleshooting performance, review them all if this is your gig!  ESXTop is your friend, there are other tools as well.  Use VMXNet3 network cards to take advantage of better networking performance.  This is also essential to use 10Gb if you have 10Gb networking.  Use the VMWare Paravirtual SCSI controller.  Not only use it, but for database servers with divided up drives, use one per drive.  Same for MS Exchange when dropping databases per drive, use a different Paravritual card for each drive.  DO NOT USE E1000e cards if you can avoid it, they are known to have issues( i found this out years ago when troubleshooting a direct iscsi path over an e1000e card in win2k12, pretty sure that KB is because of my efforts, thanks but no thanks! ).  There are several guides online for swapping these components out in a structured fashion.  Hint, its much easier with vmtools installed or upgraded ahead of a change out.

This wasnt really a takeaway, but more of an observation.  If you want to get your VCP ( like i do ), sign up for the discounted testing voucher and take your test at vmworld.  I passed my 2V0-621 on the Sunday morning before the conference started.  It was nice because i was able to finalize my studying on the plane trip there, then walk out of my hotel room, downstairs and over to the conference area to take my exam ( also picked up my conference registration that morning ).  Still have to take the training they require, but they have work at home labs i can do instead or get the company to pay for an onsite week long course.  I find it annoying they require this since i have been using vmware since 3.5 and have been to a previous week long training course.  There is another test i could of sat for as well while there, but i found out its not a proctored test and can be taken online, so i did that test online after the conference.  I suppose you could say i am taking the VCP in reverse order, LOL.  I have several certifications, so me taking things out of order is nothing new nor scary for me.

One last note, i didnt go to the VMWorld party this year due to a friend being in town on that day.  I heard from several people that it was miserably hot ( 106 degrees ) because of where it was, concrete bowl facility and the "race cars" were actually street cars like camry, sonatas, etc.  There were great reviews for the concert portion though!  Many also liked the remote control car racing.  So all in all, not bad according to heresay.  I have been to all previous parties when i attended, the concert is usually the highlight and why many go.  I think the AT&T stadium event was probably the best one yet though.