What is RSC and why does it matter in Vmware

Got latency on your VM’s? It might be RSC

Let’s say we have two VM’s with the  following specs:

  • Windows 2012r2 (Latest patches)
  • VMxNet3
  • Same VLAN
  • Same version of ESXI 3029758
  • Both HW version 11
  • Both have the latest VMware tools for the ESXI Build 3029758
    Basic host diagram

Any new latency between these two machines wouldn’t be expected considering they are both within the same physical location and all within the same topology that has been in place for months almost untouched.

After Vmotioning both to the same host, the latency goes away. That would be expected considering VM’s will use the internal networking for VM to VM networking located on the same host. Bypassing the need to go out to the VDS or vSwitch As shown below.

Local host diagram

 

So let’s go back to the title of the post. What is RSC and why does it matter to VMware?


RSC ( Receive Segment Coalescing) is a technology used to help CPU utilization on a server. It achieves this task by taking the payload off of the CPU and giving it to the network adapter, in our case the VMxNet3 VMware adapter.  RSC Strips headers from the packet’s, combines those packets, then sends those packets to the right destination.  Without RSC the receiver would get 4-5 packets, but with RSC enabled the receiver only has to process the single packet sent with the 5 packets of information stuffed inside.

With VM Hardware version 11 there was a bug introduced that caused the ESXI to not keep the data properly is the PSH flag (PSH Flag explanation ) was not written in the first packet  but was written to the packets following. If you read the article below it gives an awesome example of why this is useful.

Imagine you are walking in a line of 5 friends, friend 1 doesn’t have a pass to get in the gate, but friend 2-5 does. Being a gentleman friend 1 lets friend 2-5 go thru while he buys his ticket. But then friend 2-5 are waiting for him while they are already in the park. pretty similar concept to the PSH flags’.

While packet 2-5 have the PSH flag that grants them permission to go to the application, ESXi has a hiccup while waiting for the PSH flag on packet 1 causing it to wait before the packet can be sent and the full information is received.   Here is the KB article that has the problem highlighted Vmware KB .

What is the fix?


To fix this problem the solution is pretty simple on the OS side you can disable RSC and stop the function, but doing this keep track of the memory use on that box and the box that received a bulk of those packets.

 

Running the command  Will give you the output in below:

netsh int tcp show global

netsh int tcp show global

Received Segment Coalescing State is what we are looking for. If that shows as enabled run the command below to disable it. you should receive a similar output below after running the

netsh int tcp set global rsc=disabled

netsh int tcp set global rsc=disabled

you should receive an output similar to the one above after running the   netsh int tcp show global again.

 

What is affected?


Currently, this affects people running ESXI 6.0 build  3568940 or below also running windows 2008 r2 and above. This problem can be solved 1 of two ways. By updating to ESXi 6.0 Update 2 Build 3620759 or above, or by running the above command on your machines affected by the problem.

 


 

KB Articles below for reference also for citing sources:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2129176

https://communities.vmware.com/thread/524842?start=0&tstart=0

https://technet.microsoft.com/en-us/library/hh997024(v=ws.11).aspx

Starting a career in architecture and design

I am taking my design and deploy class next week. i am excited, but the only problem is how do you practice this new found skill without working for an MSP?

I think starting a new career in design and architecture can be kind of daunting. Where do you start? How do you proof it? Do you make mock designs and hand it off to a trusted colleague? i think all of these are good ways to practice design. but the floor is yours…

How do you practice design?

Let me know below in the comments what your idea is. I will update this post and add the good idea’s i get.

Part 2: Becoming a great engineer

I would just like to say this post is in NO WAY insinuating I am a great engineer. I believe that no matter how much you know or think you know, there is always room for improvement and development. That being said these are some things I believe makes a technologist really stand out from the rest of the crowd.

This next post will be about the social aspect of being a great engineer. Being a part of the technology community is an essential a part of someone’s career and growth within IT.

 

Social Networks


Social networks like twitter are a great place to network and connect with other “Like-minded” people. I’m not gonna lie its sounds kind of creepy when I say “Like-minded” but none the less twitter is a great place to get in touch with some amazing and very smart people. You will find some great bloggers  like Duncan Epping ( @DuncanYB ), William Lam ( @lamw ) and Sean Thulin ( @Sthulin ).

You will find some great bloggers  like Duncan Epping (@DuncanYB), William Lam (@lamw) and Sean Thulin (@Sthulin ), just to name a few. But twitter is also great for  touch with the companies that create these awesome technologies. Vmware (@VMware), IBM  (@ibm) Cisco (@Cisco) Dell EMC ( @DellEMC )

Technology forums Like VMTN and VMware Reddit are a plethora of information, i would strongly consider these as social forums you will see a lot of VExperts and VCDX’s on both sites doling out information like they are making money. They are sites that you should take a look at.

 

Networking


It is important to connect with these people for a couple of reasons, the most obvious one is the networking aspect of our jobs. Networking is essential to gaining ground in your career, always having friends in a lot of different places is not only good if you get Laid off or looking for a career move, but is also good when looking for advice. Whether you want to reach out to these contacts for advice on a new product that might be a right fit for your environment or even better if you have an issue and your contact works for that company. The opportunities are endless when you have a great network of people at your side.

I guess the hardest part of this is finding those mentioned people. Some people will find this the hardest part of networking, is breaking the ice. In my experience, I have never had an issue at a local meeting or conference that I was pushed away for ease dropping on a technical conversation. Usually starting with a ” Hey I overheard you talking about (Insert here)” usually is a pretty good ice breaker. Learning to carry on the conversation is key though and i would have to write a whole blog post on that topic. Becuase if you know me you will know that I can keep a conversation going and have no issues with talking to people i don’t know, obviously sometimes to my detriment.

 

Honesty and confidence


The Last section i have for you tonight is the most important. What makes an engineer stick out is his/her ability to talk about technology with honesty and confidence. Knowing a product is one thing, a lot of people know one technology or another but very few can master that technology and live to tell the tale. Speaking honestly about your experiences is essential to being a great engineer. there is no other community out there that can smell bull shit like IT and most of the time they don’t mind calling you on it. It takes a great engineer to say “I don’t know” it proves that you are humble and understand that you won’t have all the problems solved. But the ability to take that “I don’t know” and make it into a learning experience is key. Being honest to yourself is the most important part. You may not have the answer to every question, but a great engineer knows where to find it.

Confidence is essential for anyone not only in IT but any industry. When you are confident in your beliefs people trust you. Confidence is one of those things that strikes comfort in the heart of people. When you go to a client or manager and say “I know the issues to the problem X,Y,Z,  And these are the solutions.” It gives them the perception that you knowing what you are doing, But also makes them feel like you have done this before, even if you haven’t. Comfort from upper management or clients is how decisions get made and how things get fixed potentially. But remember the rule above. YOU MUST BE HONEST! If the task is out of your reach or if you don’t have experience don’t get into a corner you cannot get out of.

 

I hope you like this blog post. Please comment any suggestions below!

Part 1: Becoming a great engineer

I would just like to say this post is in NO WAY insinuating I am a great engineer. I believe that no matter how much you know or think you know there is always room for improvement and development. That being said these are some things I believe makes a technologist really stand out from the rest of the crowd.

Learning


The most important step to becoming an awesome engineer is the ability to learn. Might seem pretty obvious, of course, you have to learn to become an engineer, but the truth is you must have a love of learning and a passion for your trade to not only learn but retain the information that you absorb. This requires patience and practicing your trade even when away from the working environment. This is a pretty easy skill but takes a lot of dedication. Spending 8-12 Hours a day in front of the computer might make you loath technology when you are home, which is fine but Spending time working on your own learning path and interests becomes easier the deeper and more often you dive into it.

Home Labs!

Set up your own home lab! Home labs are an awesome way to test, troubleshoot, and/or break technology. They offer you the perfect opportunity to go all out on a test environment for your own sick pleasure, like breaking a perfectly good nested ESXi hosts or seeing what happens when you rip PostGres out of Vcenter. All this can be done in a home lab and also provide awesome learning opportunities without the risk of bringing down production of work environments.  Some great articles for home labs can be found on google. But don’t be taken aback by the high prices that are shown on some home labs, you can run an awesome lab on a single host with a lot of memory and CPU. I build my home lab for about 600 dollars and it works just fine for what I use it for.

Social Channels

Social channels are huge in the learning path for any technologist. Networks like twitter offer a fast and current view on new technology releases and news. It’s also great for networking, following your favorite engineers or CEO’s provides you with an opportunity to not only pick their brains on certain things but also connect with them and bring them up in topics you are unsure about. With the various technologies new and old represented on sites like twitter, there are unlimited opportunities to learn. Reddit is another great resource for information and learning. Reddit has become the epicenter for questions and answers on the web. You can find literally anything! Set up your account and starting asking tough questions that have you puzzled. this will hopefully lead to some solids answers. But also beware Reddit isn’t very moderated on some sites and you can get some wrong information, but in that case, it is usually discredited pretty fast.

Why?

The one word “Why?” is, in my opinion, the most important word someone can ask. Not only in technology but in any field. “Why?” is that gateway to every question in technology or life for that matter, It opens up a gateway of knowledge. I find this one-word phrase irreplaceable, trying to understand technology isn’t a “Know one, fit all” experience. In order to properly grasp technology and the concept of being a technologist you must understand that one product is not one technology, it is a buffet of many different technologies. Some of those technologies may be bleeding edge and barely touch upon, and some may be two decades old but still running strong.  Asking why will bring out the questions that sometimes aren’t easily discovered otherwise. “Why?” is by far the most important question a technologist can ask.

A general system error occurred: Connection refused

capture4

Logging into my dev environment today I had an issue where I couldn’t start VM’s.  This was happening across clusters and different hosts.

To get the VM’s up you can log into the ESXi host using the Vsphere client.

The issue is the workflow service either hung up or stopped in the VCenter server. In order to fix the workflow service, you will need to check the status and take the proper measures. Take a look below.


  1. Check the status with the command below. you will get an output similar to below. INFO:root: is the line where the service

service-control –status vmware-vpx-workflow

capture1

2. Once the status is found use one of the two commands to either start or stop the service.

service-control –stop vmware-vpx-workflow

capture2

service-control –start vmwarevpx-workflow

capture3

3. Try to power on a VM now and see if the results worked!


There’s no current KB article from VMware on this.  But plenty of resources on the topic.


 

Vmworld 2016

Keynote and new products


I was interested to see if they would make any major announcements about private cloud offerings, maybe something like  a major VSAN upgrade. But maybe that is too small for a major conference keynote but I don’t think so!

One major announcement at VMworld was Cross-Cloud platform VCloud. I thought this was awesome announcement managing all your cloud environments through VCloud will give VMware customers the ability to not only have a familiar platform for their cloud services but also not feel tied down to a single cloud service or having to manage multiple cloud services all from different locations. I truly believe a hybrid solution of private and public cloud will be the future for IT and business. It just makes sense. have the tier 1 apps private and have dev and lower tiered apps running public.

All in all the announcements were good but not really what I wanted to see.


The vendors

This was my first VMworld and also my first conference of this size. When I walked into the sponsor showroom it was kind of overwhelming at first. never really seen booths this big in size and very elaborate. It seemed like these vendors all brought their A game for VMworld, full-size demo’s, product reviews every hour on the hour. and awesome giveaways for attendees. It also seemed like every vendor had a magician,  which was awesome. most of the time they would do some cool tricks without even giving a product pitch. If I had to rank one vendor who went above and beyond this year it had to have been Veeam. their booth was awesome. Their Magician was amazing and their sales and tech staff were able to answer every question I had.

As for the technology that was front and center this year, One was converged infrastructure. VCE/EMC was Showing off top tier products at top tier prices, VXrail, VMax, and VBlock. Truly a great solution depending on the business you are running but also expensive. Hyper-Converged infrastructure was heavily present. With great companies like Simplivity, Nutanix, and VMwares VSAN. I was personally interested in learning these products just to expand my own knowledge of the technology and products.


The Parties!

The gatherings where awesome this year I met a lot of really smart and cool people. The Veeam party was off the hook, an awesome DJ, some wicked cool violinist, and cellist. Not to mention the full bar which is awesome. They had these awesome synchronized bracelets that were controlled by the DJ so the room was lighting up together it was a really cool to see! Whether or not you use Veeam as a product or not one thing is clear, THEY THROW AWESOME PARTIES!


Conclusion

VMworld was an awesome experience. The networking, the atmosphere, and the venue were a sight to see. Will I be going back next year? Most definitely I can’t wait. if you love technology and shooting the shit with people then VMWorld is an awesome experience and one that you should have..

 

Vmworld 2016 sessions top 10 and keynotes

So VMWorld was awesome! It was filled with great sessions great people and all around an awesome atmosphere. some of the top 10 sessions are listed here.

http://www.vmworld.com/en/sessions/top-10-us.html

A complete list of most sessions can be found below.

http://www.vmworld.com/en/sessions/2016.html

If you didn’t have a chance to make it to VMWorld this year don’t worry! The keynote sessions and the breakouts are available on VMWare YouTube page also!

https://www.youtube.com/playlist?list=PLeFlCmVOq6ysL5tbnypeSTJeCtsk7Rd1S