A general system error occurred: Connection refused

capture4

Logging into my dev environment today I had an issue where I couldn’t start VM’s.  This was happening across clusters and different hosts.

To get the VM’s up you can log into the ESXi host using the Vsphere client.

The issue is the workflow service either hung up or stopped in the VCenter server. In order to fix the workflow service, you will need to check the status and take the proper measures. Take a look below.


  1. Check the status with the command below. you will get an output similar to below. INFO:root: is the line where the service

service-control –status vmware-vpx-workflow

capture1

2. Once the status is found use one of the two commands to either start or stop the service.

service-control –stop vmware-vpx-workflow

capture2

service-control –start vmwarevpx-workflow

capture3

3. Try to power on a VM now and see if the results worked!


There’s no current KB article from VMware on this.  But plenty of resources on the topic.


 

Vmworld 2016

Keynote and new products


I was interested to see if they would make any major announcements about private cloud offerings, maybe something like  a major VSAN upgrade. But maybe that is too small for a major conference keynote but I don’t think so!

One major announcement at VMworld was Cross-Cloud platform VCloud. I thought this was awesome announcement managing all your cloud environments through VCloud will give VMware customers the ability to not only have a familiar platform for their cloud services but also not feel tied down to a single cloud service or having to manage multiple cloud services all from different locations. I truly believe a hybrid solution of private and public cloud will be the future for IT and business. It just makes sense. have the tier 1 apps private and have dev and lower tiered apps running public.

All in all the announcements were good but not really what I wanted to see.


The vendors

This was my first VMworld and also my first conference of this size. When I walked into the sponsor showroom it was kind of overwhelming at first. never really seen booths this big in size and very elaborate. It seemed like these vendors all brought their A game for VMworld, full-size demo’s, product reviews every hour on the hour. and awesome giveaways for attendees. It also seemed like every vendor had a magician,  which was awesome. most of the time they would do some cool tricks without even giving a product pitch. If I had to rank one vendor who went above and beyond this year it had to have been Veeam. their booth was awesome. Their Magician was amazing and their sales and tech staff were able to answer every question I had.

As for the technology that was front and center this year, One was converged infrastructure. VCE/EMC was Showing off top tier products at top tier prices, VXrail, VMax, and VBlock. Truly a great solution depending on the business you are running but also expensive. Hyper-Converged infrastructure was heavily present. With great companies like Simplivity, Nutanix, and VMwares VSAN. I was personally interested in learning these products just to expand my own knowledge of the technology and products.


The Parties!

The gatherings where awesome this year I met a lot of really smart and cool people. The Veeam party was off the hook, an awesome DJ, some wicked cool violinist, and cellist. Not to mention the full bar which is awesome. They had these awesome synchronized bracelets that were controlled by the DJ so the room was lighting up together it was a really cool to see! Whether or not you use Veeam as a product or not one thing is clear, THEY THROW AWESOME PARTIES!


Conclusion

VMworld was an awesome experience. The networking, the atmosphere, and the venue were a sight to see. Will I be going back next year? Most definitely I can’t wait. if you love technology and shooting the shit with people then VMWorld is an awesome experience and one that you should have..

 

Vmworld 2016 sessions top 10 and keynotes

So VMWorld was awesome! It was filled with great sessions great people and all around an awesome atmosphere. some of the top 10 sessions are listed here.

http://www.vmworld.com/en/sessions/top-10-us.html

A complete list of most sessions can be found below.

http://www.vmworld.com/en/sessions/2016.html

If you didn’t have a chance to make it to VMWorld this year don’t worry! The keynote sessions and the breakouts are available on VMWare YouTube page also!

https://www.youtube.com/playlist?list=PLeFlCmVOq6ysL5tbnypeSTJeCtsk7Rd1S

Day 0 VMworld

IMG_20160830_143156

So day 0 was pretty great I took a late flight from Boston to get here and landed around 12AM PST so I was tired. but the LIGHTS were awesome , Vegas is a lot to take in if you have never been here before, this is my first VMworld and my first time to Vegas. Day 0 was a pretty straightforward day registration was the only thing opened for the first couple of hours but the exhibitor halls opened around 5. Walking into the exhibitor hall as overwhelming there was a lot of vendors with some pretty elaborate booths.

After the exhibit closed I headed over to the VMUG party at the house of blues in Mandalay bay, the band that was playing was great playing a mix of old school feel good 80’s to more modern tunes. really a good mix of songs. Then all of the sudden Michael Dell comes out. and in my opinion is was a classy move. To me it showed that he was taking community into consideration and was willing to talk about the future of VMware and the community directly to the people looking for that kind of reassurance. Real classy Michael Dell.

Oracle licensing and VMware

I was talking to a DBA the other day about oracle licensing, so I decided to write a post about it. Not only because it is confusing as hell but it’s a good to know for any potential oracle implementations in the future. But before we begin a couple of terms we need to know and understand for any of this to make even the slightest of sense.

We are going to assume enterprise licensing will be used in this case.


Soft Partitioning-

Oracle definition- “Soft partitioning segments the operating system using OS resource managers. The operating system limits the number of CPUs where an Oracle database is running by creating areas where CPU resources are allocated to applications within the same operating system. This is a flexible way of managing data processing resources since the CPU capacity can be changed fairly easily, as additional resource is needed.” – As stated in http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf

 


Hard Partitioning-

Oracle definition- “Hard partitioning physically segments a server, by taking a single large server and separating it into distinct smaller systems. Each separated system acts as a physically independent, self-contained server, typically with its own CPUs, operating system, separate boot area, memory, input/output subsystem and network resources.” – As stated in http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf


Now if you had a virtual machine that was running an oracle database you would assume that the partitioning method that would be used is soft partitioning. Because having the ESXi software make software partitions out of a physical socket .

BUT this is not the case. There is a key phrase in that definition that really stands out.

“The operating system limits the number of CPUs where an Oracle database is running by creating areas where CPU resources are allocated to applications within the same operating system.”

Since assigning a vCPU and vCores to a machine doesn’t limit that particular VM to only use only those resources  that’s one reason why oracle does not consider VMware a soft partitioning product.  (Unless you have CPU affinity enabled)

If you had a box with 4 physical sockets and 8 cores per socket that VM will be able to use any of the 32 cores. Oracle has a simple formula to offer a good insight into how much licensing will cost for a database, it looks something like this.

# of sockets*# of cores*CPU factor*License

So let’s say for a 4 socket intel Xeon system with 8 cores per socket and enterprise licensing. CPU factor chart will be found here http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf

4*8*.5*$47,500= $760,000 Per VM Per host.

That is a lot of money! But it gets more complicated than that. Say this particular VM is in a 3 host cluster. When you introduce VMware technologies like DRS and vMotion is that considered partitioning technology? So does the cluster need to be licensed? There has been more discussion on this in the IT community than there has been about aliens in Area 51. But the short answer is yes. if you have a 3 node cluster with the same specs stated above you will need to have all of those physical machines licensed. Any CPU that the oracle server can touch ( or potentially touch) needs to be licensed under the oracle license agreement.

Here might be a couple of solutions to this issue.

The first solution is to make your own cluster for Oracle database. Make it a 2 node cluster that in the event of a host failure you have it moved to another node. Spec out the physical host configurations to match exactly what you need and not over power it. If you need a 2 socket 4 core setup then just buy 2 of those. Don’t over provision!

The second solution might be to set up CPU affinity. Set the database to only use one particular CPU in the host and make sure it stays there. Now there are obvious downsides to this, one failing over would not happen. This practice has a lot of debate surrounding it. and definitely use it at your own risk. There has been mixed results from Oracle and the team of lawyers they employ.

And the third solution is probably the most painful. Pay Oracle. It’s simple and feels like you need a shot of penicillin after but it is the easiest solution.

-Resources and sources-

http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf

http://www.oracle.com/us/corporate/pricing/technology-price-list-070617.pdf

Thanks for reading!

I try my best to give the most accurate information possible! But I don’t know everything. If you found something I said not accurate let me know!

 

 

VMWorld content builder and schedule

If you registered for VMWorld 2016 you might have gotten an email telling you about the content builder for the breakout sessions and workshops that are going to be available.

This is my first year at VMworld and I personally think it is an awesome idea. having enough time to go through the list of sessions and planning your days out weeks in advance is an awesome gift that VMware has given us! With enough time to carefully take a look at the available activities throughout the week there shouldn’t be anyone running mad around the Mandalay bay looking for breakout sessions.

But  don’t wait to long! If you do the sessions might fill up and you will be put on a waitlist for what seems like forever. This happened to me with a couple of sessions I was really looking forward to, I didn’t pull the trigger and now I might be left out in the cold.

Most of the session I scheduled are for some upcoming projects I am hoping to work on. But I also included a lot of sessions for just general knowledge of new VMware products that are coming down the pipe like container based OS and hadoop with VMware

 

Check out my schedule below, and let me know if we share any sessions!

Vmworld Schedule

 

VMFS deprecated Bug ESXI 6.0 Update 1

So if you recently added a new datastore to you vcenter you might have gotten a nice little alarm on the ESXi hosts you added the datastore to, something like below.
Deprecated VMFS

This little gem above is a bug that is included in your ESXi version 6.0 update 1. While it’s a pain to fix it is pretty easy solution and doesn’t actually mean you have deprecated VMFS volumes on your host. But just to make sure check anyway and make sure this bug applies to you.

The solution is pretty simple by restarting two management agents on each affected ESXi the error will clear, of course until you add a new datastore. Then chances are the error will reappear. If you have a massive amount of ESXi hosts and the option is available to update to ESXi 6.0U2 then I personally would update instead of restarting these agents on all your hosts.


Step 1: Log into your ESXi host via SSH or DCUI

Step 2: Run the following command to restart the management agents on the box

services.sh restart

ssh

This command will reset the management agents on the ESXi host and clear the deprecated VMFS tag that has plagued your vcenter.


Kb article is below for more from VMware.

VMFS deprecated datastore 2109735

VPostGres database filling up VCSA 5.5 U2

pre cleanup vcenterRan into an interesting issue the other day my VPostgres database was at 100% capacity. Doing some quick investigation I realized that the postgres log was filled by a warning that would display multiple times a second causing a slow leak in available storage.

Warning causing database pileup

This would happen hundreds of times an hour and slowly kill the remaining storage on the VPostgres. Here is how i fixed it below.


1.Log into the VCSA and find the error above. You can find this error in the following directory in the VCSA if you are using SSH.

#~ cd /storage/db/vpostgres/pg_log

This will contain a lot of the files with the naming convention “Postgresql-year-date


2. Stopping the services on the VCSA and database is the next step before continuing on. If the Postgres is filled to capacity there is no need to stop the postgres services. These commands should be ran to prevent any problems in the shutdown process and will help to make sure other problems don’t arise from services not started properly.

Run the command: service vmware-vpxd stop

stopvcenter

Then run the command : Service vmware-vpostgres stop

Postgres stop

In the above example the services didn’t stop properly because my lab Vcenter was already filled to capacity and the services was already stopped.


3. While you figured out that the issue is this warning that keeps popping up the next step is to go into the postgres configuration file and change the logging levels. I used WINSCP to go into the VCSA to find the configuration file but you can use whatever utility you would like.

But make sure before you change this configuration file you make a backup on the VCSA to rollback if needed. You can also make a snapshot to make sure all changes can be rolled back (Only if this Vcenter is managed by another Vcenter, Obviously you cannot take a snapshot of the vcenter that wont turn on)

Use the command below in your SSH session to make a copy of the configuration file we will use in case of rollback.

cp /storage/db/vpostgres/postgresql.conf /storage/db/vpostgres/postgresql.conf.orig

 

Logging into Winscp to find the configuration file is easy, follow the same path as above and open it up in your favorite text editor.

There is a line of code in the configuration file on-line 312: #log_min_messages = warning

What we want to do is change this line to read like: log_min_messages = error

See below for an example. Ignore the | before the command.

Config file

Save the file and we can begin to start the services on the VCSA.


4. Now lets remove the old files from the database. This can be done from WinSCP of it can be done from the SSH console to the VCSA I can show you the site of both. I personally would keep the last 2-3 months worth of logs for when there was any valuable information in there.

Run the command: RM /storage/db/vpostgres/pg_log/postgres-DateYouWanttodelete

remove old postgres

A much easier way in my opinion is doing this through WinSCP go to the same file place above Ctrl-Select your choices you want to deleted then right-click and select delete.


5. Starting the services is the best part IMHO it means the problem is solved and the solution we just worked out hasn’t failed us.  Run the two commands to start the VCSA services and we should be on our way.

Start the Vpostgres service : service vmware-vpostgres start

Start the VPXD service on the VCSA: service vmware-vpxd start

You should receive a message like the one below.

Start services


6. Last but not least it is time to see the finished result. though SSH run the following command: df -h

This command should spit out the list of your drives and capacity of those drives.

Or if you want a prettier look log into the vcsa appliance webpage and check out the dashboard

Https://VCSA:5480after database cleanup

Vmware KB article 2092127

 

Hope you enjoyed. Comment below for any critiques.

Virtual SAN 6.2 -Technical What’s New (Boston VMUG)

This was an awesome session that explained the new features coming with the 6.2 update for vSAN a couple of improvements.

  1. The first of which is QOS this is a great feature that was added to VSAN. This gives the user inside information on the IOPs that are being used per VMDK including ,setting limits on the individual VMDK, a truly great feature. GONE ARE THE DAYS OF IOP HOGS ON VSAN! Now we can control and part out IOPS to the most used VM’s on the cluster without having to worry about performance hits.
  2. VSAN dedupe and compression: this is a must have feature that just got introduced for VSAN 6.2 the ability to dedupe and compress your data keeps cost per GB down significantly.
  3. Software checksum for VSAN: the ability to check and remediate corruption on your VSAN is obviously important. And this will help and do a great job on making sure there is consistency on your data and it is corruption free while using VSAN 6.2
  4. Probably the most important of these features added are the performance Service. Now we can store and keep historical VSAN metrics on an independent database away from Vcenter for when there is issues. There is a host of other features also including cluster host and VM metrics that are tied to the VSAN performance services. Giving us an in-depth look into the capabilities of our VSAN and were potential problems will arise from.

Ontop of this there is a new health services feature that will continually check the health of the VSAN cluster and make sure nothing is out of whack! What a great tool and awesome improvement to VSAN!

 

Check out the VMware release PDF below

https://www.vmware.com/files/pdf/products/vsan/VMware_Virtual_SAN_Whats_New.pdf

Search function not working!

emptySEARCH

 

 

 

 

So ran into an issue after an upgrade from 5.5 to 6.0 U1 EP3. Search function in VCenter would not work properly and would show as an empty inventory in Web client when logged in under a domain authenticated account.

There can be two solutions to this issue.

One solution is to restart the inventory services on your VCenter server or appliance. You can find these solutions in the KB articles below

Restarting VCenter services for windows

Restarting VCenter services for VCSA

But for my case this didn’t work. I even rebooted the entire VCenter to try to get a fresh start. For me the problem was a little bit different it seems.

The issue here seems to be the Identity resource on SSO got out of sync with the domain.While administrator@vsphere.local had the search function working and the web services functioning completely, I was a puzzled. After a quick 10 minute call to VMware the gentleman on the other side recommended we recreate the SSO identity source on the VCenter.

I took a screen shot of my SSO identity settings and took a deep breath while VCenter did its thing. Surprisingly, This seemed to fix the issue. Once the new identity source was put into place the search function returned and I could get back to business! From what i was told BY the tech at VMware this seems to be an issue they noticed often with upgrades. not really a bug of sorts but a hiccup in the upgrade process.

Documentation from VMware below.

VMware 6.0 documentation on SSO identity source

 

Virtualization and technology blog