VPostGres database filling up VCSA 5.5 U2

pre cleanup vcenterRan into an interesting issue the other day my VPostgres database was at 100% capacity. Doing some quick investigation I realized that the postgres log was filled by a warning that would display multiple times a second causing a slow leak in available storage.

Warning causing database pileup

This would happen hundreds of times an hour and slowly kill the remaining storage on the VPostgres. Here is how i fixed it below.


1.Log into the VCSA and find the error above. You can find this error in the following directory in the VCSA if you are using SSH.

#~ cd /storage/db/vpostgres/pg_log

This will contain a lot of the files with the naming convention “Postgresql-year-date


2. Stopping the services on the VCSA and database is the next step before continuing on. If the Postgres is filled to capacity there is no need to stop the postgres services. These commands should be ran to prevent any problems in the shutdown process and will help to make sure other problems don’t arise from services not started properly.

Run the command: service vmware-vpxd stop

stopvcenter

Then run the command : Service vmware-vpostgres stop

Postgres stop

In the above example the services didn’t stop properly because my lab Vcenter was already filled to capacity and the services was already stopped.


3. While you figured out that the issue is this warning that keeps popping up the next step is to go into the postgres configuration file and change the logging levels. I used WINSCP to go into the VCSA to find the configuration file but you can use whatever utility you would like.

But make sure before you change this configuration file you make a backup on the VCSA to rollback if needed. You can also make a snapshot to make sure all changes can be rolled back (Only if this Vcenter is managed by another Vcenter, Obviously you cannot take a snapshot of the vcenter that wont turn on)

Use the command below in your SSH session to make a copy of the configuration file we will use in case of rollback.

cp /storage/db/vpostgres/postgresql.conf /storage/db/vpostgres/postgresql.conf.orig

 

Logging into Winscp to find the configuration file is easy, follow the same path as above and open it up in your favorite text editor.

There is a line of code in the configuration file on-line 312: #log_min_messages = warning

What we want to do is change this line to read like: log_min_messages = error

See below for an example. Ignore the | before the command.

Config file

Save the file and we can begin to start the services on the VCSA.


4. Now lets remove the old files from the database. This can be done from WinSCP of it can be done from the SSH console to the VCSA I can show you the site of both. I personally would keep the last 2-3 months worth of logs for when there was any valuable information in there.

Run the command: RM /storage/db/vpostgres/pg_log/postgres-DateYouWanttodelete

remove old postgres

A much easier way in my opinion is doing this through WinSCP go to the same file place above Ctrl-Select your choices you want to deleted then right-click and select delete.


5. Starting the services is the best part IMHO it means the problem is solved and the solution we just worked out hasn’t failed us.  Run the two commands to start the VCSA services and we should be on our way.

Start the Vpostgres service : service vmware-vpostgres start

Start the VPXD service on the VCSA: service vmware-vpxd start

You should receive a message like the one below.

Start services


6. Last but not least it is time to see the finished result. though SSH run the following command: df -h

This command should spit out the list of your drives and capacity of those drives.

Or if you want a prettier look log into the vcsa appliance webpage and check out the dashboard

Https://VCSA:5480after database cleanup

Vmware KB article 2092127

 

Hope you enjoyed. Comment below for any critiques.

Virtual SAN 6.2 -Technical What’s New (Boston VMUG)

This was an awesome session that explained the new features coming with the 6.2 update for vSAN a couple of improvements.

  1. The first of which is QOS this is a great feature that was added to VSAN. This gives the user inside information on the IOPs that are being used per VMDK including ,setting limits on the individual VMDK, a truly great feature. GONE ARE THE DAYS OF IOP HOGS ON VSAN! Now we can control and part out IOPS to the most used VM’s on the cluster without having to worry about performance hits.
  2. VSAN dedupe and compression: this is a must have feature that just got introduced for VSAN 6.2 the ability to dedupe and compress your data keeps cost per GB down significantly.
  3. Software checksum for VSAN: the ability to check and remediate corruption on your VSAN is obviously important. And this will help and do a great job on making sure there is consistency on your data and it is corruption free while using VSAN 6.2
  4. Probably the most important of these features added are the performance Service. Now we can store and keep historical VSAN metrics on an independent database away from Vcenter for when there is issues. There is a host of other features also including cluster host and VM metrics that are tied to the VSAN performance services. Giving us an in-depth look into the capabilities of our VSAN and were potential problems will arise from.

Ontop of this there is a new health services feature that will continually check the health of the VSAN cluster and make sure nothing is out of whack! What a great tool and awesome improvement to VSAN!

 

Check out the VMware release PDF below

https://www.vmware.com/files/pdf/products/vsan/VMware_Virtual_SAN_Whats_New.pdf

Search function not working!

emptySEARCH

 

 

 

 

So ran into an issue after an upgrade from 5.5 to 6.0 U1 EP3. Search function in VCenter would not work properly and would show as an empty inventory in Web client when logged in under a domain authenticated account.

There can be two solutions to this issue.

One solution is to restart the inventory services on your VCenter server or appliance. You can find these solutions in the KB articles below

Restarting VCenter services for windows

Restarting VCenter services for VCSA

But for my case this didn’t work. I even rebooted the entire VCenter to try to get a fresh start. For me the problem was a little bit different it seems.

The issue here seems to be the Identity resource on SSO got out of sync with the domain.While administrator@vsphere.local had the search function working and the web services functioning completely, I was a puzzled. After a quick 10 minute call to VMware the gentleman on the other side recommended we recreate the SSO identity source on the VCenter.

I took a screen shot of my SSO identity settings and took a deep breath while VCenter did its thing. Surprisingly, This seemed to fix the issue. Once the new identity source was put into place the search function returned and I could get back to business! From what i was told BY the tech at VMware this seems to be an issue they noticed often with upgrades. not really a bug of sorts but a hiccup in the upgrade process.

Documentation from VMware below.

VMware 6.0 documentation on SSO identity source

 

Security flaw in Client integration plugin found

So today I found a little gem that kind of concerned me. VMware placed an article in late April concerning the client integration plugin on a lot of products was vulnerable to man in the middle attacks and also hijacking the web session. Kind of a big deal if you don’t want your environment being hijacked by some Russian hijackers.

Below i have attached some articles.

 

https://www.vmware.com/security/advisories/VMSA-2016-0004

http://securityaffairs.co/wordpress/46333/security/vmware-client-integration-plugin.html