It’s been a long time since last post, I was pretty busy on a storage issue, I did a lot of work with hardware vendor and VMware for this weird issue.
During our troubleshooting, I noticed a minor problem when I try search VM in vSphere Client, everytime it gave me error message “Unable to connect to web services to execute query“, it requested me “Verify that the VMware VirtualCenter Management Webservices service is running”
I tried to reboot vCenter Server, restart Management webservices and even re-installed vSphere Client, no lucky….Finally I fixed the problem by following step:
- Stop VMware VirtualCenter Management Webservices service on vCenter Server.
- Backup Data folder in C:Program FilesVMwareInfrastructuretomcatwebappssmsWEB-INFclassescomvmwarevimsms.
- Remove all sms-*.db files in Data folder.
- Restart VMware VirtualCenter Management Webservices service.
It’s simple steps to fix the problem, but this issue confused me and VMware support for a long time. This problem appeared after we upgraded vCenter Server from 5.0 to 5.1, first thing we suspected was inventory services, error message below was logged in ds.log when we searched VM.
[2013-05-25 12:04:31,995 http-nio-/0.0.0.0-10443-exec-634 INFO com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet] Sending security error because of exception : com.vmware.vim.vcauthenticate.exception.SsoUnreachableException: com.vmware.vim.dataservices.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.sso.admin.exception.InternalError: General failure.
It looks like a authentication issue, right? So we checked SSO, service account…etc. The unclearly logs lead to a wrong way. 🙂
Since nobody complained to me, I suspected that’s a client side issue, then we tried search on another purge client but same issue. We also suspected the cache of vCenter inventory, but logs didn’t evidence it is, we cannot just reset inventory cache database since that’s production environment!
Okay, I talk too much about troubleshooting process, let’s talk about the search function of vSphere, my understood is vCenter search objects by two different way: Web Client or vSphere Client. It looks like Web Client retrieve data from database or Web Client server.
vSphere Client get data from cache database. The cache database is located in vCenter Server install folder, default path is C:Program FilesVMwareInfrastructuretomcatwebappssmsWEB-INFclassescomvmwarevimsms. the cache file is actually H2 databases, it work together with Tomcat web services, sms folder contains application files of Storage Monitoring Services, it use H2 database engine v1.2.147. Please comments if you think I’m wrong.
If the H2 database incorrupt, storage monitoring services also stop working, you can find the service in Service initializing… status with warning status in vCenter Service Status node of vSphere Client.
One solution fix two issue, I like it!