Welcome to our last discussion on the VMware vSphere 5.0 Hardening Guide of the week. Here are links to the past articles from the week: one, two, three, and four.

• Guideline Title: Verify contents of exposed configuration files
  • Title: verify-config-files
  • Discussion: Certain configuration files exist on ESXi hosts that govern host behavior and operations. These files should be logged and monitored for both authorized and unauthorized configuration changes. These files can be retrieved over HTTPS via http://<hostname>/host if the Managed Object Browser (MOB) is enabled. However, a separate VMware hardening recommendation we’ve previously covered advises that the MOB be disabled. If your organization chooses not to accept the risk of leaving the MOB enabled, these configuration files can also be retrieved via vCLI or PowerCLI.
  • Official VMware documentation
• Guideline Title: Keep ESXi system properly patched
  • Title: apply-patches
  • Discussion: ESXi is designed from the ground up to be a powerful but secure hypervisor with minimal attack surface area. A complete install disc is less than 300MB. ESXi needs patches much less frequently than ESX used to, but it does still need them. VMware Update Manager is a free tool included with vCenter to help automate the patching of ESXi hosts during production hours with no VM downtime. To stay on top of the latest VMware Security Advisories by email you can also subscribe here.
  • Official VMware documentation

Note: I’m collapsing the next three hardening guide checks into a single entry since they are almost identical. I will point out in the discussion section where they differ.

• Guideline Title: Verify Image Profile and VIB Acceptance Levels
  • Title(s): (1) verify-acceptance-level-certified, (2) verify-acceptance-level-accepted, (3) verify-acceptance-level-supported
  • Discussion: vSphere Installation Bundles (VIBs) are files that can be used to extend ESXi functionality. They might perform functions such as enabling hardware status monitoring, adding new hardware drivers, or enabling third-party security virtual appliances. These VIBs can have one of four available acceptance levels: VMwareCertified, VMwareAccepted, PartnerSupported, and CommunitySupported. As their names imply, these four levels relate to the entity that tested and possibly certified the VIBs for use. When you configure the VIB Acceptance Level, you are instructing your ESXi hosts to only install VIBs that meet or exceed the specified level of support and testing.
    • -VMwareCertified – these VIBs are created, tested, and signed by VMware. This is the recommended setting for environments hosting extremely sensitive data, including military environments authorized to handle classified data.
    • -VMwareAccepted – These VIBs are created by a VMware Partner but are tested and signed by VMware. This is the recommended setting for environments hosting sensitive data or those subject to stricter compliance requirements.
    • -PartnerSupported – These VIBs are created, tested, and signed by a certified VMware Partner. This is the recommended setting for all vSphere environments.
    • -CommunitySupported – These VIBs are neither supported nor digitally signed. CommunitySupported VIBs should not be installed on production vSphere environments.
  • Official VMware documentation

• Guideline Title: Disable SSH
  • Title: disable-ssh
  • Discussion: We all know and love SSH. It’s an encrypted way to access a command line on a remote system. But the truth is that a stopped service is infinitely harder to exploit than a running service. SSH should only be enabled as necessary for troubleshooting and then should be stopped again.
  • Official VMware documentation Read more >>

If you’re just joining us, I’m blogging my way through the VMware vSphere 5.0 Hardening Guide. The preceding posts, one and two, are also available. Today’s post discusses self-signed certificates, the DCUI console, and the memorable ESXi shell. Let’s dive in!

• Guideline Title: Enable SSL for NFC
  • Title: enable-nfc-ssl
  • Discussion: NFC (Network File Copy) is used to copy or migrate VMs between ESXi hosts. The vulnerability being addressed here is network traffic sniffing. If an attacker sniffed NFC traffic as it was being used to ship an entire VM, they would get an effective duplicate of your entire virtual server. Normally turning SSL on here would be an easy decision to recommend, but in this case, VMware’s guidance on this issue is that NFC over SSL between ESXi hosts “has … not been extensively tested and so may cause HA and other operations to fail in certain circumstances.” Ouch. We’ll keep our eye on this one and hope that NFC over SSL gets more testing in the future. In the meantime, make sure your vSphere management networks and VLANs are sufficiently isolated as a partial mitigation.
  • Official VMware documentation
• Guideline Title:  Do not use default self-signed certificates for ESXi communication
  • Title: esxi-no-self-signed-certs
  • Discussion: This hardening guidance is written towards vSphere, but it’s always good guidance for any system that uses encryption, even plain ol’ Windows Remote Desktop! Wikipedia has a detailed explanation of man-in-the-middle attacks, but here’s the gist: whenever you choose to connect to a server with an unverified SSL certificate, you have no certainty that your encrypted SSL tunnel reaches all the way to the server you’re talking to. It’s possible that your traffic is being intercepted, decrypted, stored, re-encrypted, and sent to its destination – all without you or the server you’re connecting to being aware your traffic was intercepted. Self-signed SSL certs are generic and hard to distinguish from each other. They also lose the benefit of having a Certificate Authority confirm the validity of the cert. The solution is always replace default/vendor certificates on all network endpoints with certs issued by a commercial or organizational Certificate Authority.
  • VMware Official documentation
• Guideline Title: Prevent unintended use of dvfilter network APIs
  • Title: verify-dvfilter-bind
  • Discussion: The dvfilter network APIs are part of the larger VMsafe API and they allow VMware partners to develop virtual security appliances to control the behavior of guest virtual machines. These virtual appliances have the benefit of simplifying guest VM configuration while increasing the overall security of your environment. Some great examples of this include Trend Micro Deep Security, the Cisco Virtual Security Gateway, and the Juniper Virtual Gateway. These appliances allow you to manage firewalls around your VMs as easily as dragging and dropping VMs in vCenter, enabling secure VM isolation (e.g. “DEV servers can’t talk to PROD servers”) but without sacrificing any ease of management (one of the reasons we virtualized everything in the first place, right?) These virtual security appliances using the dvfilter APIs receive privileged access to other guest VM network traffic. If you’re not using any security appliances that leverage these APIs, you’ll want to confirm they’re not in use to prevent a malicious VM from receiving the data instead.
  • Official VMware documentation is still TBD
• Guideline Title: Disable DCUI to prevent local administrative control
  • Title: disable-dcui
  • Discussion: NOTE: This is only recommended by VMware for the most secure environments. The DCUI is the name of the service that runs the gray and yellow console screen we all see when we first install ESXi and then promptly forget exists. This service can be stopped (and configured to not auto-start) which prevents any operations from occurring at the console or over a KVM. Be very careful and know the impact of this change. If you misconfigure a vSwitch and knock your host management kernel port offline, there’s no parachute to reset the host vSwitch and you might be reaching for an ESXi install disk.
  • Official VMware documentation is still TBD
• Guideline Title: Disable ESXi Shell unless needed for diagnostics or troubleshooting
  • Title: disable-esxi-shell
  • Discussion: This is known by some as the ESXi command line, formerly known as Tech Support Mode. From an ESXi console, you can press Alt+F1 and login as root to access a BusyBox environment that looks and acts a bit like Linux but has full control over host operations. The ESXi Shell should be started as necessary through the vSphere console for troubleshooting purposes and remain stopped the rest of the time.
  • Official VMware documentation

Welcome back to part 2 of our discussion of the vSphere 5.0 Hardening Guide! If you missed part 1, you can find it here. Let’s get back to it!

• Guideline Title: Communication Configure the ESXi host firewall to restrict access to services running on the host
  • Title: config-firewall-access
  • Discussion: The process of hardening a production server involves stopping unnecessary services and protecting services that need to remain running, and ESXi is no different. Unnecessary services should be disabled. Necessary services can be protected by defining allowed IPs that are allowed to connect. These IPs can be set under “Configuration -> Security Profile -> Firewall -> Properties”. Select a service then click “Firewall”, choose “Only allow connections from the following networks” and specify the IPs or ranges that will be allowed to connect.
    
  • Official VMware documentation
• Guideline Title: Configure NTP time synchronization
  • Title: config-ntp
  • Discussion: NTP serves two purposes. First,  in the event of a security breach, having your ESXi system logs synchronized will lend itself to an effective incident postmortem, which can, in turn, help increase the security of your environment and further remediate vulnerabilities. Second,  NTP will be used by any VMs that utilize the VMware Tools time synchronization. When the host has accurate time, the VMs using it as a time source will also. I prefer to pick a single device inside the firewall (possibly the firewall itself) to serve as my internal NTP source. Then I have all my other hosts sync against that device. This means that if the internet connection goes down, only one system notices and the rest of the systems all stay in sync with each other, though not necessarily with the rest of the world. So set an NTP server choose ”Configuration -> Time Configuration -> Options -> start the NTP service and select “Start and stop with host”". Click ‘Add’ and specify your site’s NTP servers.
  • Official VMware documentation
• Guideline Title: Ensure proper SNMP configuration
  • Title: config-snmp
  • Discussion: SNMP is a protocol for monitoring network resources (hosts, servers, switches, etc). If your environment is not using SNMP, it should be disabled to avoid disclosing ESXi host status information to malicious observers. (Of course, if you’re following along, you’ve already locked down which IPs and ranges can reach the SNMP agent!) The ESXi SNMP agent is disabled by default. To confirm this, run “vicfg-snmp <conn_options> –show”, or to disable it in environments where it is not used, run “vicfg-snmp <conn_options> –disable”. If SNMP is being used, make sure the parameters are properly configured using either vCLI or PowerCLI, or using an API client.
  • Official VMware documentation
• Guideline Title: Disable Managed Object Browser (MOB)
  • Title: disable-mob
  • Discussion: I can’t put it any better than the VMware-provided description for this vulnerability already has. I’ll quote it below directly from the Hardening Guidelines.
    • “The managed object browser (MOB) provides a way to explore the object model used by the VMkernel to manage the host; it enables configurations to be changed as well. This interface is meant to be used primarily for debugging the vSphere SDK but because there are no access controls it could also be used as a method obtain information about a host being targeted for unauthorized access. To determine if the MOB is enabled run the following command on the ESXi shell: “vim-cmd proxysvc/service_list”. To disable the MOB run the following command: “vim-cmd proxysvc/remove_service “/mob” “httpsWithRedirect”". Note: You cannot disable MOB while in lockdown mode. The MOB will no longer be available for diagnostics. Some 3rd party tools use this interface to gather information. Testing should be done after disabling the MOB to verify 3rd party applications are still functioning as expected. To re-enable the MOB: ~ # vim-cmd proxysvc/add_np_service “/mob” httpsWithRedirect”
  • Official VMware documentation
• Guideline Title: When adding ESXi hosts to Active Directory use the vSphere Authentication Proxy to protect passwords
  • Title: enable-auth-proxy
  • Discussion: In environments that use both ESXi Autodeploy and Active Directory to manage hosts, the password used to join Active Directory is sent in cleartext to each host as it boots up, which puts that password at great risk of interception. Instead, VMware provides a vSphere Authentication Proxy to help keep this communication secure. Here are the official deployment instructions.
  • Official VMware documentation

We’ll pick this back up tomorrow and knock out another 5 ESXi hardening checks!

VMware released the vSphere 5.0 Hardening Guide this month and I wanted to take the opportunity to blog my way through it in order to increase my familiarity, heighten its visibility, and provide a forum for discussion of the recommendations and methods to implement an assessment in your own environment.

Since the hypervisor is the heart of your datacenter, let’s start with the ESXi guidelines:

• Guideline Title: Use Active Directory for local user authentication
  • Title: enable-ad-auth
  • Discussion: The credentials to your ESXi hosts are among the most sensitive in your entire datacenter. The root password should only be given to personnel that have an explicit need for it. Other users that need to login to hosts outside of vCenter can have permissions granted to their Active Directory account. David Davis has a great video demonstrating the process. By default, an AD group names “ESX Admins” is checked for and all members will have full rights to all hosts.
  • Official VMware documentation
• Guideline Title: Establish a password policy for password complexity
  • Title: set-password-complexity
  • Discussion: This setting is used to enforce strong passwords for local accounts. You know, the local accounts you’re not using since all your hosts are joined to AD, right? But seriously, this is a great setting to use to force users to pick strong passwords, even if your hosts have no local users. Configuration drift happens, so let’s plan ahead and make sure these users will have strong passwords. Using vi, you’ll edit the file /etc/pam.d/passwd and modify the third line. The default value is “password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=8,8,8,7,6″. The retry value controls how many attempts users get to pick a strong password. The last 5 numbers control your desired complexity settings, in the following order and referencing four character classes (uppercase letters, lowercase letters, numbers, and symbols/special characters):
    • The first digit is the minimum length for passwords containing characters from 1 of the classes (least complex, e.g. using “password” as your password). You’ll want this to be a very large number.
    • The second digit is the minimum length for password containing characters from 2 of the classes. You’ll also want this to be a very large number.
    • The third digit is the minimum length of SSL certificate passphrases
    • The fourth digit is the minimum length for password containing characters from 3 of the classes. Now were getting into reasonable passwords, so allow whatever reasonable length your organization permits.
    • The fifth digit is the minimum length for password containing characters from 4 of the classes (most complex, e.g. using “p4$wOrd” as your password”). Same as the previous line.
  • Official VMware documentation
• Guideline Title: Verify Active Directory “ESX Admin” group membership.
  • Title: verify-admin-group
  • Discussion: now that our hosts are joined to AD (first bullet point above), we want to audit the membership of “ESX Admins” since those users will automatically have root-level permissions to our hosts. This is a manual check, but you’ll also want to automate it so that it can run regularly. You can write a scheduled Powershell script to read the group membership and email you the contents on a schedule, or products like ADAudit Plus from ManageEngine are perfect for this task and can email you only when group membership changes.
  • Official VMware documentation
• Guideline Title: Ensure that vpxuser auto-password change meets policy.
  • Title: vpxuser-password-age
  • Discussion: ESXi contains a built-in user account called vpxuser. This account has Administrator-level permissions to the local host and is used by vCenter to execute tasks. vCenter sets this password and changes it automatically every 30 days. If your organization requires more frequent password changes, you can change this from the vSphere client by selecting “Administration -> vCenter Server Settings -> Advanced Settings” and changing VirtualCenter.VimPasswordExpirationInDays to your desired interval.
  • Official VMware documentation
•  Guideline Title: Ensure that vpxuser password meets length policy.
  • Title:vpxuser-password-length
  • Discussion: Similar to the guideline above, this check confirms that the password length for the vpxuser password is long enough to comply with your organization’s security policy. By default it is set to 32 characters. To increase this password length, modify the “vpxd.hostPasswordLength” value in the vpxd.cfg file. On Windows this file is located at C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg  or on the vCenter Server Appliance you’ll find it at /etc/vmware-vpx/vpxd.cfg.
  • Official VMware documentation

That’s it for today! I’ll see you all back here tomorrow to continue our discussion on the vSphere 5.0 Hardening Guide.