vSphere with Tanzu and HPE Alletra dHCI¶

For more details and information on HPE Alletra dHCI deployment options, see HPE Nimble Storage dHCI and VMware vSphere New Servers Deployment Guide.

DNS & NTP

Ensure DNS forward and reverse lookups and NTP are configured and available.

Storage configuration¶

By default, the HPE Alletra dHCI deploys two basic datastores as shown in Figure 3.

Figure 3. Default datastores.

Create a vVols datastore on HPE Alletra dHCI.¶

A vVols datastore will need to be created for use by a Content Library, VM virtual disks and the deployment of vSphere with Tanzu Supervisor and guest Kubernetes clusters. This vVols datastore will also be used to provision persistent storage for the applications running in the Tanzu clusters.

1. Right click an object within vCenter and go to HPE Alletra 6000 and Nimble Storage Actions. Click Create VVol Datastore.
   
   
   
   

2. Choose the dHCI system in Group. Click Next.
   
   
   
   

3. Specify a Name for the datastore. Click Next.
   
   
   
   

4. Set a Space Limit. Click Next.
   
   
   
   

5. Set optional Performance Limits. Finally click Create.
   
   
   
   

At this stage, there should be a VMFS and vVols datastore available to deploy the various components needed for vSphere for Tanzu.

Deploying the Solution¶

Planning the environment¶

vSphere with Tanzu Solution deployment requirements¶

Storage Policies and tags¶

vSphere with Tanzu leverages the vSphere CSI Driver to provide storage to Tanzu Kubernetes Clusters. The vSphere Container Storage Interface (CSI) Driver supports both policy and tag based Storage Policy-Based Management (SPBM) to provision volumes. SPBM is a feature of VMware vSphere that allows an administrator to match VM workload requirements against storage array capabilities, with the help of VM Storage Policy. This storage policy can expose multiple array capabilities and data services to the Tanzu Kubernetes clusters, depending on the underlying storage you use. Using Storage Policies is preferred for the Tanzu Supervisor and Kubernetes nodes as each VM will have its volumes directly on the array. vVols (and VMFS tag-based) policies assigned to Tanzu Kubernetes clusters in StorageClasses, provide persistent storage for container-based applications allowing developers to tailor storage to their application needs.

Tags allow you to attach metadata to objects in the vSphere inventory to make it easier to sort and search for these objects. A tag is a label that you can apply to objects in the vSphere inventory. With vVols, the VASA provider reports the capabilities and services that a vendor's storage can provide. For VMFS, with tag-based SPBM, you can create your own specific categories and tags based on almost anything you can envision. Performance levels or tiers, disk configurations, locations, OS type, departments, and disk types such as SAS, SATA or SSD are just a few examples.

Creating vVols Storage Policy¶

For our Supervisor Cluster and namespaces, we need to create a vVols-based storage policy that will control the storage placement of objects like control plane VMs, container images, and persistent storage volumes.

Go to Menu, Policies and Profiles.

Highlight VM Storage Policies, click Create.

Specify a Name of the policy. Click Next.

Click Enable rules for "HPE Alletra 6000 and Nimble Storage" storage. Click Next.

To configure rules for the HPE Alletra 6000, click Add Rule.

Configure the rules to your application needs. Click Next.

Verify that the datastore is showing as compatible. Click Next.

Review and click Finish.

Create Subscribed Content Library¶

vSphere with Tanzu requires a subscribed Content Library to download the latest Tanzu Kubernetes Grid cluster images that will be used to deploy the Tanzu Supervisor and Kubernetes clusters.

Click on Menu then Content Libraries.

Under Content Library, Click Create.

Specify a Name for the Content Library. Click Next.

Choose Subscribed content library.

Enter the following Subscription URL: http://wp-content.vmware.com/v2/latest/lib.json Click Next.

Choose the default options on Apply security policy. Click Next.

Choose the vVols datastore to store the Content Library objects. Click Next.

Review the setting and click Finish to create the Content Library.

Create Local Content Library¶

We will also create a local Content Library for OS images, OVAs, etc. The process is the same as above.

Click on Menu then Content Libraries.

Under Content Library, Click Create.

Specify a Name for the Content Library. Click Next.

Choose Local content library. Click Next.

Choose the default options on Apply security policy. Click Next.

Choose the vVols datastore to store the Content Library objects. Click Next.

Review the setting and click Finish to create the Content Library.

vSphere with Tanzu User management¶

The vSphere with Tanzu platform involves two roles, the vSphere administrators and the DevOps engineer. It is important to configure proper role(s) and permissions within vCenter Single Sign-On for the users to interact with the platform. Please refer to vSphere with Tanzu User Roles and Workflows.

Creating a DevOps user¶

The DevOps user will be a consumer (non-vSphere admin) of the assigned vSphere namespace using the kubectl CLI. This user may or may not require access to the vCenter UI.

To create the devops@vsphere.local user:

Click Menu, Administration

Under Single Sign On, click Users and Groups.

In the Users and Groups, choose the Domain vsphere.local from the drop-down menu. Click Add.

Create the devops user, set the Password and click Add.

After the devops@vsphere.local user is created, it will be assigned to the vSphere namespace at a later step.

vSphere with Tanzu IP requirements¶

You will need three separate, routable subnets configured. One subnet will be for the vSphere with Tanzu Management network. This is the network where vCenter, the NSX Advanced Load Balancer cluster, and the Supervisor Cluster will live. The other subnets will be used for the Frontend and Workload networks.

Each network will need to be on separate vSphere Distribute Switch port groups.

Management Network The Management Network is where the vCenter, ESXi hosts, Supervisor Cluster and NSX Advanced Load Balancer are deployed.

Frontend Network External clients (such as users or applications) accessing cluster workloads use the Frontend network to access backend load balanced services using virtual IP addresses.

Workload Network This network is used by the load balancer to access the Kubernetes services on the Supervisor and Guest clusters.

Install and Configure the NSX Advanced Load Balancer¶

Deploy Controller VMs¶

To get started deploying the NSX Advanced Load Balancer, download the OVA from VMware.

You can access it from the products download page.

Once downloaded, upload the OVA to the Images content library we created earlier.

Go to Menu, click Content Library. Right click the Images content library and choose Import. This may take a few minutes to upload.

We will be deploying NSX Advanced Load Balancer in a 3-node cluster to provide high availability. We configure the cluster in a later step.

Navigate to the Images content library. Choose OVF & OVA Templates, right click on the controller OVA and click New VM from This Template...

Specify the VM name and location. Click Next.

Select the Cluster. Click Next.

Confirm settings and click Next.

Select Storage. Click Next.

Under Management Network, select the appropriate management network to be used. Example: dhci_tanzu-management-net Click Next.

• Specify the Management IP Example: 192.168.22.185
• Set the Subnet Mask Example: 255.255.248.0
• Set the Gateway Example: 192.168.20.1

Leave the other fields blank. Click Next.

Confirm settings and click Finish.

Repeat these steps to deploy all nodes for the cluster.

Once all nodes are deployed, power on each node. It can take up to 10 minutes for the nodes to be in a Ready state.

Initialize Controller nodes¶

In a browser, enter the IP for the first node in the cluster. Example: https:\\192.168.22.185 Set a username and password. Click Create Account.

In the Welcome screen,

• Set Passphrase Example: VMware1!
• Set DNS server(s) Example: 192.168.20.5, 192.168.20.6
• Set DNS Search Domain Example: hpelab.local

Click Next.

Configure an email address. Click Next.

Multi-tenant, keep Defaults.

Check Setup Cloud After box.

Click Save.

Add a License¶

From the Controller dashboard, click the Administration tab.

Expand Settings, select Licensing. Click the gear icon next to Licensing.

Choose Essentials Tier. Click Save.

Configure the Default Cloud¶

From the Controller dashboard, click the Infrastructure tab, select Clouds.

Click the Convert Cloud Type gear icon on Default-Cloud.

In the dropdown menu, choose VMware vCenter/vSphere ESX Cloud Type. Click Yes, Continue.

Enter the vCenter Address, Username and Password. Keep all other settings default. Click Next.

Choose the Data Center. Click Next.

Select the Management Network, specify the IP Subnet and Default Gateway. Example: dhci_tanzu-management-net

Enter the Static IP Address Pool. Example: 192.168.22.191-192.168.22.200

These are IPs that will be assigned to Service Engines deployed on the Management Network.

Click Save.

Configure Cluster¶

Configuring a cluster is recommended in production environments for high availability and disaster recovery.

From the Controller dashboard, click the Administration tab, select Controller > Nodes.

Click Edit.

Add a static IP for the Controller Cluster IP. Example: 192.168.22.190 (This IP must be available on the Management Network.)

For each Controller node to be added to the cluster, click Add.

Enter the IP Address, Name and Password for the admin user for each node.

Click Save.

Once all nodes are added to the cluster, click Save.

After a few minutes, the NSX Advanced Load Balancer will be reconfigured as a cluster.

Refresh the browser and navigate to the Cluster IP to continue with the configuration. Example: https://192.168.22.190

Configure a Certificate to the Controller¶

The Controller must send a certificate to clients to establish secure communication. This certificate must have a Subject Alternative Name (SAN) that matches the NSX Advanced Load Balancer Controller cluster hostname or IP address.

The Controller has a default self-signed certificate. But this certificate does not have the correct SAN. You must replace it with a valid or self-signed certificate that has the correct SAN.

From the Controller dashboard, click the Templates tab, select Security in the left-hand menu.

Select SSL/TLS Certificate. Click Create > Controller Certificate.

Enter a Certificate Name. Example: ALB Tanzu Cert

Enter the Controller IP address in Common Name. Example: 192.168.22.190

Select either EC or RSA. EC is recommended.

Under Subject Alternate Name (SAN), click Add and enter the Controller IP. Example: 192.168.22.190

Click Save

This certificate will be used to deploy the Supervisor Cluster when configuring Workload Management.

Once the certificate has been created, click Export Certificate.

In the Export Certificate window, click Copy to Clipboard under the Certificate section. Do not use the certificate key.

Finally, we need to update the portal certificate to use the newly created certificate.

From the Controller dashboard, click the Administration > Settings.

Select Access Settings, click the pencil to update settings.

Check the box Allow Basic Authentication.

Under SSL/TLS Certificate, remove any existing certificates. Click the drop down and select the new certificate we created in previous steps. Example: ALB-Tanzu-Cert

Keep the rest of the settings Default.

Click Save.

Configure a Service Engine Group¶

From the Controller dashboard, click the Infrastructure > Cloud Resources. Edit the Default-Group.

In the High Availability & Placement Settings section, select the High Availability Mode. The default option with the Essentials license is Active/Standby.

Click the Advanced tab.

Under Host & Data Store Scope section, Select Cluster > Include and choose the vSphere Cluster. Example: dHCI-Cluster

Under Data Store, select Include and choose the vSphere datastore. Example: dhci-vvol-ds01

Click Save.

Configure a Virtual IP Network¶

Configure a virtual IP (VIP) subnet for the Frontend Network. You can configure the VIP range to use when a virtual service is placed on the specific VIP network.

From the Controller dashboard, click the Infrastructure > Cloud Resources. Select Networks to see a list of networks available in vCenter Server. Verify that the proper Subnet information has been discover for the network.

Choose the network that will provide virtual IP addresses. Example: dhci_tanzu-frontend-net Click Edit.

Click Edit, under IP Subnet.

Click Add Static IP Address Pool. Select Use Static IP Address for VIPs and Service Engine.

Enter the IP Address Pool to be used by the Frontend Network for VIPs and Service Engines. Example: 192.168.102.51-192.168.102.90

Click Save.

The IP Subnet should now show Configured and the list of IPs should be listed under the IP Address Pool.

Click Save

Configure a Static Route¶

A default gateway enables the Service Engine to route traffic to the pool servers on the Workload Network. You must configure the Frontend Network gateway IP as the default gateway. You must configure static routes so that the Service Engines can route traffic to the Workload Networks and Client IP correctly.

From the Controller dashboard, click Infrastructure > Cloud Resources > Routing.

Click Create.

In Gateway Subnet, enter the Workload Network subnet. Example: 192.168.100.0/23

In Next Hop, enter the Frontend Network gateway IP address. Example: 192.168.102.1

Click Save.

Configure IPAM¶

Configure IPAM for the Controller and assign it to the Default-Cloud configuration. IPAM is required to allocate virtual IP addresses when virtual services get created.

From the Controller dashboard, click Templates > Profiles > IPAM/DNS Profiles. Click Create and select IPAM Profile.

Specify a Name.

Click Add Usable Network.

Under Cloud for Usable Network, choose Default-Cloud.

Set Usable Network to the Frontend Network. Example: dhci_tanzu-frontend-net

Click Save.

Verify the NSX Advanced Load Balancer¶

With the NSX Advanced Load Balancer deployed and configured, verify it is functioning correctly.

From the Controller dashboard, click Infrastructure > Clouds.

Verify status of Default-Cloud is green.

Enabling Workload Management¶

With the NSX Advanced Load Balancer deployed and configured, we are ready to enable Workload Management and deploy the vSphere with Tanzu Supervisor Cluster.

Deploying the Supervisor Cluster¶

Start by going to the Menu, click Workload Management.

If you have added a Tanzu (Basic, Standard, Advanced) license to your vCenter, then you will have the Get Started option. If a license has not been applied, you will need to complete the form to start the 60-day evaluation period for vSphere with Tanzu.

Step 1. Choose the vCenter Server and Network

Choose vSphere Distributed Switch (VDS) for the networking stack. Click Next.

Step 2. Cluster

Choose the compatible Cluster. Click Next.

Step 3. Storage

Choose the vVol policy we created in an earlier step. Click Next.

Step 4. Configure the Load Balancer

The following information comes from the NSX Advanced Load Balancer we deployed in an earlier step.

• Specify a Name. Example: nsx-alb
• Set the Load Balancer type: NSX Advanced Load Balancer
• Enter the Management IP Address of the NSX Advanced Load Balancer and port number (Port 443 is default): Example: 192.68.22.190:443
• Enter the Load Balancer Username: Example: admin
• Enter the Load Balancer Password: Example: VMware1!
• Enter the Server Certificate: Paste the NSX Advanced Load Balancer certificate here.

Step 5. Management Network

• Set Network Mode. Static (DHCP can be used if available)
• Set Management Network vDS port group. Example: dhci_tanzu-management-net
• Set Starting IP Address that will be the first IP in a range of 3 to assign to Supervisor control plane nodes' management interfaces. Example: 192.168.22.201
• Set Subnet Mask. Example: 255.255.248.0
• Set Gateway. Example: 192.168.20.1
• Set DNS Server(s). Example: 192.168.20.5, 192.168.20.6
• Optional: Set DNS Search Domain(s). Example: hpelab.local
• Set NTP Server(s). Example: 192.168.20.1

Step 6: Workload Network

• Set Network Mode. Static (DHCP can be used if available)
• Set Internal Network for Kubernetes Services. Use Default setting Example: 10.96.0.0/23
• Set Workload Network vDS port group. Example: dhci_tanzu-workload-net
• Set Network Name: Use Default setting
• Set IP Address Range to be used by the Tanzu Kubernetes guest clusters or other resources. Example: 192.168.100.51 - 192.168.100.90
• Set Subnet Mask: Example: 255.255.254.0
• Set Gateway: Example: 192.168.100.1
• Set DNS Server(s): Example: 192.168.20.5, 192.168.20.6
• Set NTP Server(s): Example: 192.168.20.1

Step 7: Tanzu Kubernetes Grid Service

Choose the subscribed content library tanzu-content-library we created in an earlier step.

Step 8: Review and Confirm

You can edit the Control Plane Size depending on the number of Pods you will have in your environment. In this case, we will keep the default setting.

Once ready click Finish.

The deployment of the Supervisor Cluster will take some time.

Once complete, the Config Status should show Running.

Create Namespace¶

With the Supervisor Cluster deployed, we are now ready to deploy the Tanzu Kubernetes Clusters. In order to do that, we need to create a Namespace.

Click on the Namespaces tab.

Click Create Namespace.

Choose the cluster Example: dHCI Cluster and specify a name for the namespace. Example: devteam

Click Create.

Once the Namespace has been created, you should see the following.

Assign storage, permissions, and VM Classes¶

From here we will configure user permissions, storage, and assign VM classes. We can also set limits to the resources available to the Namespace.

Permissions¶

We need to add the devops@vsphere.local user we created in an earlier step and assign Edit permissions in order for the user to deploy objects within the Tanzu Kubernetes cluster.

Click Add Permission

Select the vsphere.local identity source.

Enter the devops user and assign Can edit role. Click OK.

Storage¶

Click Add Storage

Select the following Storage Policies. Click OK.

VM Service¶

Click Add VM Class.

Select the following VM Classes. Click OK.

Click Add Content Library

Choose the subscribed content library tanzu-content-library. Click OK.

Capacity and Usage¶

(Optional) vSphere with Tanzu admins can set resource limits (CPU, Memory, Storage) available to the Namespace. At this time, we won't make any changes here.

Link to CLI tools¶

Once permissions and resources have been assigned to the devteam Namespace, we are ready to start interacting with vSphere with Tanzu. In order to do that we will need to download the CLI tools.

Click Open on the link to the CLI Tools.

This will open a browser window. Select Operating System and download and install the CLI tools.

Authenticate to the Supervisor Cluster¶

With the CLI tools installed, lets verify kubectl is working correctly.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+vmware.wcp.2", GitCommit:"619e0e4c3763423b0fe361c592ad3cd1e4914655", GitTreeState:"clean", BuildDate:"2022-02-15T10:23:25Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Next we need the Supervisor Cluster IP.

Now we are ready to login to the Supervisor Cluster.

Example:

kubectl vsphere login --server 192.168.102.52 --insecure-skip-tls-verify
Welcome to Photon 3.0 (\m) - Kernel \r (\l)

Username: devops@vsphere.local
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
Logged in successfully.

You have access to the following contexts:
   192.168.102.52
   devteam

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`

Now change the context to work with the devteam Namespace.

$ kubectl config use-context devteam
Switched to context "devteam".

We are now logged into the Supervisor Cluster and are ready to deploy a Tanzu Kubernetes cluster.

(Optional) Global proxy configuration vs cluster specific proxy¶

If you are behind a corporate proxy, vSphere with Tanzu supports two methods to configure proxy settings at the Supervisor Cluster level or at the Guest Kubernetes cluster level.

Global proxy configuration

To configure a global proxy configuration that applies to all guest Kubernetes clusters, the following example YAML manifest is ran at the Supervisor Cluster level.

apiVersion: run.tanzu.vmware.com/v1alpha1
kind: TkgServiceConfiguration
metadata:
  name: tkg-service-configuration
spec:
  network:
    proxy:
      httpProxy: http://192.168.1.1:8080 #Proxy URL for HTTP connections
      httpsProxy: http://192.168.1.1:8080 #Proxy URL for HTTPS connections
      noProxy: [192.0.0.0/8,172.17.0/16,.domain.local,.local,.svc,.svc.cluster.local] #SVC Pod, Egress, Ingress CIDRs

Kubernetes cluster proxy configuration

You can use a proxy server with an individual Tanzu Kubernetes cluster by applying the proxy server configuration to the cluster manifest.

spec:
  settings:
    proxy:
        httpProxy: http://192.168.1.1:8080  #Proxy URL for HTTP connections
        httpsProxy: http://192.168.1.1:8080 #Proxy URL for HTTPS connections
        noProxy: [192.0.0.0/8,172.17.0/16,.domain.local,.local,.svc,.svc.cluster.local] #SVC Pod, Egress, Ingress CIDRs

Deploy Tanzu Kubernetes Cluster¶

We need to deploy a Tanzu Kubernetes cluster to deploy workloads to. A Tanzu Kubernetes cluster consists of control plane nodes and worker nodes.

Information is provided to kubectl using YAML manifests.

Here is an example of a YAML file tkc.yaml Tanzu Kubernetes cluster that we will be deploying.

apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
  name: tanzu-devops
spec:
  settings:
    storage:
      defaultClass: dhci-vvol-tanzu-gold-policy
    network:
      serviceDomain: hpelab.local
  topology:
    controlPlane:
      replicas: 3
      vmClass: guaranteed-large
      storageClass: dhci-vvol-tanzu-gold-policy
      tkr:
        reference:
          name:  v1.23.8---vmware.3-tkg.1
    nodePools:
    - name: worker-nodepool-a1
      replicas: 3
      vmClass: guaranteed-xlarge
      storageClass: dhci-vvol-tanzu-gold-policy
      tkr:
        reference:
          name:  v1.23.8---vmware.3-tkg.1

To explain the components of this deployment.

This describes the type of objects that we will be deploying namely a TanzuKubernetesCluster with the name of tanzu-devops.

apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
  name: tanzu-devops

The spec defines the characteristics of the cluster.

• spec.settings.storage.defaultClass - Sets a default StorageClass using one of the Storage Policy we assigned to the Namespace.
• spec.settings.network.serviceDomain - Sets the service domain for the cluster.
• spec.topology.controlPlane - Sets the control plane node size based on vmClass assigned to the Namespace, the StorageClass to use when deploying the VMs, Kubernetes version (Example: 1.23.8) for the cluster.
• spec.topology.nodePools - Sets the worker node size based on vmClass assigned to the Namespace, the StorageClass to use when deploying the VMs, and Kubernetes version (1.22.9) for the cluster.
• To view versions of Kubernetes available, use the following command: kubectl get tanzukubernetesreleases

To create the Tanzu Kubernetes cluster, run the following command referencing the tkc.yaml file.

kubectl create -f tkc.yaml
tanzukubernetescluster.run.tanzu.vmware.com/tanzu-devops created

This deployment can take some time (20-30 minutes). We can monitor the deployment using the watch or -w command. Use Ctrl+C to exit watch.

watch kubectl get virtualmachine

or 

kubectl get virtualmachine -w

Login to Tanzu Kubernetes Cluster¶

Once the Tanzu Kubernetes Cluster is up and running, we need to login.

kubectl vsphere login --server 192.168.102.52 --tanzu-kubernetes-cluster-name tanzu-devops --tanzu-kubernetes-cluster-namespace devteam --vsphere-username devops@vsphere.local --insecure-skip-tls-verify

You should see output similar to below.

$ kubectl vsphere login --server 192.168.102.52 --tanzu-kubernetes-cluster-name tanzu-devops --tanzu-kubernetes-cluster-namespace devteam --vsphere-username devops@vsphere.local --insecure-skip-tls-verify
Welcome to Photon 3.0 (\m) - Kernel \r (\l)

KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
Logged in successfully.

You have access to the following contexts:
   192.168.102.52
   devteam
   tanzu-devops

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`

Next switch contexts so that kubectl is targeting our new TKG cluster.

$ kubectl config use-context tanzu-devops
Switched to context "tanzu-devops".

If you have gotten this far, your vSphere with Tanzu cluster is ready for developers to deploy applications.

Example DevOps Use Case with MongoDB¶

Now that vSphere with Tanzu is fully deployed, we can deploy workloads that leverages persistent storage from the HPE Alletra dHCI platform.

The following steps will performed to deploy MongoDB.

• Verify StorageClass availability and cluster permissions
• Configure Helm repo for MongoDB
• Deploy MongoDB ReplicaSet Helm chart
• Create DNS entries to MongoDB services
• Validate NSX Advanced Load Balancer
• Connect to MongoDB ReplicaSet using MongoDB Compass

Verify StorageClass availability¶

vSphere with Tanzu leverages Cloud Native Storage and Storage Policy Based Management to provision vSphere vVols based volumes, also known as First Class disks (FCDs), for use by workloads deployed on vSphere with Tanzu workloads. This removes many of the limitations of using third party CSI drivers which have limited access using traditional Fibre Channel or iSCSI data paths for block storage to Kubernetes clusters running as VMs.

To verify the available StorageClasses.

$ kubectl get sc
NAME                                    PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
dhci-vvol-tanzu-gold-policy (default)   csi.vsphere.vmware.com   Delete          Immediate           true                   129m

Configure user permissions (clusterrolebindings)

The final step before deploying workloads, we need to assign the devops user the ability to run any type of container in the environment.

kubectl create clusterrolebinding default-tkg-admin-privileged-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated

Configure Helm repo¶

In order to install MongoDB, we need to configure the Helm repository that contains the chart.

helm repo add bitnami https://charts.bitnami.com/bitnami

Next update Helm to pull the latest information from the repository.

helm repo update

To inspect the version of MongoDB available on the repository.

 helm search repo mongodb
NAME                                            CHART VERSION   APP VERSION     DESCRIPTION
bitnami/mongodb                                 13.5.0          6.0.3           MongoDB(R) is a relational open source NoSQL da...
bitnami/mongodb-sharded                         6.2.0           6.0.3           MongoDB(R) is an open source NoSQL database tha...

Deploy MongoDB ReplicaSet Helm chart¶

With the MongoDB repository in place and a StorageClass available, deploying complex MongoDB resources is greatly simplified by using Helm.

The following is an example of deploying MongoDB using Helm customizations that allow access from outside the vSphere with Tanzu environment to the MongoDB nodes. The ReplicaSet deployment will use LoadBalancer services that will be routed through the NSX Advanced Load Balancer.

helm install mongodb \
    --set architecture=replicaset \
    --set replicaSetName=dev-mongodb \
    --set replicaCount=3 \
    --set auth.rootPassword=secretpassword \
    --set auth.username=dev-user \
    --set auth.password=dev-password \
    --set auth.database=dev-database \
    --set persistence.size=50Gi \
    --set service.type=LoadBalancer \
    --set externalAccess.enabled=true \
    --set externalAccess.autoDiscovery.enabled=true \
    --set rbac.create=true \
    --set clusterDomain=hpelab.local \
    bitnami/mongodb

You should see similar output:

dev-user:~/tanzu$ helm install mongodb \
>     --set architecture=replicaset \
>     --set replicaSetName=dev-mongodb \
>     --set replicaCount=3 \
>     --set auth.rootPassword=secretpassword \
>     --set auth.username=dev-user \
>     --set auth.password=dev-password \
>     --set auth.database=dev-database \
>     --set persistence.size=50Gi \
>     --set service.type=LoadBalancer \
>     --set externalAccess.enabled=true \
>     --set externalAccess.autoDiscovery.enabled=true \
>     --set clusterDomain=hpelab.local \
>     --set rbac.create=true \
>     bitnami/mongodb
NAME: mongodb
LAST DEPLOYED: Tue Jan 24 10:46:11 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: mongodb
CHART VERSION: 13.5.0
APP VERSION: 6.0.3

** Please be patient while the chart is being deployed **

MongoDB® can be accessed on the following DNS name(s) and ports from within your cluster:

    mongodb-0.mongodb-headless.default.svc.hpelab.local:27017
    mongodb-1.mongodb-headless.default.svc.hpelab.local:27017
    mongodb-2.mongodb-headless.default.svc.hpelab.local:27017

To get the root password run:

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

To get the password for "dev-user" run:

    export MONGODB_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath="{.data.mongodb-passwords}" | base64 -d | awk -F',' '{print $1}')

To connect to your database, create a MongoDB® client container:

    kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.3-debian-11-r0 --command -- bash

Then, run the following command:
    mongosh admin --host "mongodb-0.mongodb-headless.default.svc.hpelab.local:27017,mongodb-1.mongodb-headless.default.svc.hpelab.local:27017,mongodb-2.mongodb-headless.default.svc.hpelab.local:27017" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

To connect to your database nodes from outside, you need to add both primary and secondary nodes hostnames/IPs to your Mongo client. To obtain them, follow the instructions below:

  NOTE: It may take a few minutes for the LoadBalancer IPs to be available.
        Watch the status with: 'kubectl get svc --namespace default -l "app.kubernetes.io/name=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/component=mongodb,pod" -w'

    MongoDB® nodes domain: You will have a different external IP for each MongoDB® node. You can get the list of external IPs using the command below:

        echo "$(kubectl get svc --namespace default -l "app.kubernetes.io/name=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/component=mongodb,pod" -o jsonpath='{.items[*].status.loadBalancer.ingress[0].ip}' | tr ' ' '\n')"

    MongoDB® nodes port: 27017

Validate NSX Advanced Load Balancer¶

We can inspect the NSX Advanced Load Balancer to verify that the Virtual Services were created successfully.

In a browser, navigate to the NSX Advanced Load Balancer Cluster URL and login. Example: https:\\192.168.22.190

From the Applications tab, in the Dashboard you should see various tiles representing the various Virtual Services that have been configured.

Click Virtual Services in the left hand menu. Now you can inspect the Virtual Services that were created automatically for vSphere with Tanzu, including the API services, Control Plane, vSphere CSI Driver, and our MongoDB application.

After verifying that the Virtual Services have been created, we need to create DNS A records for each of the MongoDB ReplicaSet nodes.

Create DNS entries to MongoDB services¶

One of the final steps to get access to the MongoDB nodes outside of the Kubernetes cluster requires the creation of DNS A records to map to the exposed LoadBalancer services.

The following A records will need to be created. This information comes from the output of the Helm deployment.

MongoDB can be accessed on the following DNS name(s):

    mongodb-0.mongodb-headless.default.svc.hpelab.local
    mongodb-1.mongodb-headless.default.svc.hpelab.local
    mongodb-2.mongodb-headless.default.svc.hpelab.local

In order to find the IP address assigned to the LoadBalancer service per node, run the following command:

$ kubectl get services
NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)           AGE
...
mongodb-0-external         LoadBalancer   10.102.215.237   192.168.102.56   27017:30751/TCP   72m
mongodb-1-external         LoadBalancer   10.100.185.170   192.168.102.57   27017:32610/TCP   72m
mongodb-2-external         LoadBalancer   10.99.222.65     192.168.102.58   27017:30634/TCP   72m
...

Next create a DNS A record for each service. The following is an example PowerShell command that will create a DNS A record in Microsoft DNS.

Add-DnsServerResourceRecordA -computername "dnsserver.example.com" -Name "mongodb-0.mongodb-headless.default.svc" -ZoneName "example.com" -AllowUpdateAny -CreatePtr -IPv4Address "192.168.102.56"

Connect to MongoDB ReplicaSet using MongoDB Compass¶

With the application deployed and services exposed, we can now connect to MongoDB using MongoDB Compass.

Launch MongoDB Compass. In the New Connection window, enter the following connection string or use Advanced Connection Options.

mongodb://dev-user:dev-password@mongodb-0.mongodb-headless.default.svc.hpelab.local:27017,mongodb-1.mongodb-headless.default.svc.hpelab.local:27017,mongodb-2.mongodb-headless.default.svc.hpelab.local:27017/?authSource=dev-database&replicaSet=dev-mongodb

Upon successful login, you should see the dev-database on the left. Create a collection in MongoDB, click Create Collection and specify a name for the collection. Example: dev-collection

Highlight the newly created collection.

Finally with a collection now ready, add data.

Example: A CSV containing a list of addresses was imported.

Example DevOps Use Case with NGINX¶

Now lets deploy a web application to demonstrate additional storage operations including volume expansion.

Sample web application deployment.

dev-user:~/tanzu$ cat nginx.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-vol
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
---  
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  labels: 
    run: nginx 
  name: nginx-deploy-sample 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      run: nginx-main 
  template: 
    metadata: 
      labels: 
        run: nginx-main 
    spec: 
      containers: 
      - image: public.ecr.aws/z9d2n7e1/nginx:1.19.5 
        name: nginx 
        volumeMounts: 
        - name: export 
          mountPath: "/usr/share/nginx/html"
        ports:
        - containerPort: 80 
      volumes: 
        - name: export 
          persistentVolumeClaim:
            claimName: nginx-vol

We will create the webserver deployment using the kubectl apply command.

$ kubectl apply -f nginx.yaml
persistentvolumeclaim/nginx-vol created
deployment.apps/nginx-deploy-sample created
$ kubectl rollout status deployment nginx-deploy-sample
deployment "nginx-deploy-sample" successfully rolled out
$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-deploy-sample-6865586d68-hkcdf   1/1     Running   0          35s

Expanding a persistent volume¶

We can use the Volume Expansion feature found in Kubernetes to expand volumes based as our application storage requirements increase.

We will show a simple example expanding a PVC using the PVCs we have available and the vSphere CSI Driver.

Existing PVC at 75 GB

We can verify the existing size of the PVC by running the kubectl get pvc command.

$ kubectl get pvc nginx-vol
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
nginx-vol    Bound    pvc-e489ca30-672a-44c5-996f-1c96f3c2436a   75Gi       RWO            dhci-vvol-tanzu-gold-policy   30m

Let's also log into the Pod to observe it from the view of the application.

$ kubectl exec -it nginx-deploy-sample-6865586d68-hkcdf -- /bin/bash
root@nginx-deploy-sample-6865586d68-hkcdf:/# df -h | grep /usr/share/nginx/html
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb         73G   60M   69G   1% /usr/share/nginx/html
...

Expand PVC to 100 GB

In order to expand the PVC, we need use the kubectl patch command to increase the backing volume size.

kubectl patch pvc/nginx-vol --patch '{"spec": {"resources": {"requests": {"storage": "100Gi"}}}}'
persistentvolumeclaim/nginx-vol patched

$ kubectl get pvc nginx-vol
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
nginx-vol    Bound    pvc-e489ca30-672a-44c5-996f-1c96f3c2436a   100Gi      RWO            dhci-vvol-tanzu-gold-policy   34m

Now we can log in to verify the application recognizes the volume expansion.

$ kubectl exec -it nginx-deploy-sample-6865586d68-hkcdf -- /bin/bash
root@nginx-deploy-sample-6865586d68-hkcdf:/# df -h | grep /usr/share/nginx/html
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdb         98G   60M   94G   1% /usr/share/nginx/html
...

We can clean up the deployment using the kubectl delete command:

$ kubectl delete -f nginx.yaml
persistentvolumeclaim "nginx-vol" deleted
deployment.apps "nginx-deploy-sample" deleted

VMs and Kubernetes¶

Finally one the of the most powerful pieces of vSphere with Tanzu is that the Tanzu Supervisor Clusters and the guest Tanzu Kubernetes clusters are native objects within vSphere. No longer are your Kubernetes clusters that are running in VMs hidden from the view of your vSphere admins.

Using VM Storage Policies for all workloads (both VMs and containerized applications running in vSphere with Tanzu) allows the administrator the ability to match the VM Storage Policies to the workloads or to tune the performance of an application as needed during runtime to either increase performance or reduce bottlenecks caused by resource contention. The rules found in the VM Storage Policy can be created and applied to individual or groups of vVols-based VMs or persistent storage for applications in Kubernetes for powerful automated storage management functionality.

To apply or edit the rules of a policy. Go to the vCenter Menu, Policies and Profiles, VM Storage Policies. Rules can be changed easily at any time.

Choose the policy and click Edit.

For example here, we are editing the Performance limits of a storage policy for the HPE Alletra dHCI.

Cloud Native Storage UI in vCenter¶

In the devteam Namespace, you have visibility to the volumes that have been created in your Tanzu Kubernetes clusters.

If we click on the number of PVCs, we can a more detailed list of the volumes backing the PVCs.

VMware Admins can also view the volumes backing in use by Tanzu Kubernetes clusters in vSphere with Tanzu by highlighting the Datacenter object in the vSphere Inventory and going to the Monitor tab and clicking Container Volumes under Cloud Native Storage.

When the PVC is mounted to a Pod, then it will also show the Pod information which is very helpful when tracking workloads.

Summary¶

This configuration guide offers practical insights into how Hewlett Packard Enterprise brings together the key components of its portfolio as well as strategic partnerships with VMware to simply and easily create a containerized environment that is able to be shared. The HPE Alletra dHCI solution combines the powerful HPE Proliant platform with the capable HPE Alletra Storage array to deliver unified setup, configuration, and management. These benefits are enhanced by a containerized persistent storage solution for both on-premises locations and the cloud, and they are topped off by integration with vSphere with Tanzu. As one of the earliest adopters of containers for persistent storage, Hewlett Packard Enterprise has gained the depth of experience and capability to provide a true hybrid cloud for customers who want to leverage containerized stateful applications alongside their traditional VM infrastructure.

The solution enables several critical use cases for virtualized and containerized environments:

• Dev/test workflows to develop in the cloud and deploy on-premises
• Disaster recovery for complete data protection across on-premises
• Instant availability of the right data for DevOps and CI/CD
• Sophisticated analytics to analyze data in the cloud through Infosight and without disrupting the on-premises workloads
• Consistent advanced data services and APIs
• Everything as a service

The solution described in this configuration guide is just the beginning of the capabilities and integrations that Hewlett Packard Enterprise plans to offer in the future. Through partnerships with VMware and the integration with vSphere with Tanzu, this newly created solution will become the launching point for the ability to manage all your containerized workloads on-premises and provide robust solutions to modernize your applications. Hewlett Packard Enterprise will continue to provide insights and best practices for stateful and stateless applications though simplified setup, configuration, and monitoring.