In this tutorial, you will learn how to setup Kubernetes ingress using Nginx ingress controller and to route traffic to deployments using wildcard DNS. If you want to understand how Kubernetes ingress works, please read this blog post on Kubernetes Ingress Tutorial. A Kuberntes cluster kubectl.
Kubernetes is one of the most powerful methods of deploying clusters for the management and deployment of containers. NGINX is one of the most popular web servers on the planet, and also one of the best ways to illustrate how to deploy a container. Combine the two and you can enjoy a highly scalable web server, ready to help your business grow.
But how do you deploy that NGINX container on a Kubernetes cluster? I'm going to show you. A word of warning, I'm using an Antsle cloud server, which makes deploying the Kubernetes platform incredibly simple. The operating system hosting Kubernetes is Ubuntu Server 16.04. I will assume you already have Kubernetes up and running. For those that don't have an Antsle, and need to first install Kubernetes, check out how to here. For this demonstration, I'll be deploying on three virtual machines:
- kubernetes at 192.168.1.190
- kubernetes2 at 192.168.1.191
- kubernetes3 at 192.168.1.192
The machine with hostname kubernetes will serve as my master, while kubernetes2/3 will serve as nodes.
With that out of the way, let's deploy.
Setting up hostnames
The first thing we have to do is map out hostnames on each machine. So for each machine, we'll issue the command sudo nano /etc/hosts and map the other machine's IP address to hostname. So on kubernetes, my hosts additions will be:
On kubernetes2, the additions will be:
On kubernetes3, the additions will be:
Once you've made the additions, save and close the file. Make sure you can ping each server, via hostname.
Initialize the master node
With everything in place, it's time to initialize the master node. Log into kubernetes (my master node) and issue the command:
This command can take a minute or two to complete, as the necessary images might have to be pulled. Once it completes, you should see similar output shown in Figure A.
Figure A
Included with the output will be your token and discovery token. Make sure you copy those down (or just copy the entire join command), as you'll need that information to join the nodes to the cluster.
The next step is clearly outlined in the output of the initialization command. Effectively, you must issue the following commands:
Once you've issued the above commands, check on the status of the nodes (there'll only be one at this point), with the command:
You should see the master node listed (Figure B).
Figure B
The reason our Master Node is listed as not ready is because it has yet to have a Container Networking Interface (CNI). Let's deploy a Calico CNI for the master with the command:
Let's make sure Calico was deployed correctly with the command kubectl get pods —all-namespaces.
The output of the above command (Figure C) should show Calico running.
Figure C
Run kubectl get nodes again, and you should see the Master Node is now listed as Ready.
Adding nodes to the cluster
Next we head over to our nodes to add them to the cluster. Remember the join command in the output from the Master Node initialization command? Head over to kubernetes2 and issue that command, which will look something like this:
Once that command completes, do the same on kubernetes3. After you've issued the join command on your nodes, go back to the Master Node and issue the command kubectl get nodes and you should see all nodes ready (Figure D).
Figure D
Deploy the NGINX container to the cluster
It's now time to deploy the NGINX container. From the master node, issue the command:
Next we make the NGINX container available to the network with the command:
Issue the command kubectl get svc to see your NGINX listing (as well as the assigned port, given by Kubernetes - Figure E)
Figure E
Let's test this with the command:
NOTE: The 30655 port was assigned during the create service command. It will be unique to your deployment.
The output of the curl command should display the HTML of the NGINX index.html page. If you see that, congratulations, your NGINX container has been deployed on your Kubernetes cluster. If you point a web browser to http://IP_OF_NODE:ASSIGNED_PORT (Where IP_OF_NODE is an IP address of one of your nodes and ASSIGNED_PORT is the port assigned during the create service command), you should see the NGINX Welcome page!
Basic deployment
What we've done is a very basic Kubernetes deployment of NGINX on a cluster. There is so much more to learn about using Kubernetes. This, however, should give you a good start as well as help you easily deploy NGINX on your Kubernetes cluster.
Data Center Trends Newsletter
DevOps, virtualization, the hybrid cloud, storage, and operational efficiency are just some of the data center topics we'll highlight. Delivered Mondays and Wednesdays
Sign up today Sign up today Also See
- How to use Antsle to quickly deploy a virtual machine (TechRepublic)
- Meet the Antsle: The perfect out-of-the box virtual machine solution (TechRepublic)
- 10 Kubernetes tips for getting the most out of the open source container system (TechRepublic)
- How to run NGINX as a Docker container (TechRepublic)
- 3 quick steps to optimize the performance of your NGINX server (TechRepublic)
- Kubernetes 1.9 brings beta support for Windows apps (ZDNet)
This blog will explain the steps required to deploy and manage a 3-node MarkLogic cluster using Kubernetes and will also set up an NGINX container to manage ingress/access to each MarkLogic node. Let’s get started!
MarkLogic officially supports Docker containers with version 9.0-5 or later. Please use discretion when working with these examples using older versions of MarkLogic.
Assumptions
The following assumptions are made before beginning this walk-through. You’ll need to make sure you’ve got this initial setup completed before continuing.
- Docker v 1.12+
- Docker for Mac Installed
- Mac OS Sierra (should be compatible with earlier versions though)
- VirtualBox version5.0.30+
- Minikube version 0.12.2+
- Kubectl installed
- MarkLogic Server 8.0.5+
- You have a decent familiarity with Docker in general.
- You have a private Docker Registry setup to hold the MarkLogic and NGINX images we’re going to build in the following steps.
Getting Started
Running containers inside of Kubernetes relies upon having a local/private registry (or public) available to pull the images from. This document will not cover setting up the private registry. See Setting up a Private Registry for more information. You will also need to download the MarkLogic 64-bit RPM (AMD64, Intel EM64T)) from MarkLogic.
- Create a folder on your local system to hold our configuration files/scripts and MarkLogic RPM
- Move the MarkLogic RPM you downloaded to this folder
- Create a file called “Dockerfile” and copy the code below to setup our MarkLogic Docker container
This configuration will create a Docker image with MarkLogic installed and will copy the necessary initialization scripts to set up each MarkLogic node and create the cluster. Don’t build the image just yet, as we still need to create the scripts.
Initialization Scripts
There are four scripts used in this example.
- entry-point.sh – This is the entry point script for the container. This script will run in the background to ensure the container doesn’t exit after the scripts are finished running, start the MarkLogic server, and will also intercept the term signal when the container exits to gracefully shutdown MarkLogic.
- setup-master.sh – This script is called by the entry-point script and performs the procedures for initializing the MarkLogic server and setting up security
- setup-child.sh – This script is also called by the entry-point script and performs the steps for initializing the MarkLogic server as well as joining the server to the MarkLogic cluster
- mlconfig.sh – This is a simple configuration file that can be modified for your particular settings/environment and is used by the initialization scripts above
Ok, that’s it for creating the Docker file to build the image as well as creating our initialization scripts. Next, let’s go ahead and build our image. In a terminal window, navigate to the folder you created to store the Dockerfile and scripts, and enter:
Your command should look something like “
docker build -t 192.168.1.11/my-ml-image:v1 .
“. Don’t forget to add the “.” at the end of the command (this tells Docker to look in the current directory for the Dockerfile to use). Now that you’ve built your image, you need to push it to your registry.Creating the Kubernetes Configuration Files
Kubernetes is an open-source tool for automating the deployment, scaling, and management of containerized applications. We’ll use the Kubectl CLI downloaded earlier to deploy, inspect, and interact with our deployed environment. Additionally, Minikube will provide the environment for hosting our Kubernetes-managed services. The Minikube VM is already preconfigured with the necessary services to run our applications. Now, we need to create the Kubernetes configuration files to use when deploying and managing our MarkLogic environment. The files can be in either JSON or YAML format. I’ve chosen YAML for this example. Please make sure you use correct formatting (indentation) when copying these files. You can use an online tool iike YAMLLint to make sure the file is correctly formatted, if you run into problems.
Let’s build the configuration for our MarkLogic Pods.
NOTE: Pods are individual units that contain 1..n Docker containers. 1..n Pods run on individual Nodes
MarkLogic PetSet Configuration
One of the first things to note is that the “kind” of service being deployed is a PetSet. “A PetSet is a group of stateful pods that require a stronger notion of identity. The document refers to these as “clustered applications”. Normally, pods are created and destroyed with no notion of persistence or identity. In our case, we need to be able to persist the data for each node as well as the same container “identity” so that when a MarkLogic pod is replaced for whatever reason, it is recognizable by the cluster and will use the same data as the previous pod it replaced. Let’s inspect the configuration sections.
Update to PetSets
Note: PetSets have been replaced with a “StatefulSet” in versions 1.5 Beta+ of Kubernetes. A StatefulSet is a Controller that provides a unique identity to its Pods. It provides guarantees about the ordering of deployment and scaling.
If you run Minikube version 0.14.0+ you must use a StatefulSet instead of a PetSet. Modify the following lines in the PetSet configuration to create a StatefulSet. After modification, save the file and you’re done.
- line 1 – change apiVersion from apps/v1alpha1 to apps/v1beta1
- line 2 – change kind from PetSet to StatefulSet
- lines 12-13 regarding the annotations – comment out or remove
Let’s quickly review the various sections within the file.
- Lines 3 – 11
- metadata.name – This will be the name for our service.
- spec.serviceName – This references the networking service we’ll create later on.
- spec.replicas – Specifies we want to initially create and always have exactly 3 Pods running.
- spec.template.metadata.labels.app – The “labeling” applied to this service. Read more about Labels & Selectors on the Kubernetes website.
- Lines 14 – 43
- spec.terminationGracePeriodSeconds – Allows 30 seconds for the Pod to gracefully terminate before killing the process.
- spec.containers – Allows you to specify 1..n containers to run.
- name – The based name to use for each container. I.e. marklogic-0, marklogic-1, etc.
- image – The image each container will be based upon.
- imagePullPolicy – Specifies to ALWAYS pull a fresh image from the registry when the service is instantiated. Can also be set to “IfNotPresent” or “Never”.
- command – The default command/script to run when the container is started.
- ports – The ports on the container we’re going to publicly expose.
- lifecycle.preStop.exec.command – Specifies a script/command to run when the container receives a SIGTERM signal.
- volumeMounts – Specifies the folder/file within the container to mount to the “volume mount”.
- imagePullsecrets – Specifies the name of the “secret” were using for authenticating to our Docker Registry when the container pulls the image.
- Lines 44 – 53
- volumeClaimTemplates
- metadata.name – The name to use for this “volumeClaimTemplate”. Note, this name is referenced in the volumnMounts.name of the container specification.
- annotations
- volume.alpha.kubernetes.io/storage-class – Specifies the underlying driver to use for the volume claim. We’re simply mounting to our host file system, but other drivers are available to support Cloud deployments and more.
- spec
- accessModes – Specifies the Read/Write permissions. See Access Modes in the Persistent Volume documentation.
- resources.requests.storage – The amount of storage we want to request when the volume claim is made
A “volumeClaimTemplate” basically defines a template for each container to use when claiming persistent storage on the host. These volume claims request storage on our host system (Minikube in this case) for persisting our MarkLogic data. This data will persist between Pod restarts so long as you don’t “delete” the Minikube instance or delete the PersistentVolume from the environment. In our Kubernetes configuration file, we’re specifying we want to map the entire /var/opt/MarkLogic directory to our host file system. This mapping could be made more specific if you’d like (i.e. instead of mappint the entire /var/opt/MarkLogic directory you could map sub-directories within the MarkLogic one). See Persistent Volumes for additional information.
- volumeClaimTemplates
Creating the Registry Authentication Secret
This “Secret” is used by Kubernetes to authenticate with the Docker Registry when it pulls an image. You may or may not require this depending on how your registry is set up, but I’ll include an example Secret below:
- In your working directory, create a file called registry-secret.yml and paste the code block below.
- Follow the instructions in the heading of the example below to create your own unique “secret” to use for registry authentication.
The important piece here is the .dockerconfigjson. The string here for the value is our encrypted credentials for authentication to the private Docker Registry. Yours will obviously be different. The comments at the beginning of the file should help to explain how to create this string.
That completes our brief introduction to the PetSet specification for our MarkLogic deployment. Next we’ll look at our Service that will support the networking interface to our Pods.
MarkLogic Service
Let’s review the different portions of this configuration.
- Lines 1 – 10
- kind – This specifies the type of configuration being deployed
- metadata
- name – The name of the service supporting our MarkLogic PetSet. Note, you can deploy multiple services that can support various portions of your deployment architecture.
- namespace – Resources deployed and managed by Kubernetes can be assigned to a namespace. The default namespace is “default”
- labels
- app – Specifies the label to be applied to the service.
- annotations
- service.alpha.kubernetes.io/tolerate-unready-endpoints – An annotation on the Service denoting if the endpoints controller should go ahead and create endpoints for unready pods.
- Lines 11 – 41
- spec
- clusterIP – Specifying “None” creates a Headless Service
- selector
- app – Associates this service to the MarkLogic PetSet
- ports – List the ports to expose on each Pod being run
- protocol – The specific network protocol to allow
- port – The abstracted Service port, which can be any port other pods use to access the Service
- targetPort – The port the container accepts traffic on
- name – Friendly name to assign to this port
Repeat this section for every port you want to expose on your containers. Next, let’s set up the NGINX load balancer.
- spec
NGINX Load Balancer
The NGINX provides two services for our MarkLogic cluster.
- Obviously, it provides load balancing between the nodes in the cluster from outside the Kubernetes internal network.
- Allows us to access each node individually. Remember, we have a single service supporting all three nodes in the cluster (single ingress point to the environment) so in order to reach a specific node within the cluster from the outside we need NGINX to “front” all requests to the environment and provide the correct routing to individual MarkLogic hosts.
This blog post will not go into detail about NGINX, but will explain the configurations implemented. The first thing to do is create our custom NGINX image. Just like before when we created our MarkLogic image, we need to create a Dockerfile for our NGINX configuration. This image will be based upon the standard NGINX image you’d get from the Docker Registry and will also include the custom NGINX configuration we need for our setup.
- Create another folder to hold our NGINX Dockerfile and associated configuration file(s)
- Your Dockerfile should look like this:
- Now, create a file called nginx.conf. This file will configure our NGINX image to correctly route and loadbalance between our MarkLogic hosts. It should have the following code copied to it:
- Create a file called upstream-defs.conf and copy the code below to it. This file will be included in the nginx.conf file above.
- Build the image and push it to your registry.
Remember, if you exposed additional ports in your Dockerfile, PetSet/StatefulSet, and Service and those ports need to be publicly accessible, you’ll need to modify the NGINX configuration to support their routing. Also, if you increase or decrease the number of MarkLogic Pods being deployed, you’ll need to update the configuration as well so reflect this change. The NGINX configuration shown in this example expects 3 MarkLogic pods to be running.
Now, we’ll create specification for the NGINX Replication Controller.
Create a file called nginx-ingress.rc.yml and copy the code below into the file.
Let’s go through the various portions of the NGINX Replicaton Controller configuration.
- Lines 1 – 6
- kind – Specifies the type of resource being deployed
- metadata
- name – The name to use for our resource
- labels
- app – Label applied to the resource
- Lines 15 – 32
- image, imagePullPolicy, and name are all basically the same as the PetSet specification except for the actual image we’re going to pull. In this case, we want the <your registry/your image name>:<your image tag> NGINX image.
- ports – The ports that should be publicly exposed.
- Lines 38 – 39
- imagePullSecrets – Again, the same as our PetSet specification above.
Starting up the Cluster
The first thing to do is to make sure our Miikube VM is up and running.
You should see the following:
Additionally, your VirtualBox manager should show the Minikube VM running.
Ok, now our minikube environment is up and running. If you want to see the configuration for the node, you can open the /Users/<username>/.minikube/machines/minikube/config.json file.
We need to deploy our resources in a specific order or you might get errors. The first item will be the Secret our services will use to authenticate to the private Docker registry when pulling images into the pod. In a terminal window, navigate to the folder where you stored your Kubernetes configuration files then type:
Next, deploy the network service
Now, deploy the petset or stateful set, whichever you’re implementing.
Lastly, we’ll deploy our NGINX pod for servicing requests to our MarkLogic nodes
Let’s take a look at the Kubernetes dashboard to see what’s been deployed and the status of each. The command below should open a browser window and navigate to the Kubernetes Management Dashboard. From here you can deploy, manage, and inspect your various services/pods.
Minikube Dashboard
Error Messages
You may see the following error when viewing the dashboard or the logs.
This is expected as during the very first run of our petset | statefulset as the image has to be pulled in its entirety from the registry. After the initial pull, subsequent pulls will only need to retrieve the modified image layers. The initial image pull can take upwards of a minute or so. Also be advised that our NGINX configuration is set up to expect 3 MarkLogic nodes. If you deploy less than 3 you’ll get an error for this container.
Additionally, you can see that PersistentVolume claims have been established for each MarkLogic node. The volume claims map to /tmp/hostpath_pv/<unique ID> on the host (minikube). If you run:
you’ll be connected to the minikube host environment. You can then list the contents of each MarkLogic node directory.
You can see which volume maps to which pod by clicking the Persistent Volumes link in the Admin section of the Kubernetes Dashboard and inspecting the persistent volume name.
If you want to see the output of each pod and the messages displayed during the pod’s initialization you can click the to the right of each Pod. This will open up a separate browser tab and display the output for that pod.
Opening up MarkLogic Admin Pages
The NGINX load balancer is set to route page requests to each node using the “least conn” logic so going to the root URL will result in “random” hosts being returned. The NGINX load balancer will also allow you to navigate to a specific node in the cluster.
Open a web browser and go to
http://<ip address of your kubernetes dashboard>:8001
. You should hit one of the running MarkLogic nodes. To hit a specific node, enter http://<ip address of your kubernetes dashboard>:8001/[ml0 | ml1 | ml2]
.Summary
So that’s it! You have successfully created and deployed a 3-node MarkLogic cluster with an NGINX load balancer using Kubernetes to deploy and manage it. Although this was a fairly long article, the majority of it was the scripts. Hopefully, this will demonstrate the capabilities available. While this may not be a “production grade” deployment, it does illustrate the basic concepts. There are a multitude of ways to perform this sort of deployment; this just happens to be one that worked for me.
Additional Resources
- Read this blog on Building a MarkLogic Docker Container
- Read this blog on Automating MarkLogic Docker Installs
- Read this blog on Creating an MLCP Docker container with pre-loaded data
- Read this blog on MarkLogic Docker Container on Windows
- View this webinar on Running Marklogic in Containers (Docker and Kubernetes)