You need either a Linux, MacOS or a Windows client with the ability to ssh to an AWS IP address.
Jumpserver
Jumpservers have been deployed for each student with all prerequisites installed. First, go to the student data spreadsheet and select a host by entering your name. Then, download the ssh-private-key (id_rsa_student#) and change the file permissions. Finally, ssh to the ipaddress of your assigned jumpserver using the -i option to specify the identity file to be used. The username for the Jumpserver is "centos".
Linux/MAC
For Mac and Linux clients you must change the permission on the file.
Use the following steps to log in to PuTTY by using your SSH private key:
Specify the IP address of your assigned jumpserver then enter a name for the session and click Save.
Click Connection > Data in the left navigation pane and set the Auto-login username to "centos".
Click Connection > SSH > Auth in the left navigation pane and configure the SSH private key to use by clicking Browse under Private key file for authentication.
Navigate to the location where you saved your SSH private key file, select the file, and click Open.
The file path for the SSH private key file now displays in the Private key file for authentication field.
Click Session in the left navigation pane, then click Save in the Load, save or delete a stored session section.
Click Open to begin your session with the server.
Clients with local policy or VPN issues
Use the Google Cloud Shell. (Requires a Google account)
Once your Google Cloud Shell has started, you will have to copy the contents of you id_rsa_student#.pem file to a local file in the cloud shell. Then change the permission on the file and ssh into the jump host.
Deploy a Kubernetes cluster with all the addons you need to get a production ready container orchestration platform
Configure kubectl to manage your cluster
Why is this Important?
There are many ways to deploy a kubernetes cluster from a fully manual procedure to using a fully automated or opinionated SaaS. Cluster sizes can also widely vary from a single node deployment on your laptop, to thousands of nodes in a single logical cluster, or even across multiple clusters. Thus, picking a deployment model that suits the scale that you need as your business grows is important.
Change directories into the lab directory:
cd ~/lab
Deploy your cluster using the command below:
konvoy up --yes
The output should be similar to:
This process will take about 15 minutes to complete (additional time may be required for larger clusters), do you want to continue [y/n]: y
STAGE [Provisioning Infrastructure]
Initializing provider plugins...
...
Terraform has been successfully initialized!
...
STAGE [Deploying Enabled Addons]
helm [OK]
dashboard [OK]
fluentbit [OK]
awsebscsiprovisioner [OK]
traefik [OK]
opsportal [OK]
kommander [OK]
prometheus [OK]
elasticsearch [OK]
dex [OK]
elasticsearchexporter [OK]
kibana [OK]
traefik-forward-auth [OK]
prometheusadapter [OK]
dex-k8s-authenticator [OK]
velero [OK]
STAGE [Removing Disabled Addons]
Kubernetes cluster and addons deployed successfully!
Run `konvoy apply kubeconfig` to update kubectl credentials.
Navigate to the URL below to access various services running in the cluster.
https://a7e039f1a05a54f45b36e063f5aee077-287582892.us-west-2.elb.amazonaws.com/ops/landing
And login using the credentials below.
Username: goofy_einstein
Password: tUeARRKxM8PfrIy2cjFc1jI0Hr2I0duzlttr1LzRTKoDooQJ0d1yyutjNv4NLHvy
If the cluster was recently created, the dashboard and services may take a few minutes to be accessible.
If you get any error during the deployment of the addons (it can happen with network connectivity issues), then, you can run the following command to redeploy them:
konvoy deploy addons --yes
As soon as your cluster is successfully deployed, the URL and the credentials to access your cluster are displayed. When you lauch your dashboard URL in your browser the first screen will ask you to select "login or generate token", select login and use the credentials provided.
If you need to get this information later, you can execute the command below:
konvoy get ops-portal
Click on the Kubernetes Dashboard icon to open it.
To configure kubectl to manage your cluster, you simply need to run the following command:
konvoy apply kubeconfig
You can check that the Kubernetes cluster has been deployed using the version 1.16.3 with 3 control nodes and 4 workers nodes
kubectl get nodes
The output should be similar to:
NAME STATUS ROLES AGE VERSION
ip-10-0-129-247.us-west-2.compute.internal Ready <none> 10m v1.16.3
ip-10-0-129-41.us-west-2.compute.internal Ready <none> 10m v1.16.3
ip-10-0-129-88.us-west-2.compute.internal Ready <none> 10m v1.16.3
ip-10-0-130-84.us-west-2.compute.internal Ready <none> 10m v1.16.3
ip-10-0-193-118.us-west-2.compute.internal Ready master 11m v1.16.3
ip-10-0-193-232.us-west-2.compute.internal Ready master 12m v1.16.3
ip-10-0-194-21.us-west-2.compute.internal Ready master 13m v1.16.3
2. Expose a Kubernetes Application using a Service Type Load Balancer (L4)
Objectives
Deploy a Redis pod and expose it using a Service Type Load Balancer (L4) and validate that the connection is exposed to the outside
Deploy a couple hello-world applications and expose them using an Ingress service (L7) and validate that the connection is exposed to the outside
Why is this Important?
Exposing your application on a kubernetes cluster in an Enterprise-grade environment can be challenging to set up. With Konvoy, the integration with AWS cloud load balancer is already done by default and Traefik is deployed to allow you to easily create Ingresses.
Deploy a redis Pod on your Kubernetes cluster running the following command:
Finally, run the following command to see the URL of the Load Balancer created on AWS for this service:
kubectl get svc redis
The output should be similar to:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis LoadBalancer 10.0.51.32 a92b6c9216ccc11e982140acb7ee21b7-1453813785.us-west-2.elb.amazonaws.com 6379:31423/TCP 43s
You need to wait for a few minutes while the Load Balancer is created on AWS and the name resolution in place.
until nslookup $(kubectl get svc redis --output jsonpath={.status.loadBalancer.ingress[*].hostname})
do
sleep 1
done
You can validate that you can access the redis Pod from your laptop using telnet:
telnet $(kubectl get svc redis --output jsonpath={.status.loadBalancer.ingress[*].hostname}) 6379
The output should be similar to:
Trying 52.27.218.48...
Connected to a92b6c9216ccc11e982140acb7ee21b7-1453813785.us-west-2.elb.amazonaws.com.
Escape character is '^]'.
quit
+OK
Connection closed by foreign host.
If you don't have telnet installed in your machine, you can use nc instead.
3. Expose a Kubernetes Application using an Ingress (L7)
Deploy 2 web application Pods on your Kubernetes cluster running the following command:
kubectl run --restart=Never --image hashicorp/http-echo --labels app=http-echo-1 --port 80 http-echo-1 -- -listen=:80 --text="Hello from http-echo-1"
kubectl run --restart=Never --image hashicorp/http-echo --labels app=http-echo-2 --port 80 http-echo-2 -- -listen=:80 --text="Hello from http-echo-2"
Then, expose the Pods with a Service Type NodePort using the following commands:
You can also set some Traefik annotations to use some advanced features as described in this document.
4. Leverage Network Policies to restrict access
By default, all the pods can access all the services inside and outside the Kubernetes clusters and services exposed to the external world can be accessed by anyone. Kubernetes Network Policies can be used to restrict access.
When a Kubernetes cluster is deployed by Konvoy, a Calico cluster is automatically deployed on this cluster. It allows a user to define network policies without any additional configuration.
Objectives
Create a network policy to deny any ingress
Check that the Redis and the http-echo apps aren't accessible anymore
Create network policies to allow ingress access to these apps only
Check that the Redis and the http-echo apps are now accessible
Why is this Important?
In many cases, you want to restrict communications between services. For example, you often want some micro services to be reachable only other specific micro services.
In this lab, we restrict access to ingresses, so you may think that it's useless as we can simply not expose these apps if we want to restrict access. But, in fact, it makes sense to also create network policies to avoid cases where an app is exposed by mistake.
Create a PersistentVolumeClaim (pvc) to use the AWS EBS CSI driver
Create a service that will use this PVC and dynamically provision an EBS volume
Validate persistence
Why is this Important?
The goal of CSI is to establish a standardized mechanism for Container Orchestration Systems to expose arbitrary storage systems to their containerized workloads. The CSI specification emerged from cooperation between community members from various Container Orchestration Systems including Kubernetes, Mesos, Docker, and Cloud Foundry.
By creating an industry standard interface, the CSI initiative sets ground rules in order to minimize user confusion. By providing a pluggable standardized interface, the community will be able to adopt and maintain new CSI-enabled storage drivers to their kubernetes clusters as they mature. Choosing a solution that supports CSI integration will allow your business to adopt the latest and greatest storage solutions with ease.
When Konvoy is deployed on AWS, a StorageClass is created automatically as you can see below:
Run the following command to check the status of the PersistentVolumeClaim:
kubectl describe pvc dynamic
The output should be similar to:
Name: dynamic
Namespace: default
StorageClass: awsebscsiprovisioner
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 3s (x3 over 21s) persistentvolume-controller waiting for first consumer to be created before binding
Mounted By: <none>
As you can see, it is waiting for a Pod to use it to provision the AWS EBS volume.
Create a Kubernetes Deployment that will use this PersistentVolumeClaim using the following command:
NAME: jenkins
LAST DEPLOYED: Wed Aug 7 17:21:32 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
jenkins 5 1s
jenkins-tests 1 1s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
jenkins 0/1 1 0 1s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins Pending awsebscsiprovisioner 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
jenkins-c79f457cb-ccttb 0/1 Pending 0 1s
==> v1/Role
NAME AGE
jenkins-schedule-agents 1s
==> v1/RoleBinding
NAME AGE
jenkins-schedule-agents 1s
==> v1/Secret
NAME TYPE DATA AGE
jenkins Opaque 2 1s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins LoadBalancer 10.0.9.26 <pending> 8080:30323/TCP 1s
jenkins-agent ClusterIP 10.0.41.64 <none> 50000/TCP 1s
==> v1/ServiceAccount
NAME SECRETS AGE
jenkins 1 1s
Then, run the following command to get the URL of the Load Balancer created on AWS for this service:
kubectl get svc jenkins
The output should be similar to:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins LoadBalancer 10.0.9.26 a71b8025991124a90b2babf7ba2a75da-492974167.us-west-2.elb.amazonaws.com 8080:30323/TCP 16m
You need to wait for a few minutes while the Load Balancer is created on AWS and the name resolution in place.
until nslookup $(kubectl get svc jenkins --output jsonpath={.status.loadBalancer.ingress[*].hostname})
do
sleep 1
done
echo "Open http://$(kubectl get svc jenkins --output jsonpath={.status.loadBalancer.ingress[*].hostname}):8080 to access the Jenkins UI"
Go to the URL displayed to access Jenkins.
Login with the user admin and the password password.
7. Deploy Apache Kafka using KUDO
The Kubernetes Universal Declarative Operator (KUDO) is a highly productive toolkit for writing operators for Kubernetes. Using KUDO, you can deploy your applications, give your users the tools they need to operate it, and understand how it's behaving in their environments — all without a PhD in Kubernetes.
Run the following commands to deploy KUDO on your Kubernetes cluster:
kubectl kudo init
The output should be similar to:
$KUDO_HOME has been configured at /home/centos/.kudo.
✅ installed crds
✅ installed service accounts and other requirements for controller to run
✅ installed kudo controller
Check the status of the KUDO controller:
kubectl get pods -n kudo-system
The output should be similar to:
NAME READY STATUS RESTARTS AGE
kudo-controller-manager-0 1/1 Running 0 84s
KUDO is creating CRDs (new objects) in Kubernetes and you can get information about these objects like you can get informations about pods, deployments, ...
Run this command to get the list of CRDs created by KUDO:
The Traefik dashboard indicates the nginx application is ready to receive traffic but if you try access nginx with URL listed below, you will notice 404 Not Found error like:
curl -k https://$(kubectl get svc traefik-kubeaddons -n kubeaddons --output jsonpath="{.status.loadBalancer.ingress[*].hostname}")/applications/nginx/
Don't forget the trailing slash at the end of the URL. Otherwise, you won't generate a 404 error.
Let's troubleshoot this failure with Konvoy Kibana.
With Konvoy Kibana's near real time log collection and indexing, we can easily identify the ingress traffic was eventually handled by a pod kubernetes.pod_name:nginx-755464dd6c-dnvp9 in nginx service. The log also gave us more information on the failure, "GET /applications/nginx/ HTTP/1.1" 404, which tell us that nginx can't find resource at path /applications/nginx/.
That is neat! Because w/o Kibana, you wouldn't know which Pod in our nginx service handles this request. (Our nginx deployment example launched 3 Pods to serve HTTP request) Not mention if there are multiple nginx service exists in the same K8s cluster but hosted at different namespace.
To fix this failure requires some knownledge on Nginx configuration. In general, when nginx is launched with default configuration, it serves a virtual directory on its ROOT path (/). When receives HTTP requests, the nginx walk through its virtual directory to return back resources to the client.
In terms of our example, the Ingress configuration we submitted to k8s was configured to a path at /applications/nginx/. The traefik ingress controller sees this Ingress configuration and forwards any resource request at path /applications/nginx/ to the down stream nginx service at the same path. The pod kubernetes.pod_name:nginx-755464dd6c-dnvp9 received this request but nginx instance in this pod failed to locate any resource under path /applications/nginx/. That is the reason we saw this failure, "GET /applications/nginx/ HTTP/1.1" 404.
You can, of course, configure nginx instance to serve resources at path /applications/nginx/. But an alternative solution is leverage traefik to strip PATH /applications/nginx/ to ROOT (/) before route requests to nginx.
According to Traefik documentation PathPrefixStrip, the annotation (traefik.ingress.kubernetes.io/rule-type) is exactly what we need to direct traefik to strip ingress HOST PATH to ROOT PATH forementioned.
Edit the cluster.yaml file to change the Kubernetes version from 1.16.3 to 1.16.4 in the 2 corresponding fields:
...
spec:
kubernetes:
version: 1.16.4
...
konvoy up --yes --upgrade --force-upgrade
The output should be similar to:
This process will take about 15 minutes to complete (additional time may be required for larger clusters)
STAGE [Provisioning Infrastructure]
Initializing provider plugins...
Terraform has been successfully initialized!
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
random_id.id: Refreshing state... (ID: jKY)
...
No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
cluster_name = konvoy_v1.1.1-8ca6
vpc_id = vpc-0941bb098eb24080d
STAGE [Running Preflights]
...
STAGE [Determining Upgrade Safety]
ip-10-0-193-118.us-west-2.compute.internal [OK]
ip-10-0-193-232.us-west-2.compute.internal [OK]
ip-10-0-194-21.us-west-2.compute.internal [OK]
ip-10-0-128-239.us-west-2.compute.internal [WARNING]
- All replicas of the ReplicaSet "default/nginx-7c45b84548" are running on this node.
ip-10-0-128-64.us-west-2.compute.internal [WARNING]
- Pod "default/jenkins-c79f457cb-vrjjq" is using EmptyDir volume "plugins", which is unsafe for upgrades.
- Pod "default/jenkins-c79f457cb-vrjjq" is using EmptyDir volume "tmp", which is unsafe for upgrades.
- Pod "default/jenkins-c79f457cb-vrjjq" is using EmptyDir volume "plugin-dir", which is unsafe for upgrades.
- Pod "default/jenkins-c79f457cb-vrjjq" is using EmptyDir volume "secrets-dir", which is unsafe for upgrades.
- Pod "default/http-echo-2" is not being managed by a controller. Upgrading this node might result in data or availability loss.
- Pod managed by ReplicaSet "default/jenkins-c79f457cb" is running on this node, and the ReplicaSet does not have a replica count greater than 1.
- All replicas of the ReplicaSet "default/jenkins-c79f457cb" are running on this node.
- Pod managed by ReplicaSet "default/kudo-kafka-generator-d655d6dff" is running on this node, and the ReplicaSet does not have a replica count greater than 1.
- All replicas of the ReplicaSet "default/kudo-kafka-generator-d655d6dff" are running on this node.
ip-10-0-129-247.us-west-2.compute.internal [WARNING]
- Pod "default/http-echo-1" is not being managed by a controller. Upgrading this node might result in data or availability loss.
- Pod managed by StatefulSet "kudo-system/kudo-controller-manager" is running on this node, and the StatefulSet does not have a replica count greater than 1.
ip-10-0-129-41.us-west-2.compute.internal [OK]
ip-10-0-129-88.us-west-2.compute.internal [WARNING]
- Pod managed by ReplicaSet "default/ebs-dynamic-app-68b598758" is running on this node, and the ReplicaSet does not have a replica count greater than 1.
- All replicas of the ReplicaSet "default/ebs-dynamic-app-68b598758" are running on this node.
ip-10-0-130-84.us-west-2.compute.internal [WARNING]
- Pod managed by ReplicaSet "default/kudo-kafka-consumer-6b4dd5cd59" is running on this node, and the ReplicaSet does not have a replica count greater than 1.
- All replicas of the ReplicaSet "default/kudo-kafka-consumer-6b4dd5cd59" are running on this node.
- Pod "default/redis" is not being managed by a controller. Upgrading this node might result in data or availability loss.
STAGE [Upgrading Kubernetes]
...
PLAY [Upgrade Nodes] ********************************************************************************************************************************************************************
...
TASK [kubeadm-upgrade-nodes : drain node] ***********************************************************************************************************************************************
changed: [10.0.129.184 -> ec2-54-191-70-155.us-west-2.compute.amazonaws.com]
...
STAGE [Deploying Enabled Addons]
helm [OK]
dashboard [OK]
awsebscsiprovisioner [OK]
opsportal [OK]
fluentbit [OK]
traefik [OK]
kommander [OK]
elasticsearch [OK]
prometheus [OK]
traefik-forward-auth [OK]
dex [OK]
prometheusadapter [OK]
kibana [OK]
elasticsearchexporter [OK]
velero [OK]
dex-k8s-authenticator [OK]
STAGE [Removing Disabled Addons]
Kubernetes cluster and addons deployed successfully!
Run `konvoy apply kubeconfig` to update kubectl credentials.
Navigate to the URL below to access various services running in the cluster.
https://a1efd30f824244733adc1fb95157b9b1-2077667181.us-west-2.elb.amazonaws.com/ops/landing
And login using the credentials below.
Username: angry_williams
Password: TNFGnFrZjhqaF0SNLoCzN3gvqrEsviTYxvMyuPv8KHU13ob6eNa0N7LfSVhd07Xk
If the cluster was recently created, the dashboard and services may take a few minutes to be accessible.
If there is any error during the upgrade, run the konvoy up --yes --upgrade --force-upgrade again. It can happen when the drain command times out.
Without the --force-upgrade flag, the Kubernetes nodes that have under replicated pods wouldn't be upgraded.
Check that the Jenkins and the ebs-dynamic-app apps are still accessible.
The Redis and the http-echo apps aren't running anymore as they haven't been deployed using a deployment.
12. Destroy a Konvoy cluster
When you run konvoy down, the command removes all of the AWS infrastructure resources create for the cluster, including any volumes that are backing PersistentVolumesClaims with a Delete ReclaimPolicy.
To completely remove Konvoy cluster resources:
Change to the directory that contains your cluster’s state files, then run the following command:
konvoy down --yes
The konvoy down command then begins removing cluster resources by deleting load balancers, security groups and volumes. It deletes these resources using the AWS API to ensure they are deleted quickly.
After konvoy down removes these resources, it uses Terraform to delete the resources created by the konvoy up command and Terraform provisioning.
Appendix 1. Setting up an external identity provider
Your Konvoy cluster contains a Dex instance which serves as an identity broker and allows you to integrate with Google's OAuth.
Google's OAuth 2.0 APIs can be used for both authentication and authorization.
In the Credentials tab of that project start with setting up the OAuth consent screen.
Indicate an Application name and add the DNS name via which your Konvoy cluster is publicly reachable (<public-cluster-dns-name>) into Authorized domains.
Save the OAuth consent screen configuration.
Press Create credentials, select OAuth client ID, and then Web application.
Under Authorized redirect URIs insert https://<public-cluster-dns-name>/dex/callback.
Don't forget to hit ENTER when setting up oauth in the google console for the redirect url and other fields, otherwise the values are not saved if you just hit the save button.
Save the configuration and note down the client ID and the client secret.
Run the following command (after inserting your email address) to provide admin rights to your Google account:
