Deploy Rancher on Azure for Kubernetes Management

Lately I have been hearing a lot about a solution named Rancher in the Kubernetes space. Rancher is an open source Kubernetes Multi-Cluster Operations and Workload Management solution. You can learn more about Rancher here: https://www.rancher.com.

In short you can use Rancher to deploy and manage Kubernetes clusters deployed to Azure, AWS, GCP their managed Kubernetes offerings like GCE, EKS, AKS or even if you rolled your own. Rancher also integrates with a bunch of 3rd party solutions for things like authentication such as Active Directory, Azure Active Directory, Github, and Ping and logging solutions such as Splunk, Elasticsearch, or a Syslog endpoint.

Recently training opened up for some Rancher/Kubernetes/Docker training so I decided to go. The primary focus was on Rancher while also covering some good info on Docker and Kubernetes. This was really good training with a lot of hands on time, however there was one problem with the labs. The labs had instructions and setup scripts ready to go to run Rancher local on your laptop or on AWS via Terraform. There was nothing for Azure.

I ended up getting my Rancher environment running on Azure but it would have been nice to have some scripts or templates ready to go to spin up Rancher on Azure. I did find some ARM templates to spin up Rancher but they deployed an old version and it was not clear in the templates on where they could be updated to deploy the new version of Rancher. I decided to spend some time building out a couple of ARM templates that can be used to quickly deploy Rancher on Azure and add a Kubernetes host to Rancher. In the ARM template I pulled together it pulls the Rancher container from Docker Hub so it will always deploy the latest version. In this blog post I will spell out the steps to get your Rancher up and running in under 15 minutes.

First off you can find the ARM Templates here on my Github here: https://github.com/Buchatech/DeployRanchertoAzure.

The repository consists of ARM templates for deploying Rancher and a host VM for Kubernetes. NOTE: These templates are intended for labs to learn Rancher. They are not intended for use in production.

In the repo ARM Template #1 named RancherNode.JSON will deploy an Ubuntu VM with Docker and the latest version of Rancher (https://hub.docker.com/r/rancher/rancher) from Docker Hub. ARM Template #2 named RancherHost.JSON will deploy an Ubuntu VM with Docker to be used as a Kubernetes host in Rancher.

Node Deployment

Deploy the RancherNode.JSON ARM template to your Azure subscription through “Template Deployment” or other deployment method. You will be prompted for the following info shown in the screenshot:

Host Deployment

Deploy the RancherHost.JSON ARM template to your Azure subscription through “Template Deployment” or other deployment method. Note that that should deploy this into the same Resource Group that you deployed the Rancher Node ARM template into. You will be prompted for the following info shown in the screenshot:

After the Rancher Node and Rancher Host ARM templates are deployed you should see the following resources in the new Resource Group:

NameType
RancherVNet Virtual network
RancherHost Virtual machine
RancherNode Virtual machine
RancherHostPublicIP Public IP address
RancherNodePublicIP Public IP address
RancherHostNic Network interface
RancherNodeNic Network interface
RancherHost_OSDisk Disk
RancherNode_OSDisk Disk

Next navigate the Rancher portal in the web browser. The URL is the DNS name of the Rancher Node VM. You can find the DNS name by clicking on the Rancher Node VM in the Azure portal on the overview page. Here is an example of the URL:

https://ranchernode.centralus.cloudapp.azure.com

The Rancher portal will prompt you to set a password. This is shown in the following screenshot.

After setting the password the Rancher portal will prompt you for the correct Rancher Server URL. This will automatically be the Rancher Node VM DNS name. Click Save URL.

You will then be logged into the Rancher portal. You will see the cluster page. From here you will want to add a cluster. Doing this is how you add a new Kubernetes cluster to Rancher. In this post I will show you how to add a cluster to the Rancher Host VM. When it’s all said and done Rancher will have successfully deployed Kubernetes to the Rancher Host VM. Note that you could add a managed Kubernetes such as AKS but we won’t do that in this blog. I will save that for a future blog post!

Click on Add Cluster

Under “From my own existing nodes” Click on custom, give the cluster a name and click Next.

Next check all the boxes for the Node Options since all the roles will be on a single Kubernetes cluster. Copy the code shown at the bottom of the page, click done and run the code on the Rancher Host.

In order to run the code on the Rancher Host you need to SSH in and run it from there. To do this follow these steps:

  1. In the Azure Portal, from within the resource group click on the Rancher Host VM.
  2. On the Overview page click on Connect.
  3. Copy “ssh ranchuser@rancherhost.centralus.cloudapp.azure.com” from the Connect to virtual machine pop up screen.
  4. Open a terminal in either Azure cloud shell or with something like a terminal via VS Code and past the “ssh ranchuser@rancherhost.centralus.cloudapp.azure.com” in.

Running the code will look like this:

When done you can run Docker PS to see that the Rancher agent containers are running.

In the Rancher portal under clusters you will see the Rancher host being provisioned

The status will change as Kubernetes is deployed.

Once it’s done provisioning you will see your Kubernetes cluster as Active.

From here you can see a bunch of info about your new Kubernetes cluster. Also notice that you could even launch Kubectl right from hereand start running commands! Take some time to click around to see all the familiar stuff you are used to working with in Kubernetes. This is pretty cool and simplifies the management experience for Kubernetes. 

If you want to add more nodes or need the configuration code again just click the ellipsis button and edit.

In Edit Cluster you can change the cluster name, get and change settings and copy the code to add more VMs to the cluster.

That’s the end of this post. Thanks for reading. Check back for more Azure, Kubernetes, and Rancher blog posts.

Read More

Where to host Docker Containers on Azure (AKS, ASE, or ASF)?

Azure Kubernetes Service (AKS) service Azure App Service Environment (ASE) Azure Service Fabric (ASF) Comparison

Scenario:

So, your team recently has been tasked with developing a new application and running it. The team made the decision to take a microservices based approach to the application. Your team also has decided to utilize Docker containers and Azure as a cloud platform. Great, now it’s time to move forward right? Not so fast. There is no question that Docker containers will be used, but what is in question is where you will run the containers. In Azure containers can run on Azure’s managed Kubernetes (AKS) service, an App Service Plan on Azure App Service Environment (ASE), or Azure Service Fabric (ASF). Let’s look at each one of these Azure services including an overview, pro’s, cons, and pricing.

This Azure Kubernetes Service (AKS) Pros and Cons chart is clickable.
This Azure App Service Environment (ASE) Pros and Cons chart is clickable.
This Azure Service Fabric (ASF) Pros and Cons chart is clickable.

Conclusion:

Choose Azure Kubernetes Service if you need more control, want to avoid vendor lock-in (can run on Azure, AWS, GCP, on-prem), need features of a full orchestration system, flexibility of auto scale configurations, need deeper monitoring, flexibility with networking, public IP’s, DNS, SSL, need a rich ecosystem of addons, will have many multi-container deployments, and plan to run a large number of containers. Also, this is a low cost.

Choose Azure App Service Environment if don’t need as much control, want a dedicated SLA, don’t need deep monitoring or control of the underlying server infrastructure, want to leverage features such as deployment slots, green/blue deployments, will have simple and a low number of multi-container deployments via Docker compose, and plan to run a smaller number of containers. Regarding cost, running a containerized application in an App Service Plan in ASE tends to be more expensive compared to running in AKS or Service Fabric. The higher cost of running containers on ASE is because with an App Service Plan on ASE, you are paying costs for a combination of resources and the managed service. With AKS and ASF you are only paying for the resources used.

Choose Service Fabric if you want a full micros services platform, need flexibility now or in the future to run in cloud and or on-premises, will run native code in addition to containers, want automatic load balancing, low cost.

A huge thanks to my colleague Sunny Singh (@sunnys101) for giving his input and reviewing this post. Thanks for reading and check back for more Azure and container contents soon.

Read More

Monitoring Azure Kubernetes Service (AKS) with Azure Monitor & Log Analytics

Part of running Kubernetes is being able to monitoring the cluster, the nodes, and the workloads running in it. Running production workloads regardless of PaaS, VM’s, or containers requires a solid level of reliability. Azure Kubernetes Service comes with monitoring provided from Azure bundled with the semi-managed service. Kubernetes also has built in monitoring that can also be utilized.

It is important to note that AKS is a free service and Microsoft aims to achieve at least 99.5% availability for the Kubernetes API server on the master node side.

But due to AKS being a free service Microsoft does not carry an SLA on the Kubernetes cluster service itself. Microsoft does provide an SLA for the availability of the underlying nodes in the cluster via the Azure Virtual Machines SLA. Without an official SLA for the Kubernetes cluster service it becomes even more critical to understand your deployment and have the right monitoring tooling and plan in place so when an issue arises the DevOps or CloudOps team can address, investigate, and resolve any issues with the cluster.

The monitoring service included with AKS gives you monitoring from two perspectives including the first one being directly from an AKS cluster and the second one being all AKS clusters in a subscription. The monitoring looks at two key areas “Health status” and “Performance charts” and consists of:

Insights – Monitoring for the Kubernetes cluster and containers.

Metrics – Metric based cluster and pod charts.

Log Analytics – K8s and Container logs viewing and search.

Azure Monitor

Azure Monitor has a containers section. Here is where you will find a health summary across all clusters in a subscription including ACS. You also will see how many nodes and system/user pods a cluster has and if there are any health issues with the a node or pod. If you click on a cluster from here it will bring you to the Insights section on the AKS cluster itself.

If you click on an AKS cluster you will be brought to the Insights section of AKS monitoring on the actual AKS cluster. From here you can access the Metrics section and the Logs section as well as shown in the following screenshot.

Insights

Insights is where you will find the bulk of useful data when it comes to monitoring AKS. Within Insights you have these 4 areas Cluster, Nodes, Controllers, and Containers. Let’s take a deeper look into each of the 4 areas.

Cluster

The cluster page contains charts with key performance metrics for your AKS clusters health. It has performance charts for your node count with status, pod count with status, along with aggregated node memory and CPU utilization across the cluster. In here you can change the date range and add filters to scope down to specific information you want to see.

Nodes

After clicking on the nodes tab you will see the nodes running in your AKS cluster along with uptime, amount of pods on the node, CPU usage, memory working set, and memory RSS. You can click on the arrow next to a node to expand it displaying the pods that are running on it.

What you will notice is that when you click on a node, or pod a property pane will be shown on the right hand side with the properties of the selected object. An example of a node is shown in the following screenshot.

Controllers

Click on the Controllers tab to see the health of the clusters controllers. Again here you will see CPU usage, memory working set, and memory RSS of each controller and what is running a controller. As an example shown in the following screenshot you can see the kubernetes dashboard pod running on the kubernetes-dashboard controller.

The properties of the kubernetes dashboard pod as shown in the following screenshot gives you information like the pod name, pod status, Uid, label and more.

You can drill in to see the container the pod was deployed using.

Containers

On the Containers tab is where all the containers in the AKS cluster are displayed. An as with the other tabs you can see CPU usage, memory working set, and memory RSS. You also will see status, the pod it is part of, the node its running on, its uptime and if it has had any restarts. In the following screenshot the CPU usage metric filter is used and I am showing a containers that has restarted 71 times indicating an issue with that container.

 In the following screenshot the memory working set metric filter is shown.

You can also filter the containers that will be shown through using the searching by name filter.

You also can see a containers logs in the containers tab. To do this select a container to show its properties. Within the properties you can click on View container live logs (preview) as shown in the following screenshot or View container logs. Container log data is collected every three minutes. STDOUT and STDERR is the log output from each Docker container that is sent to Log Analytics.

Kube-system is not currently collected and sent to Log Analytics. If you are not familiar with Docker logs more information on STDOUT and STDERR can be found on this Docker logging article here:  https://docs.docker.com/config/containers/logging.

Read More

Deploy MySQL and WordPress on Azure Kubernetes Service (AKS)

In this blog post I am going to walk through the steps for deploying WordPress to Azure Kubernetes Service (AKS) using MySQL and WordPress Docker images. Note that using the way I will show you is one way. Another way to deploy WordPress to AKS would be using a Helm Chart. Here is a link to the WordPress Helm Chart by Bitnami https://bitnami.com/stack/wordpress/helm. Here are the images we will use in this blog post:
MySQL WordPress
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
– port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
accessModes:
– ReadWriteOnce
resources:
requests:
storage: 20Gi

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
– image: mysql:5.6
name: mysql
env:
– name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
– containerPort: 3306
name: mysql
volumeMounts:
– name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
– name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
– port: 80
selector:
app: wordpress
tier: frontend
type: LoadBalancer

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
accessModes:
– ReadWriteOnce
resources:
requests:
storage: 20Gi

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
– image: wordpress:4.8-apache
name: wordpress
env:
– name: WORDPRESS_DB_HOST
value: wordpress-mysql
– name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
– containerPort: 80
name: wordpress
volumeMounts:
– name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
– name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim

The first thing we need to do is save these files as mysql-deployment.yaml and wordpress-deployment.yaml respectively.

Next, we need to setup a password for our MySQL DB. We will do this by creating a secret on our K8s cluster. To do this launch the bash or PowerShell in Azure cloud shell like in the following screenshot and run the following syntax:

kubectl create secret generic mysql-pass –from-literal=password=YOURPASSWORDHERE

Note: Replace “PASSWORDHERE” in the syntax with your own password.

The secret is now created. To ensure it was created you can run the following syntax to list the secrets:

kubectl get secrets

You also can see the secret in the Kubernetes dashboard as shown in the following screenshot.

Next the mysql-deployment.yaml and wordpress-deployment.yaml files from the beginning of this post need to be uploaded to Azure cloudrive storage.

You can also do this in the Cloudshell as shown in the following screenshot.

Run ls in the shell to make sure the files are on your clouddrive.

You will need your home drive. Mine was. /home/steve. To see this, click on Download. It will show you what yours is.

Next create the MySQL Pod and service by running the following syntax.

kubectl apply -f /home/steve/mysql-deployment.yaml

NOTE: You could use kubectl create /home/steve/mysql-deployment.yaml instead of apply to create the MySQL pod and service. I use apply because I typically use the declarative object configuration approach. kubectl apply essentially equals kubectl create + kubectl replace. In order to update an object after it has been created using kubectl create you would need to run kubectl replace.

There are pros and cons to using each and it is more of a preference for example when using the declarative approach there is no audit trail associated with changes. For more information on the multiple Kubernetes Object Management approaches go here: https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview.

Note that in the mysql yaml file it has syntax to create a persistent volume. This is needed so that the database stays in tact even if the pod fails, is moved etc. You can check to ensure the persistent volume was created by running the following syntax:

kubectl get pvc

Also, you can run the following syntax to verify the mysql pod is running:

kubectl get pods

Deploying the WordPress Pod and service is the same process. Use the following syntax to create the WordPress pod and service:

kubectl apply -f /home/steve/wordpress-deployment.yaml

Again, check to ensure the persistent volume was created. Use the following syntax:

kubectl get pvc

NOTE: When checking right after you created the persistent volume it may be in a pending status for a while like shown in the following screenshot:

You can also check the persistent volume using the K8s dashboard as shown in the following screenshot:

With the deployment of MySQL and WordPress we created 2 services. The MySQL service has a clusterip that can only be accessed internally. The WordPress service has an external IP that is also attached to an Azure Load Balancer for external access. I am not going to expand on what Kubernetes services are in this blog post but know that they are typically used as an abstracted layer in K8s used for access to Pods on the backend and follow the Pods regardless of the node they are running on. For more information about Kubernetes services visit this link: https://kubernetes.io/docs/concepts/services-networking/service.

In order to see that the services are running properly and find out the external IP you can run the following syntax:

kubectl get services (to see all services)

or

kubectl get services wordpress (to see just the WordPress service)

You also can view the services in the K8s dashboard as shown in the following screenshot:

Well now that we have verified the pods and the services are running let’s check out our new WordPress instance by going to the external IP in a web browser.

Thanks for checking out this blog post. I hope this was an easy to use guide to get WordPress up and running on your Azure Kubernetes Service cluster. Check back soon for more Azure and Kubernetes/Container content.

Read More

Getting Started with Azure Kubernetes Service (AKS)

Azure Kubernetes Service (AKS) is a fully managed Kubernetes (K8s) offering from Microsoft on the Azure platform. AKS reduces the management overhead of running your own K8s instance while still being able to take full advantage of Container Orchestration. Microsoft takes care of the K8s health monitoring and maintenance. With AKS you only manage the agent nodes while Microsoft manages the master nodes. Also with AKS you get integration to many of the Azure services such as load balancers, RBAC, Azure storage etc.

In this blog post I am going to walk through the setup of an AKS cluster step by step. This is to serve as a intro to AKS to show how easy it is to get started with Kubernetes in Azure. In a follow up blog post I will dive into AKS more showing how to deploy an instance MySQL and WordPress containers on AKS. Before we get into the setup of AKS there are a few things to note:

  1. With the AKS managed service you only pay for the agent nodes within your AKS cluster. There is no cost for the master nodes and the managed service itself is free.
  2. At the time of this blog post AKS only supports Linux containers. There is a work around for this until Windows nodes and containers come to AKS.
  3. AKS is only available in the following Azure regions:
    -Australia East
    -Canada Central
    -Canada East
    -Central US
    -East US
    -East US2
    -Japan East
    -North Europe
    -Southeast Asia
    -UK South
    -West Europe
    -West US
    -West US 2
  4. The Kubernetes API server is exposed as a public fully qualified domain name (FQDN). Access should be restricted on this. It can be restricted using K8s RBAC and AAD.

Deploy AKS

Housekeeping is done, now let’s get into the deployment of AKS. The first thing you need to do within the Azure portal is go to Create a resource and search on Kubernetes. Select the Kubernetes Service.

Click on create.

You will now see the setup. The setup consists of the following sections shown in the following screenshot:

Let’s walk through each section.

Basics

Here you need to give your AKS instance a name, select the region, K8s version, DNS prefix, and number of nodes and count.

Authentication

Kubernetes has its own RBAC within its authentication and authorization system. Azure Active Directory (AAD) can be integrated with this for authentication. Once the AAD and K8s integration is setup AAD users can be used for Kubernetes role-based access control (RBAC) to cluster resources. Select yes to enable RBAC and integration with AAD.

It is recommended to setup your own service principle in AAD. For this blog post I let the deployment create one. The service principle is used by K8s for managing Azure cloud resources attached to the cluster. The service principle interacts with Azure APIs. For example when you setup a load balancer service in K8s AKS creates and Azure load balancer. The service principle is what is used for authentication to create the load balancer.

Networking

In this section you chose what you want for networking with AKS. If you select basic AKS will create all needed VNets, Subnets, NSG’s etc. AKS clusters cannot use the following ranges 169.254.0.0/16, 172.30.0.0/16, and 172.31.0.0/16. If you select advanced you can chose an existing VNet or create a new one specifying the subnet, IP range and DNS settings etc. You would select Advanced if you need more control over the virtual networking. 

HTTP application routing is used to make application endpoints publicly accessible in the AKS cluster. Enabling this essentially configures an Ingress controller in the AKS cluster. When getting started with AKS I recommend leaving this disabled and doing more research on K8s Ingress Controllers here https://kubernetes.io/docs/concepts/services-networking/ingress as there are other options making applications publicly accessible. In the meantime while getting started with AKS you can use the load balancer service type for external access to your applications running on AKS.

Monitoring

With AKS you have the option to utilize Container monitoring from Azure Monitor. This will give you performance and health monitoring. The monitoring data comes directly from an AKS cluster or from all the AKS clusters via Azure Monitor more specifically Log Analytics. In the future I plan to post a deeper blog about monitoring AKS.

If you chose to enable this you will need to setup a new  Log Analytics workspace or use an existing one.

Tags

You can set tags for the AKS cluster.

Create

After all the sections are completed the new AKS will need to validate. After it is validated click on Create.

Exploring AKS

After the AKS cluster is created you will see it in Azure under Kubernetes services.

Also you may notice two new resource groups in your Azure subscription. The first resource group will be the one you created during the AKS creation. This is the resource group that will contain the Azure K8s cluster service. If you selected an advanced network configuration during deployment to create a new VNet you will see that as well.

You will also see a second resource group with a name format similar to MC_ResourceGroupNAME_AKSClusterNAME_REGION. As shown in the following screenshot I have a resource group named MC_AKS12118RG_AKS12118_centralus. This resource group contains the individual AKS cluster resources such as the nodes.

This resource group also contains supporting Azure services like DNS, public IP’s, storage, load balancers, network security groups and more. Note do not make changes to the resources in this resource group directly. You should only make changes through the AKS service and K8s itself. For example when you deploy a new load balancer service in K8s the corresponding Azure load balancer will automatically be created.

Access Kubernetes Dashboard

Next you can access the K8s cluster via a shell or access the dashboard. Before you can access the dashboard the service principle account that was created during the AKS deployment will need a ClusterRoleBinding that assigns the K8s role dashboard-admin it. Run the following syntax from the Azure cloud shell to do this:

kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

If the service principle account does not have the K8s dashboard-admin role you will see the following error when accessing the dashboard:

After the service principle account is all set go ahead and run the following syntax from Azure cloud shell.

az aks browse --resource-group CLUSTERRESOURCEGROUPNAME --name NAMEOFTHEAKSCLUSTER

Running that syntax will output a URL similar to the one shown in the following screenshot. Click on this and it will open the K8s dashboard in a new browser tab.

The following 3 screenshots show some of the K8s dashboard.

Read More

Azure Cost Management (Cloudyn)

IT financial management (ITFM) is an important part of IT operations as business dependency on IT continues to grow in the age of digital transformation. ITFM is a part of ITIL as a Service Strategy element in the framework. ITFM is a key part of CloudOps as well because spending in the cloud is based on an OPEX model and every single cost is tracked. ITFM and cost management in the cloud should be used to effectively and concisely connect the dollars spent on IT to the value delivered to the business. We can do this with Azure Cost Management. In this post, I am going to give an overview of Azure cost management highlighting many of the things you can do with it. Let’s dive into the solution now.

Overview

In June of 2017, Microsoft acquired Cloudyn a startup that had tooling for cloud monitoring and analytics tools focused on cloud financial management. Cloudyn’s solution is multi-cloud covering Azure, Azure Stack, AWS, and GCP. Through the acquisition of Cloudyn Microsoft was able to bring the tooling into the Azure ecosystem giving Azure customers an enhanced way to track and control cloud spend improving the improving the Azure cloud governance story.  As of right now, there is a free level and a paid level for Azure cost management. The following table lists what features are available with each level.

FREE capabilities:
Reporting Report on cost and usage
Data enrichment Categorize by resource tags
Budgets Create and manage cost and usage budgets
Alerting Create alerts on cost and usage budgets
Recommendations Eliminate idle cloud resources

Right-size cloud resources

PAID capabilities:
Chargeback features including cost markup, redistribution, and custom charges
Import external budgets
Customize recommendation thresholds
Categorize costs with custom meta-tags

Since the acquisition, Microsoft has added a link to the Cloudyn portal directly in Azure and integration with your Azure subscriptions giving you the ability to launch a new Cloudyn account that is tied to your subscription. Microsoft added Cost Management in Azure and this is where you will find Cloudyn and sign up. As shown in the following screenshot you can see the “Go to Cost Management” button. After clicking on that you will go the Cloudyn portal and will be able to add your various cloud accounts.  The thing that I really like about Azure cost management is that there is a ton of data and dashboards that are available right out of the box after adding a cloud account. There is not a bunch of configuration that you need to do to get the default dashboards and optimization tools.

After you are all signed up and have your cloud accounts added your dashboards will start to show data. The next two screenshots show a couple of the default dashboards.

The management dashboard gives a good summary of your cloud financials on one pane of glass.

 

The cost controller dashboard shows cost trends, some forecasting info, a breakdown of costs and more.

As you can see from the previous screenshots there are several other dashboards with other content. You can modify any of these dashboards adding or removing widgets. You also can create your own dashboard adding whatever widgets you want to it.

In Azure cost management, you can add cost centers known as Cost Entities. Entities are intended to mirror your organization’s hierarchical structure such as business units, divisions, departments, or teams within your organization some examples are engineering, R&D, development, marketing etc. The goal of the entities is to give you a way to track cloud spend by the entities. Keep in mind the cost entities can be anything that fits the way you want to structure and track cloud costs. You also can leverage tags, add budgets, and then associate costs and or budgets to the cost entities into cost models. Cost models give you a way to distribute and allocate costs. You can track costs back to these cost entities and you can track costs against budgets for showback or chargeback scenarios. Below is a screenshot of the cost entities screen. Keep an eye out for a detailed blog from me walking through how to structure and set up this part of Azure cost management. This area of Azure cost management warrants its own dedicated blog.

Here is an example of a budget set on a cost entity.

Read More

Azure Mobile App

Microsoft has a mobile Azure mobile app for Android and IOS. At first I was skeptical about the need of a mobile app for cloud but I found myself actually using it a few times for various tasks that I did not want to log onto my computer to do. In this blog post I am going to give one example. Before I jump into the example let’s explore the app.

First off you can load the app from Itunes or Google Play. You also can check it out here: https://azure.microsoft.com/en-us/features/azure-portal/mobile-app/ and here: https://play.google.com/store/apps/details?id=com.microsoft.azure&hl=en.

With the app in general you can see your Azure resources, their metrics, their health along with alerts, and diagnose and fix some issues through some actions you can perform on the resources via the mobile app. Some of the actions you can perform are Restart a web app or connect to a VM. Something else you can do with the app is access the Azure cloud shell. It supports Bash and PowerShell.  The following are some screenshots from the app.

Here is the app on my Android:

After the app launches for the first time you will be prompted to log into your subscription. Once you are logged in you will see all of your resources.

You can actually click on the filter icon to scope down to a specific type of resources.

The last screenshot here is of the Azure cloud shell in the mobile app.

Now lets talk about one reason you may use the app. I host an Azure user group website on WordPress on Azure. I have an availability monitor in Application Insights monitoring the site. If the site goes down I get an email from Application Insights as shown in the following screenshot.

I also get a notification in the UG board Slack channel by Logic Apps if the site is down. Well one day I got the notification from Slack on my phone.

I was not at my computer and did not want to go to it just to see what was going on with the site. I checked and sure enough the site was down.

Instead of logging onto my computer to troubleshoot I just used the app on my phone. Logging in I was able to see the site was up.

After clicking on the web app I was able to quickly restart it. It was up after that and I did it all from my phone.

I know restarting a web app is a basic thing. It saves time not having to log all the way into a computer to do this. I recommend trying out the mobile app. You never know when it might come in handy for a quick way to get info about one of your Azure resources and even help you troubleshoot something.

Read More

Setup CI/CD pipeline with VSTS & Azure Stack

We all know that DevOps brings together people, processes, and technology. In the Microsoft DevOps world A large part of the technology piece is utilizing Visual Studio Team Services (VSTS) for continuous deployment of workloads to Azure.

Microsoft launched their Hybrid Cloud on July 10th 2017. Azure Stack is the secret sauce of Microsoft’s the Hybrid Cloud. Microsoft’s offering is the only one true Hybrid Cloud in the market bringing Azure to on-premises data centers.

As Microsoft continues to move their Hybrid Cloud forward the DevOps integration and capabilities we have for Azure extend to Azure Stack. Again I was fortunate to participate in a preview of the VSTS integration with Azure Stack. I was happy to see Microsoft putting a priority on this functionality because DevOps on Azure Stack is a HUGE need. Cloud is often the catalyst to helping organizations adopt a DevOps culture fostering digital transformation. Some organizations not being able to put all workloads in public cloud Azure Stack is a good way for them to get the same cloud capabilities on-premises DevOps integration being one of them. The setup and integration between VSTS and Azure Stack is working nicely. The team at Microsoft has given me permission to share about this topic via my blog.

In this blog post I am going to cover setting up VSTS to work with Azure and setting up a continuous-integration and-continuous deployment (CI/CD) pipeline to Azure Stack. With Microsoft DevOps you can utilize the pieces of VSTS that make sense for you to use leaving the control up to you. Through VSTS you can use many other DevOps tools such as Jenkins, Octopus deploy, GitHub, Bitbucket etc into your pipeline making Azure Stack just as flexible as Azure is. Let’s Jump in!

Steps to prep Azure Stack for Visual Studio Team Services (VSTS)

#1 Ensure you have installed the Azure Stack PowerShell and Azure PowerShell modules.

Details can be found here:

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-powershell-install

#2 Add the Azure Stack environment using the following syntax

# Navigate to the downloaded folder and import the **Connect** PowerShell module

Set-ExecutionPolicy RemoteSigned

Import-Module PATH\AzureStack.Connect.psm1

# Register an AzureRM environment that targets your Azure Stack instance

Add-AzureRMEnvironment `

-Name “AzureStackAdmin” `

-ArmEndpoint “https://adminmanagement.local.azurestack.external

# Set the GraphEndpointResourceId value

Set-AzureRmEnvironment `

-Name “AzureStackAdmin” `

-GraphAudience “https://graph.windows.net/

# Get the Active Directory tenantId that is used to deploy Azure Stack

$TenantID = Get-AzsDirectoryTenantId `

-AADTenantName “YOURDOMAIN.onmicrosoft.com” `

-EnvironmentName “AzureStackAdmin”

# Sign in to your environment

Login-AzureRmAccount `

-EnvironmentName “AzureStackAdmin” `

-TenantId $TenantID

NOTE: You will need the environment name and the tenant ID for the next script.

#3 Create SPN

Original SPN creation script can be found here:

https://github.com/Microsoft/vsts-rm-documentation/blob/master/Azure/SPNCreation.ps1

Documentation on creating an SPN can be found here:

https://www.visualstudio.com/en-us/docs/build/concepts/library/service-endpoints#sep-azure-rm

Below I will display the script I used. Note that you will need the following parameters for the script:

$subscriptionName

“Enter Azure Stack Subscription name. You need to be Subscription Admin to execute the script”)]

$password

“Provide a password for SPN application that you would create”

$environmentName

“Provide Azure Stack environment name for your subscription”

$AzureStackTenantID

“Provide tenant ID from when Azure Stack enviroment was added”

EXAMPLE:

.\CreateSPN.ps1 -subscriptionName “Default Provider Subscription” -password PASSWORDHERE -environmentName AzureStackAdmin -AzureStackTenantID ID HERE

Here is the script I used that you can run:

param

(

[Parameter(Mandatory=$true, HelpMessage=”Enter Azure Stack Subscription name. You need to be Subscription Admin to execute the script”)]

[string] $subscriptionName,

[Parameter(Mandatory=$true, HelpMessage=”Provide a password for SPN application that you would create”)]

[string] $password,

[Parameter(Mandatory=$false, HelpMessage=”Provide a SPN role assignment”)]

[string] $spnRole = “owner”,

[Parameter(Mandatory=$false, HelpMessage=”Provide Azure Stack environment name for your subscription”)]

[string] $environmentName,

[Parameter(Mandatory=$false, HelpMessage=”Provide tenant ID from when Azure Stack enviroment was added”)]

[string] $AzureStackTenantID

)

#Initialize

$ErrorActionPreference = “Stop”

$VerbosePreference = “SilentlyContinue”

$userName = $env:USERNAME

$newguid = [guid]::NewGuid()

$displayName = [String]::Format(“VSO.{0}.{1}”, $userName, $newguid)

$homePage = “http://” + $displayName

$identifierUri = $homePage

#Initialize subscription

$isAzureModulePresent = Get-Module -Name AzureRM* -ListAvailable

if ([String]::IsNullOrEmpty($isAzureModulePresent) -eq $true)

{

Write-Output “Script requires AzureRM modules to be present. Obtain AzureRM from https://github.com/Azure/azure-powershell/releases. Please refer https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/DeployAzureResourceGroup/README.md for recommended AzureRM versions.” -Verbose

return

}

Import-Module -Name AzureRM.Profile

Write-Output “Provide your credentials to access Azure subscription $subscriptionName” -Verbose

Login-AzureRmAccount -SubscriptionName $subscriptionName -EnvironmentName $environmentName -TenantId $AzureStackTenantID

$azureSubscription = Get-AzureRmSubscription -SubscriptionName $subscriptionName

$connectionName = $azureSubscription.SubscriptionName

$tenantId = $azureSubscription.TenantId

$id = $azureSubscription.SubscriptionId

#Create a new AD Application

Write-Output “Creating a new Application in AAD (App URI – $identifierUri)” -Verbose

$azureAdApplication = New-AzureRmADApplication -DisplayName $displayName -HomePage $homePage -IdentifierUris $identifierUri -Password $password -Verbose

$appId = $azureAdApplication.ApplicationId

Write-Output “Azure AAD Application creation completed successfully (Application Id: $appId)” -Verbose

#Create new SPN

Write-Output “Creating a new SPN” -Verbose

$spn = New-AzureRmADServicePrincipal -ApplicationId $appId

$spnName = $spn.ServicePrincipalName

Write-Output “SPN creation completed successfully (SPN Name: $spnName)” -Verbose

#Assign role to SPN

Write-Output “Waiting for SPN creation to reflect in Directory before Role assignment”

Start-Sleep 20

Write-Output “Assigning role ($spnRole) to SPN App ($appId)” -Verbose

New-AzureRmRoleAssignment -RoleDefinitionName $spnRole -ServicePrincipalName $appId

Write-Output “SPN role assignment completed successfully” -Verbose

#Print the values

Write-Output “`nCopy and Paste below values for Service Connection” -Verbose

Write-Output “***************************************************************************”

Write-Output “Connection Name: $connectionName(SPN)”

Write-Output “Subscription Id: $id”

Write-Output “Subscription Name: $connectionName”

Write-Output “Service Principal Id: $appId”

Write-Output “Service Principal key: <Password that you typed in>”

Write-Output “Tenant Id: $tenantId”

Write-Output “***************************************************************************”

Output should be similar to this:

You will use information from the Service Connection output in the next step.

Steps to configure Azure Stack as a Service Endpoint in VSTS

Log into your VSTS account at visalstudio.com

Navigate to one of your projects.

Go into Settings.

Click on Services.

Click on New Service Endpoint

A window will pop up. Click on “use full version of the endpoint dialog.”

Next input the needed data. This data comes from the Service Connection info that you copied.

You can put whatever you want in the Connection name and the Subscription Name. Note do not verify the connection. It will not succeed as VSTS cannot access your private Azure Stack yet. Click OK when done.

Setup build agent on Azure Stack host

Next you need to setup the build agent on the Azure Stack host. (Note: In this post I am using the ASDK.) From within VSTS download the Windows agent. Extract the download to a local folder.

Go to Security under your profile in VSTS.

Next add a Personal access token (PAT) for Azure Stack.

Copy the token. Note it will not be shown again ever after you leave this screen.

In the folder with the extracted build agent you will see the following. We need to run the run.cmd file from an elevated command prompt.

Here is a screenshot of running the run.cmd. I recommend deploying the build agent as a service. You will use your personal access token (PAT) here and the azure stack admin account.

After the run.cmd finished the folder with the extracted contents should look like the following:

You can now see the agent in VSTS.

That’s it for the setup for connecting VSTS to Azure Stack. Next let’s look at setting up a continuous-integration and-continuous deployment (CI/CD) pipeline for VM-deployment to Azure Stack.

 

THE BUILD

What I cover here is focused on infrastructure as code (IaC) using ARM templates. If you need to set up CI/CD to Azure Stack for Web Apps, Mobile Apps, Containers, etc the process is the same as it is on Azure with the only difference being that you point to Azure Stack. Also note that in this post I am using the ASDK not multi-node.

Within VSTS create a new repository and place your ARM template in it.

Next click on Build and Release. Create a new Build Definition.

In the build definition. Point the Get sources to the repository you just created. Add 2 tasks under Phase 1. The first task will copy the ARM template to the build staging directory. The second task will publish the ARM template so that a release definition can pick it up. Both tasks are shown in the following screenshots.

Copy Files to task

Publish Artifact task

OPTIONAL: To setup continuous integration click on Triggers. Here you can set a schedule to run the builds or you can click on the repository as shown in the screenshot and then check Enable continuous integration. By checking the box next to Enable continuous integration it tells VSTS that anytime content in the repo is changed to run a build.

Click on Save & queue. This will start the build.

The build will start. As long as everything is setup properly within your build it will succeed as shown in the following Screenshot.

That’s all for our build. Next up we need to create a release definition (RD) pipeline. The RD will take the build artifacts and deploy to an environment/s you specify.

Read More

Monitor Azure WebJobs Status with Application Insights

Within the Azure App Service is something called WebJobs that enables developers to run a script or program in the background within the same context as a web app, API app, or mobile app. Wejobs are included in app service with no extra cost. Webjobs are often used to run regular jobs and batch work as background services. Webjobs exist to make it easier to develop, run background tasks, and scale your web applications.

Webjobs have been around for a while and are considered a part of the serverless computing available on Azure. Today Azure Functions another newer and improved serveless technology service the evolution of WebJobs. When developers need serverless today Azure Functions is typically chosen over webjobs. There are certain cases and scenarios when webjobs are still used instead of Azure Functions and I will not be diving into that topic in this blog post. For more information on when to use what serverless technology on Azure check out the following links:

– A comparison between WebJobs and Functions: Choose between Flow, Logic Apps, Functions, and WebJobs.

– Minnesota’s Azure user group meeting from December 2017 covered comparing the various serverless technologies in Azure. It was presented by Joe Koletar. The meeting notes and PowerPoint download can be found here:

http://www.mnazureusergroup.com/2017/12/22/december-2017-meeting-serverless-computing-notes-and-download

For more information on Azure WebJobs check out these two links:

– Run Background tasks with WebJobs in Azure App Service

https://docs.microsoft.com/en-us/azure/app-service/web-sites-create-web-jobs

– Develop and deploy WebJobs using Visual Studio – Azure App Service

https://docs.microsoft.com/en-us/azure/app-service/websites-dotnet-deploy-webjobs

I recently needed to setup monitoring for Azure webjobs status. In this environment there was a mix of continuous webjobs along with some triggered webjobs. Monitoring WebJobs is different compared to monitoring other Azure App Services such as web apps. Web apps can easily be monitored for up/down status and performance for things like in/out traffic, usage, and errors. Background services like WebJobs does not have a defined start or end to the work they do. WebJobs either run continuously or for short amounts of time to perform a task. In this case performance was not a concern but the status of the WebJobs was needed. You can see the status of the WebJobs in the Azure portal as shown in the following screenshot.

The problem here is this is not on a monitoring dashboard, you have to navigate here to see it, you need to click the refresh button for an update, and there is no alert setup when the status is in a non-desired state.

WebJobs does come with a logs website that shows the status of all of your WebJobs and more. This logs site is shown in the following screenshot:

The logs site is nice but the issue with it is that you have to be on the site to see the status of the WebJobs along with the previously mentioned issues viewing the status in the Azure portal. A good solution for monitoring the WebJobs would be a way to check the heartbeat of the WebJobs, the status, and alert you if one of the WebJobs is in a non-desired state. The good news is that this can be accomplished utilizing Application Insights. This is not new but does take some effort to setup.  I am going to detail how to set this up. Here is a summary of what needs to be done.

  1. Need an instance of Application Insights
  2. Need an authorization header from the WebJobs REST API.
  3. Need to create a webtest manually or using Visual Studio enterprise.
  4. Create a multi-step availability test in the Application Insights instance utilizing the webtest file.
  5. Create an alert on the availability test to notify when a WebJob is in a non-desired state.
  6. Add the results of the WebJobs availability test to a dashboard in Azure.

Let’s get started.

Read More

5th Book Published! Azure Stack Book!

The latest book project I have be a part of has completed and recently published. Back in August in this blog post (http://www.buchatech.com/2017/08/azure-stack-book-coming-soon-training) I mentioned this book was on its way. It is a book about Azure Stack that was officially published on December 21, 2017 by Pearson publishing. This book release has been very exciting as it is a part of the Unleashed series and this one marks the 5th book I have published. Here is a screenshot of all 5 from my Amazon author page:

In total I have published 2 books on System Center Data Protection Manager, 2 books on System Center Service Manager, and now this book covering Microsoft’s Hybrid Cloud with Azure and Azure Stack. This book also comes at the right time as I recently made a transition to a new company (Avanade) with a new focus on Cloud (Azure/Azure Stack) and DevOps. 2018 and beyond look to be exciting times as I “Hit Refresh” on my career focus.

Books like this require a team effort. On this book I was honored to work with an expert team of authors. All of the authors are fellow Microsoft MVP’s. The other authors are: Kerrie Meyler,‎ Mark Scholman,‎ Jakob Gottlieb Svendsen,‎ Janaka Rangama. Me and the other authors are pictured below + a former Microsoft MVP Nirmal.

A part of the books team also included some members of the Azure Stack product group and Azure CAT team. We lucked out having Daniel Savage Principal PM Manager from the Azure Stack team write the foreword and Marc van Eijk Senior Program Manager from the Azure CAT team serve as our technical reviewer keeping us authors in line. 🙂

Each of us authors had so much to contribute and added much value across a variety of topics for Azure Stack. In this book I focused on bringing the readers into the cloud journey, showing the value of ITIL applied to cloud as well as the value of DevOps and then bringing ITIL and DevOps together applying them to Hybrid Cloud, took a deep dive into resource providers and management of Azure Stack through a CloudOps perspective.

Other topics covered in the book consist of preparing for Azure Stack deployments both with the development kit and integrated system, deep dive into the architecture of Azure Stack including the development kit and integrated system, data center integration with Azure Stack, configuring Azure Stack including delegation and for tenants, provisioning in Azure Stack, using OMS/DSC/VM extensions with Azure Stack, Customizing Azure Stack, automating in Azure Stack, and much more.

This book gives you the information you need around Azure Stack single and multi-node. It is a great place to start as you venture into the world of Microsoft Hybrid Cloud. The plan is to update this book as Microsoft continues to mature Azure Stack so this book will continue to be relevant.

Here is the book cover:

Here is the official description for the book:

“Microsoft Hybrid Cloud Unleashed brings together comprehensive and practical insights into hybrid cloud technologies, complete CloudOps and DevOps implementation strategies, and detailed guidance for deploying Microsoft Azure Stack in your environment.

Written by five Microsoft Cloud and Datacenter Management MVPs, this book is built on real-world scenarios and the authors’ extraordinary hands-on experiences as early adopters. Step by step, the authors help you integrate your optimal mix of private and public cloud, with a unified management experience that lets you move workloads at will, achieving unprecedented flexibility.

The authors also guide you through all aspects of building your own secure, high-performance hybrid cloud infrastructure. You’ll discover how Azure Stack enables you to run data centers with the same scalability, redundancy, and reliability as Microsoft’s Azure data centers; how to integrate Azure infrastructure and platform services with internal operations; and how to manage crucial external dependencies. The book concludes with a deep dive into automating and customizing Azure Stack for maximum reliability, productivity, and cost savings.

Detailed information on how to

  •     Run a private/hybrid cloud on your hardware in your data center, using APIs and code identical to public Azure
  •     Apply ITIL and DevOps lifecycles to your hybrid cloud implementation
  •     Gain a deep understanding of Azure Stack architecture, components, and internals
  •     Install and configure Azure Stack and master the Azure Stack Portal
  •     Integrate and utilize infrastructure, core, and custom resource providers
  •     Effectively provision, secure, and manage tenants
  •     Manage, monitor, troubleshoot, and back up Azure Stack with CloudOps
  •     Automate resource provisioning with PowerShell, the Azure CLI, templates, and Azure Stack’s API
  •     Write your own Azure Resource Manager templates
  •     Centrally automate cloud management and complex tasks connected to external systems
  •     Develop customized, production-ready Azure Stack marketplace items”

Here is a link to the book:

https://www.amazon.com/Microsoft-Hybrid-Cloud-Unleashed-Azure/dp/0672338505

Happy Azure Stacking!

Read More