Deploy a Ubuntu Based Flannel K8S Cluster in Azure with ARM Template and Kubeadm

The infomation migth be outdated here as acs-engine adds support for Flannel recently with PR 2967

However, if you want to gain more control on your kubernetes cluster in Azure, in our case, by using kubeadm, this article still applies.

0. Prerequisites

  • Azure subscription
  • An Azure account has sufficient permission to create a service principal

1. Create a service principal which will be used to manage azure resources in K8S cluster

Follow Install Azure CLI 2.0 install Azure CLI. From command prompt/shell, login and create a service principal by issueing below commands, replace YOUR_SUBSCRIPTION_ID with your Azure subscription ID.

az login
az account set --subscription "YOUR_SUBSCIPTION_ID"
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/YOUR_SUBSCRIPTION_ID"

Once the service principal get created, the result looks like below

{
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "azure-cli-2018-04-26-07-03-35",
  "name": "http://azure-cli-2018-04-26-07-03-35",
  "password": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy",
  "tenant": "zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz"
}

Record appId and password, they will be used as configured parameters in ARM template.

2. Customize your K8s deployment script

Azure Custom Script Extension will be ussed to install docker and kubeadm from ARM template. To do that, create a file called script.sh, paste below content into it

#!/bin/sh
apt-get update
apt-get install -y docker.io
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
cat <<EOF >/etc/kubernetes/kubeadm.conf
piVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
cloudProvider: azure
kubernetesVersion: 1.10.2
apiServerExtraArgs:
  cloud-provider: azure
  cloud-config: /etc/kubernetes/cloud-config
controllerManagerExtraArgs:
  cloud-provider: azure
  cloud-config: /etc/kubernetes/cloud-config
networking:
  podSubnet: 10.244.0.0/16
EOF
sed -i -E 's/(.*)KUBELET_KUBECONFIG_ARGS=(.*)$/\1KUBELET_KUBECONFIG_ARGS=--cloud-provider=azure --cloud-config=\/etc\/kubernetes\/cloud-config \2/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
/sbin/sysctl -p /etc/sysctl.conf

Azure Custom Script Extension requies a base64 encoded script to execute

The script must be base64 encoded. The script can optionally be gzip'ed.

Here is the command to generate a base64 encoded script with gzip enabled

cat script.sh | gzip -9 | base64 -w 0

The result will be below, copy/paste it to the ARM template in Step 3

H4sIAOcA8FoCA61UXW/UMBB8z69YCmoByfalqioU9U4q1bVCUIo44OmkyrHdnBVfbNnOtYf48Wzc3GdB9IGXrDy7npn1rvLyBSt1w8Is4y6SSkVoneRRrY+6CZEbA2QJ0opaeartXi0cHsIfyjsoet4EZ30ksxhdANF6k3UfIAESVDDmuKh5pQIVxraSVtZWRlFh5wwpGKp2kdRqSStXwS/oT8ClBJIJHuHsbHxzCSOmYqplwbZeIKHRIVLJ6rZUvlGxRzKpyiSO2lhNt9LaMtgcyYNqNDcw57rJUOEZb9RdNgh1kct5iiKaJy43IqwvxYabu8zpH8oHbZtiRUHrd8nXIufGzXie1bqRBVzzEJW/wDu6aj2PeCVL7/fF24WWyhfAf7ZeZRulNXNO8wE9Rvt6ovxC+fEDzuncV6HIABIJcXssK1wkwQL2m9jOZhiit8Yof80bnOx/F0Do3np8iKojdFZO2hIx7Az7OjmhAzpg+WkaWVC4JBrIGI4Ce03fvvn4/f340/jbbRcvbj5ffri6Pf96NRl2uVdsmv8tT8iu8WHyDSv40dlw2vmebhmf7jiH6TE7euwtLHGAc9lH1i8ODTgRLRRubT4gO6uhxMzCAZLS0mtZqT6Q5o4I3D6iXeSlUQGGkB/AaLSWwf17LsPpvyhYSL+LhABxTwuy32x+HQVSBAAA

3. ARM template

Create a file called template.json, paste below content into it

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "numberOfInstances": {
      "type": "int",
      "defaultValue": 2,
      "metadata": {
        "description": "Number of VM instances to create, default is 2"
      }
    },
    "adminUsername": {
      "type": "string",
      "defaultValue": "Admin username>",
      "metadata": {
        "description": "Admin username for the Virtual Machines"
      }
    },
    "sshKeyData": {
      "type": "string",
      "defaultValue": "SSH public key>",
      "metadata": {
        "description": "SSH public key for the Virtual Machines"
      }
    },
    "imagePublisher": {
      "type": "string",
      "defaultValue": "Canonical",
      "metadata": {
        "description": " Publisher for the OS image, the default is Canonical"
      }
    },
    "imageOffer": {
      "type": "string",
      "defaultValue": "UbuntuServer",
      "metadata": {
        "description": "The name of the image offer. The default is Ubuntu"
      }
    },
    "imageSKU": {
      "type": "string",
      "defaultValue": "16.04-LTS",
      "metadata": {
        "description": "Version of the image. The default is 16.04-LTS"
      }
    },
    "vmSize": {
      "type": "string",
      "defaultValue": "Standard_F2s",
      "metadata": {
        "description": "VM size"
      }
    },
    "aadClientId": {
      "type": "string",
      "metadata": {
        "description": "AAD client Id"
      }
    },
    "aadClientSecret": {
      "type": "string",
      "metadata": {
        "description": "AAD client secret"
      }
    }      
  },
  "variables": {
    "apiVersionCompute": "2016-03-30",
    "apiVersionStorage": "2017-10-01",
    "apiVersionNetwork": "2018-02-01",
    "apiVersionAvailabilitySet": "2017-12-01",
    "vmName": "[concat('k8snode-', uniqueString(resourceGroup().id))]",    
    "storageAccountName": "[concat('k8sstorage', uniqueString(resourceGroup().id))]",    
    "vnetName": "[concat('k8svnet-', uniqueString(resourceGroup().id))]",
    "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('vnetName'))]",
    "subnetName": "k8ssubnet",        
    "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]",
    "availabilitySetName": "[concat('k8savset-', uniqueString(resourceGroup().id))]",
    "publicIPAddressName": "[concat('k8spublicip-', uniqueString(resourceGroup().id))]",
    "publicIPAddressType": "Static",
    "networkSecurityGroupName": "[concat('k8snsg-', uniqueString(resourceGroup().id))]",
    "routeTableName": "[concat('k8sroutetable-', uniqueString(resourceGroup().id))]",
    "routeTableID": "[resourceId('Microsoft.Network/routeTables', variables('routeTableName'))]",
    "sshKeyPath": "[concat('/home/',parameters('adminUsername'),'/.ssh/authorized_keys')]",    
    "addressPrefix": "172.16.0.0/16",
    "subnetPrefix": "172.16.0.0/24",
    "nicName": "[concat('k8snicname-', uniqueString(resourceGroup().id))]",
    "cseName" : "[concat('k8scse-', uniqueString(resourceGroup().id))]"
  },
  "resources": [
    {
      "apiVersion": "[variables('apiVersionStorage')]", 
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[variables('storageAccountName')]",
      "location": "[resourceGroup().location]",
      "sku": {
          "name": "Standard_LRS"
      },
      "kind": "Storage",
      "properties": {}
    },
    {
      "apiVersion": "[variables('apiVersionNetwork')]", 
      "type": "Microsoft.Network/networkSecurityGroups",
      "name": "[variables('networkSecurityGroupName')]",
      "location": "[resourceGroup().location]",
      "properties": {
        "securityRules": [
          {
            "name": "SSH",
            "properties": {
              "description": "Allow inbound SSH port.",
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "22",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "allow_kube_tls",
            "properties": {
              "description": "Allow inbound SSH port.",
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "6443",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 300,
              "direction": "Inbound"
            }
          }          
        ]
      }
    },
    {
      "apiVersion": "[variables('apiVersionNetwork')]",
      "type": "Microsoft.Network/publicIPAddresses",
      "sku": {
        "name": "Basic",
        "tier": "Regional"
      },      
      "name": "[concat(variables('publicIPAddressName'), '-', copyIndex())]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "publicIPLoop",
        "count": "[parameters('numberOfInstances')]"
      },
      "properties": {
        "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
        "dnsSettings": {
          "domainNameLabel": "[concat(variables('vmName'), '-', copyIndex())]"
        }
      }
    },
    {
      "apiVersion": "[variables('apiVersionNetwork')]",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('vnetName')]",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[concat('Microsoft.Network/routeTables/', variables('routeTableName'))]",        
        "[concat('Microsoft.Network/networkSecurityGroups/', variables('networkSecurityGroupName'))]"
      ],      
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix')]"
          ]
        },
        "subnets": [
          {
            "name": "[variables('subnetName')]",
            "properties": {
              "addressPrefix": "[variables('subnetPrefix')]",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"
              },
              "routeTable": {
                "id": "[variables('routeTableID')]"
              }
            }
          }
        ]
      }
    },
    {
      "apiVersion": "[variables('apiVersionNetwork')]",
      "type": "Microsoft.Network/routeTables",      
      "location": "[resourceGroup().location]",
      "name": "[variables('routeTableName')]"
    },
    {
      "apiVersion": "[variables('apiVersionNetwork')]",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[concat(variables('nicName'), '-', copyIndex())]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "nicLoop",
        "count": "[parameters('numberOfInstances')]"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'), '-', copyIndex())]",
        "[concat('Microsoft.Network/virtualNetworks/', variables('vnetName'))]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', concat(variables('publicIPAddressName'), '-', copyIndex()))]"
              },
              "subnet": {
                "id": "[variables('subnetRef')]"
              }
            }
          }
        ]
      }
    },
    {
      "apiVersion": "[variables('apiVersionAvailabilitySet')]",
      "type": "Microsoft.Compute/availabilitySets",
      "sku": {
        "name": "Classic"
      },      
      "name": "[variables('availabilitySetName')]",
      "location": "[resourceGroup().location]",
      "properties": {
        "platformFaultDomainCount": 3,
        "platformUpdateDomainCount": 5
      }
    },    
    {
      "apiVersion": "[variables('apiVersionCompute')]",    
      "type": "Microsoft.Compute/virtualMachines/extensions",
      "name": "[concat(variables('vmName'), '-', copyIndex(), '/', variables('cseName'))]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "cseLoop",
        "count": "[parameters('numberOfInstances')]"
      },      
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'), '-', copyIndex())]"
      ],
      "properties": {
        "publisher": "Microsoft.Azure.Extensions",
        "type": "CustomScript",
        "typeHandlerVersion": "2.0",
        "autoUpgradeMinorVersion": false,
        "settings": {
          "script": "<Replace it with base64 output from step 2>"
        }
      }
    },
    {
      "apiVersion": "[variables('apiVersionCompute')]",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[concat(variables('vmName'), '-', copyIndex())]",
      "location": "[resourceGroup().location]",
      "copy": {
        "name": "vmLoop",
        "count": "[parameters('numberOfInstances')]"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), '-', copyIndex())]",
        "[concat('Microsoft.Compute/availabilitySets/', variables('availabilitySetName'))]",
        "[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]"
      ],
      "properties": {
        "availabilitySet": {
          "id": "[resourceId('Microsoft.Compute/availabilitySets', variables('availabilitySetName'))]"
        },
        "hardwareProfile": {
          "vmSize": "[parameters('vmSize')]"
        },
        "osProfile": {
          "computerName": "[concat(variables('vmName'), '-', copyIndex())]",
          "adminUsername": "[parameters('adminUsername')]",
		      "customData": "[base64(concat('#cloud-config\n\nwrite_files:\n- path: \"/etc/kubernetes/cloud-config\"\n  permissions: 0644\n  content: |\n    {\n    \"cloud\":\"AzurePublicCloud\",\n    \"tenantId\": \"', subscription().tenantId, '\",\n    \"subscriptionId\": \"', subscription().subscriptionId, '\",\n    \"aadClientId\": \"', parameters('aadClientId'), '\",\n    \"aadClientSecret\": \"', parameters('aadClientSecret'), '\",\n    \"resourceGroup\": \"', resourceGroup().name, '\",\n    \"location\": \"', resourceGroup().location, '\",\n    \"subnetName\": \"', variables('subnetName'), '\",\n    \"securityGroupName\": \"', variables('networkSecurityGroupName'), '\",\n    \"vnetName\": \"', variables('vnetName'), '\",\n    \"routeTableName\": \"', variables('routeTableName'), '\",\n    \"vnetResourceGroup\": \"\",\n    \"primaryAvailabilitySetName\": \"', variables('availabilitySetName'), '\",\n    \"cloudProviderBackoff\": false,\n    \"cloudProviderBackoffRetries\": 0,\n    \"cloudProviderBackoffExponent\": 0,\n    \"cloudProviderBackoffDuration\": 0,\n    \"cloudProviderBackoffJitter\": 0,\n    \"cloudProviderRatelimit\": false,\n    \"cloudProviderRateLimitQPS\": 0,\n    \"cloudProviderRateLimitBucket\": 0,\n    \"useManagedIdentityExtension\": false,\n    \"useInstanceMetadata\": true\n    }'))]",
          "linuxConfiguration": {
            "disablePasswordAuthentication": true,
            "ssh": {
              "publicKeys": [
                {
                  "path": "[variables('sshKeyPath')]",
                  "keyData": "[parameters('sshKeyData')]"
                }
              ]
            }
          }
        },
        "storageProfile": {
          "osDisk": {
            "osType": "Linux",
            "name": "[concat(variables('vmName'), '_', copyIndex(), '_osdisk')]",
            "vhd": {
              "uri": "[concat(reference(resourceId('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))).primaryEndpoints.blob, 'vhds/', variables('vmName'), '_', copyIndex(), '_osdisk.vhd')]"
            },            
            "createOption": "FromImage"
          },
          "imageReference": {
            "publisher": "[parameters('imagePublisher')]",
            "offer": "[parameters('imageOffer')]",
            "sku": "[parameters('imageSKU')]",
            "version": "latest"
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('nicName'), '-', copyIndex()))]"
            }
          ]
        }
      }
    }
  ]
}

Replace "script": "<Replace it with base64 output from step 2>" with the base64 output from step 2.

NOTE: this script will also genereate a configuration file /etc/kubernetes/cloud-config, the magic is in "customerData" part of above template, it uses cloud-init

4. Customize ARM template parameters

Create a file called params.json, modify all parametes to suit the needs, fill aadClientId with appId and aadClientSecret with password from Step 1's output.

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "adminUsername": {
      "value": "<Replace it with Admin username>"
    },
    "sshKeyData": {
      "value": "<Replace it with SSH public key>"
    },
    "numberOfInstances": {
      "value": 2
    },
    "imagePublisher": {
      "value": "Canonical"
    },
    "imageOffer": {
      "value": "UbuntuServer"
    },
    "imageSKU": {
      "value": "16.04-LTS"
    },
    "vmSize": {
      "value": "Standard_B2s"
    },
    "aadClientId": {
	    "value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "aadClientSecret": {
	    "value": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
	  }    
  }
}

5. Deploy ARM template

From shell, begin to deploy ARM template with commands below, replace resource group name and location to suit the needs.

az group create --name Flannel --location "SouthEast Asia"
az group deployment create --name FlannelDeployment --resource-group Flannel --template-file template.json --parameters @params.json

6. Deploy Flannel network with kubeadm

Once VMs get deployed successfully, SSH into the the first node k8snode-{uniquestring}-0, run below commands to deploy K8S

sudo -i
cd /etc/kubernetes
kubeadm init --config kubeadm.conf

kubeadm will output result similar to below if it succeeded without any errors

[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[init] WARNING: For cloudprovider integrations to work --cloud-provider must be set for all kubelets in the cluster.
	(/etc/systemd/system/kubelet.service.d/10-kubeadm.conf should be edited for this purpose)
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8snode-342zzth442uje-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s.arrteam.top k8snode-342zzth442uje-0.southeastasia.cloudapp.azure.com] and IPs [10.96.0.1 172.16.0.4]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8snode-342zzth442uje-0] and IPs [172.16.0.4]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 121.002020 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node k8snode-342zzth442uje-0 as master by adding a label and a taint
[markmaster] Master k8snode-342zzth442uje-0 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 1krztl.24cx0v0ybfpxuism
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.16.0.4:6443 --token 1krztl.24cx0v0ybfpxuism --discovery-token-ca-cert-hash sha256:d9b9bf2e300b99418003abf1560c5a22194493594dacb60179778a340a146e8f

Now, copy kubernetes cluster configuration to $HOME/.kube, essentially kubectl needs this config file to get cluster info.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Install Flannel by using below commands

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Addtionally, if other nodes need to be added into K8S cluster, we can login to k8snode-{uniquestring}-1, ..., k8snode-{uniquestring}-n node, run below commands

sudo -i
kubeadm join 172.16.0.4:6443 --token 1krztl.24cx0v0ybfpxuism --discovery-token-ca-cert-hash sha256:d9b9bf2e300b99418003abf1560c5a22194493594dacb60179778a340a146e8f