Categories
Uncategorized

Cleaning up Hard Drive Space in Linux

If you run into a situation where a Linux machine is running out of space, here’s a way to check what is taking up so much space and clean the server up.

Install ncdu:

sudo apt-get install ncdu

Afterwards, start ncdu and you’ll be presented with an interface that’ll walk you through the directories taking the most space:

Troubleshooting – My Drive is 100% Full

If you’re having trouble installing the above, you likely have a full drive that cannot do anything else. Here are a few steps that may help:

First, see if there are any individual files you can delete to clear some space (such as in the user directory). If this isn’t an option, go into /var/tmp and delete the files in there () this is not always a safe choice).

After that, run the following:

sudo apt-get clean
sudo apt install byobu
sudo purge-old-kernels
sudo apt autoremove
sudo update-grub

With this you should be able to install the above and check into what is taking up so much space.

Categories
Uncategorized

Getting Slack Notifications with Azure Monitor

An option when trying to get alerts from Azure Monitor (such as web tests, processing alerts and more) is to get them to a Slack channel to allow for an alternative to receiving alerts via email.

Setting up Slack

To start, you’ll need to have a Slack workspace with an available app. If needed, create the channel desired to receive notifications.

Create a new Slack app named “Azure Notifications” and turn on the “Incoming Webhooks” capability, which will provide a URL (webhook) that you’ll use later to receive the notifications from Azure.

Creating Conversion Logic App in Azure

Next, you’ll need to create a logic app in Azure that takes the alert provided from Azure and converts the message into something that can be displayed in Slack.

Create a logic app, open Code View and use the following (replacing the URI provided with the one above):

{
    "definition": {
        "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
        "actions": {
            "Http": {
                "inputs": {
                    "body": {
                        "text": "@{if(equals(triggerBody()?['data']?['essentials']?['monitorCondition'],'Resolved'), ':green_heart:', ':red_circle:')} @{triggerBody()?['data']?['essentials']?['alertRule']}"
                    },
                    "headers": {
                        "Content-Type": "application/json"
                    },
                    "method": "POST",
                    "uri": "YOUR_WEBHOOK_HERE"
                },
                "runAfter": {},
                "type": "Http"
            }
        },
        "contentVersion": "1.0.0.0",
        "outputs": {},
        "parameters": {},
        "triggers": {
            "manual": {
                "inputs": {
                    "schema": {
                        "properties": {
                            "data": {
                                "properties": {
                                    "alertContext": {
                                        "properties": {
                                            "condition": {
                                                "properties": {
                                                    "allOf": {
                                                        "items": {
                                                            "properties": {
                                                                "dimensions": {
                                                                    "items": {
                                                                        "properties": {
                                                                            "name": {
                                                                                "type": "string"
                                                                            },
                                                                            "value": {
                                                                                "type": "string"
                                                                            }
                                                                        },
                                                                        "required": [
                                                                            "name",
                                                                            "value"
                                                                        ],
                                                                        "type": "object"
                                                                    },
                                                                    "type": "array"
                                                                },
                                                                "metricName": {
                                                                    "type": "string"
                                                                },
                                                                "metricNamespace": {
                                                                    "type": "string"
                                                                },
                                                                "metricValue": {
                                                                    "type": "number"
                                                                },
                                                                "operator": {
                                                                    "type": "string"
                                                                },
                                                                "threshold": {
                                                                    "type": "string"
                                                                },
                                                                "timeAggregation": {
                                                                    "type": "string"
                                                                }
                                                            },
                                                            "required": [
                                                                "metricName",
                                                                "metricNamespace",
                                                                "operator",
                                                                "threshold",
                                                                "timeAggregation",
                                                                "dimensions",
                                                                "metricValue"
                                                            ],
                                                            "type": "object"
                                                        },
                                                        "type": "array"
                                                    },
                                                    "windowSize": {
                                                        "type": "string"
                                                    }
                                                },
                                                "type": "object"
                                            },
                                            "conditionType": {
                                                "type": "string"
                                            },
                                            "properties": {}
                                        },
                                        "type": "object"
                                    },
                                    "essentials": {
                                        "properties": {
                                            "alertContextVersion": {
                                                "type": "string"
                                            },
                                            "alertId": {
                                                "type": "string"
                                            },
                                            "alertRule": {
                                                "type": "string"
                                            },
                                            "alertTargetIDs": {
                                                "items": {
                                                    "type": "string"
                                                },
                                                "type": "array"
                                            },
                                            "description": {
                                                "type": "string"
                                            },
                                            "essentialsVersion": {
                                                "type": "string"
                                            },
                                            "firedDateTime": {
                                                "type": "string"
                                            },
                                            "monitorCondition": {
                                                "type": "string"
                                            },
                                            "monitoringService": {
                                                "type": "string"
                                            },
                                            "originAlertId": {
                                                "type": "string"
                                            },
                                            "resolvedDateTime": {
                                                "type": "string"
                                            },
                                            "severity": {
                                                "type": "string"
                                            },
                                            "signalType": {
                                                "type": "string"
                                            }
                                        },
                                        "type": "object"
                                    }
                                },
                                "type": "object"
                            },
                            "schemaId": {
                                "type": "string"
                            }
                        },
                        "type": "object"
                    }
                },
                "kind": "Http",
                "type": "Request"
            }
        }
    }
}

You can test by running the code directly, and see if you get a red circle:

Assuming this is in place, the final step is creating an action group that allows for setting alerts to send to the Slack channel.

Before going to the next step, take note of the webhook provided by the Logic App:

Setting up Action Group

The final step is setting up an action group with the Logic App webhook, while turning on the Common Alert Schema.

Categories
Uncategorized

Automatically Upgrading Azure Kubernetes Services

Azure provides a means to upgrade Kubernetes clusters using the AKS service. This guide will walk you through using an automation account to upgrade the services on a regular basis, making the process something you don’t need to worry about.

Note that you may want to hold off on doing this for production systems – if for some reason an upgrade were to break currently functionality, there is no means for reverting a cluster back to an original version.

Create a Powershell Core Function App

First, create a function app that runs on PowerShell Core:

After creating the function app, you’ll also want to make sure to increase the standard timeout rate, since this call can take some time to process. Change host.json to have the following:

{
    "functionTimeout": "00:10:00"
}

If you have a large number of clusters you’ll be checking regularly, you should use a standard App Service plan instead, to remove the timeout entirely.

Import Azure CLI into the Function App

Next, you’ll want to import Azure CLI into the Function App, to allow for calling the az command.

First, you’ll need to install Azure CLI on you local machine. You’ll be copying this directory created into the function app created to use, so after installing, locate the Azure CLI files at C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2.

Connect to FTP using the publish profile for the Function App (access this through the portal) and copy the entire CLI2 folder into the /home directory. Make sure that all of the files are copied successfully.

To verify everything is working, run the following command:

D:/home/CLI2/wbin/az.cmd

If you get a successful call back, you’ve imported Azure CLI correctly and can now write the code to programmatically upgrade the AKS clusters.

Create Service Principal

Next, you’ll create a service principal that has access to the clusters in mind, so you have an account that can log in and perform the upgrade duties. Run the following command locally when logged in to the subscription desired:

az ad sp create-for-rbac -n "aks-autoupgrade-sp" --role contributor --scopes /subscriptions/{SubID}

After this is done, you should receive output showing the name, password, and tenant. Add the three of these as configuration values for the Function App as:

  • AZ_USER – appId
  • AZ_PASS – password
  • AZ_TENANT – tenant

Create Timer Function

Next, create a timer function that runs every day, let’s say at noon:

0 0 12 * * *

Use the following codebase.

param($Timer)

Write-Output "Logging in as Service Principal $env:AZ_USER"
D:/home/CLI2/wbin/az.cmd login --service-principal -u $env:AZ_USER -p $env:AZ_PASS --tenant $env:AZ_TENANT | Out-Null

if (!($?)) {
    Write-Error "Login failed, exiting script."
    exit 1;
}

Write-Output "Getting list of AKS clusters...";
$aksClusters = D:/home/CLI2/wbin/az.cmd aks list | ConvertFrom-Json;

if ($aksClusters.length -le 0) {
  Write-Output "No AKS clusters found, exiting...";
  exit 1;
}

$aksClusters | ForEach-Object -Process {
  $clusterName = $_.name
  Write-Output "$clusterName : checking for upgrades..."
  $upgrades = D:/home/CLI2/wbin/az.cmd aks get-upgrades `
    -g $_.resourceGroup -n $_.name | ConvertFrom-Json
  $agentPoolProfiles = $upgrades.agentPoolProfiles

  if ($agentPoolProfiles.upgrades -eq $null) {
    Write-Output "No upgrades available."
    return;
  }

  $latestUpgrade = $agentPoolProfiles.upgrades[-1];

  $currentKubernetesVersion = $upgrades.agentPoolProfiles.kubernetesVersion;
  $newKubernetesVersion = $latestUpgrade.kubernetesVersion

  Write-Output `
    "Upgrade available: $currentKubernetesVersion => $newKubernetesVersion"

  Write-Output "Upgrading to $newKubernetesversion automatically."
  D:/home/CLI2/wbin/az.cmd aks upgrade -k $newKubernetesVersion -g $_.resourceGroup -n $_.name --no-wait --yes
}

Write-Output "Complete."
exit 0;

You can run the function to make sure it is running as intended, while commenting out the az aks upgrade line to ensure no upgrades occur.

Setting up Failure Alerts

The final (optional) step is setting up a means to alert in case of failure. When creating the Function App, an Application Insights resource should have been created as well. Go the ‘Alerts’ section in the App Insight resource, and create an alert:

Add your email as an action group to notify if there is an issue with failures.

Reference

https://stackoverflow.com/questions/56544059/azure-cli-commands-not-working-inside-azure-function-apps-portal

Categories
Uncategorized

Setting up SFTP File Migration to Azure Storage Account

This guide assumes you are using a Linux SFTP server for receiving files, and then uploading those files into an Azure storage account.

Initial Setup

Install Azure CLI on the SFTP server with the following command:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Then login using an account with access to the storage account:

az login

Perform File Migration

The next step is actually sending the files over to a storage account. For this example, we’ll send all of the files in a particular folder to the storage account.

az storage blob upload-batch -d YOUR_CONTAINER --account-name STORAGE_ACCOUNT_NAME -s DIRECTORY

Once those files are moved over, move them into a processed/ directory:

mkdir /data/ftpuser/upload/processed

sudo mv * /processed/

Automate Migration Process

The final step is making sure to automate the process to allow for migrating files on a regular interval.

First, create a file named /bin/migrate.sh:

#!/bin/bash

echo "Starting file -> AZ storage upload"
sudo mkdir -p /data/ftpuser/queued/
sudo mv /data/ftpuser/upload/* /data/ftpuser/queued/
az storage blob upload-batch -d CONTAINER_NAME --account-name STORAGE_ACCOUNT_NAME -s DIRECTORY
if [ $? -ne 0 ]; then
    echo FAIL
    exit 1;
fi
sudo mkdir -p /data/ftpuser/processed
sudo mv /data/ftpuser/queued/* /data/ftpuser/processed/

Allow the script to be executable:

chmod u+x /bin/migrate.sh

Then edit cronjobs to run every 15 minutes:

sudo crontab -e

*/15 * * * * ./bin/migrate.sh >> /var/log/migrate_output.log 2>&1

You can check the logs of the process here:

/var/log/migrate_output.log

Reference

https://kvz.io/schedule-tasks-on-linux-using-crontab.html

Categories
Uncategorized

Manually Renew A Let’s Encrypt Certificate from Kubernetes

I work with a series of Kubernetes clusters that are restricted to public access, but still are encrypted using Let’s Encrypt certificates. Normally, when renewal is requied, this process is automatically done for you.

In the case above, I have to kick off the renewal process directly. This can be done easily with deleting the TLS secret generated for the certificate. This can be done through the UI:

Or via CLI:

kubectl delete secret tls-secret

Once this is done, the cert should automatically attempt a renewal. You can check the status of this with:

kubectl describe certificate tls-secret --namespace ingress-nginx

You should see the following:

Categories
Uncategorized

Setting up Slack Notifications with Jenkins

Setting up Slack notifications for build results with Jenkins is pretty simple. You’ll need the following:

  • A Jenkins server that you have admin rights to.
  • Slack Notification Plugin
  • A Slack workspace, ideally with a channel dedicated to Jenkins alerts.

Setup

To set up Slack, go to <your-workspace-name>.slack.com/apps and search for Jenkins CI:

Add the configuration and set the desired channel where alerts will appear.

After this, you’ll see some directions on setting up Slack on the Jenkins side.

When you’ve finished the setup, make sure to test the connection – you should see a message in Slack like the following:

Adding Jenkins Calls to Pipeline

Once the setup is complete, you can use the following examples to call Slack during the pipeline jobs:

stages {
  stage('Build') {
    steps {
      echo 'build'
    }
  }
  stage('Deploy') {
    steps {
      echo 'Deploy'
      slackSend color: 'good', message: "deployment successful => https://yoursite.com"
    }
  }
}
...
post {
  failure {
    slackSend color: 'danger', message: "deployment failed (<${env.BUILD_URL}|Open>)"
  }
  always {
    cleanWs()
  }
}

This example will call a successful message when all stages complete, and sends a failure message if the build as a whole fails.

Categories
Uncategorized

Setting up Jenkins to Auto-Update

When setting up Jenkins, you’ll have to perform upgrades on the system yourself. You can do this pretty easily, but let’s automate the process so that each night, Jenkins will download and install the latest version.

To do this you’ll need to have SSH access to the server.

Creating Upgrade Script

First, SSH into the server.

Create a bin/ folder in the user’s directory:

cd ~
mkdir bin

If it doesn’t already exist, create a .bash_profile file and add the following:

 export PATH=$PATH:/home/<YOUR_USER>/bin

Now create the script file for use:

cd bin
touch upgrade-jenkins

Create a script with the following contents:

#!/bin/bash

sudo wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war
sudo mv jenkins.war /usr/share/jenkins/jenkins.war
sudo systemctl restart jenkins

And make the script executable:

chmod u+x upgrade-jenkins

You can finally test the script by calling upgrade-jenkins. When it runs, Jenkins should restart with the latest version.

Setting up for Automatic Updates

The next step is setting up the script to run each night at some point to promote automatic upgrades, so you don’t have to worry about updating the system. To do this, we’ll create a cron job that runs the above script once a day.

Set up a crontab file using crontab -e, and add the following to the file to run everyday at 3am. (make sure to check date to see if the system is using UTC):

0 3 * * * upgrade-jenkins

Once this is in place, you’ll be running the script above once a day, which will download and run the latest version of Jenkins on a regular basis.


Reference:

https://www.taniarascia.com/how-to-create-and-use-bash-scripts/

Categories
Uncategorized

Setting up a Linux SFTP Server

To get started, you’ll need to use the following:

  • A virtual machine with a public IP address.
  • SSH address to the above virtual machine

Configuring the User

First, SSH into the server and install OpenSSH:

sudo apt install -y ssh openssh-server

Now create the group to have access, alongside a user to be able to access:

sudo groupadd sftpg
sudo useradd -g sftpg ftpuser
sudo passwd ftpuser

Now set up the directories to allow for the following:

  • The user can view the contents of their own folder.
  • The user can upload files to the uploads/ folder.
sudo mkdir -p /data/ftpuser/upload
sudo chown -R root.sftpg /data/ftpuser
sudo chown -R ftpuser.sftpg /data/ftpuser/upload

Setting up SFTP

To set up SFTP, append the following to the end of /etc/ssh/sshd_config:

Match Group sftpg
     ChrootDirectory /data/%u
     ForceCommand internal-sftp

As you’re using a password to authenticate the above user, make sure the following is in the file as well:

PasswordAuthentication yes

Restart the service:

sudo systemctl restart sshd

Verification

To verify everything is in place, try logging into the server as the new user:

sftp ftpuser@YOUR_DOMAIN

Once connected in, try uploading a file:

PUT D:/local-file.png file.png

And then check that the file is there:

GET file.png D:/downloaded-file.png

Categories
Uncategorized

Setting up Artifactory to use a root domain with a Reverse Proxy

When installing Artifactory, you’ll be able to start using it as `http://YOUR_DOMAIN:8081/artifactory. To change this to a simpler domain, you can read the following guide. You’ll need to have the following:

  • Artifactory administrative access
  • SSH access to the server.

Setting up Reverse Proxy to Simplify Domain

First, SSH into the server and install Apache:

sudo apt install apache2

This should create a default page you can access by hitting the root URL:

Now configure the appropriate modules:

sudo a2enmod proxy_http proxy_ajp rewrite deflate headers proxy_balancer proxy_connect proxy_html ssl lbmethod_byrequests slotmem_shm proxy

While logged in as an admin in Artifactory, access Admin -> HTTP Settings and make the following changes:

After saving, you’ll see the ability to View/Download the configuration on the right side of the page. Download the contents and replace the contents of /etc/apache2/sites-enabled/000-default.conf on the Artifactory server with them.

Restart Apache with sudo systemctl restart apache2.

Now confirm that you can request Artifactory using http://YOUR_DOMAIN.

Setting up SSL using Let’s Encrypt

Now that you can access the domain, let’s secure the domain using Let’s Encrypt.

Use the following to check which version of Linux you’re running: lsb_release -a

Most likely, you’ll be using Ubuntu 18.04, if so, you can use these steps. If not, check out the CertBot page for appropriate directions.

You should be able to generate and configure the cert automatically running the directions above. Assuming this is successful, select to redirect HTTP traffic to HTTPS, removing HTTP access.

Now confirm that you can both:

  1. Access the HTTPS version of your domain above.
  2. When trying to access HTTP, it automatically redirects to HTTPS.

Reference: https://www.jfrog.com/confluence/display/RTF/Configuring+Apache

Categories
Uncategorized

Resetting Artifactory Admin Password

If you’ve lost access to the administrator account in your on-premised Artifactory, this guide will help walk you through setting it back up. You’ll need to have SSH access to the server in order to complete this guide.

To get started, SSH into the server and open$ARTIFACTORY_HOME/etc/security/access/bootstrap.creds. Enter the following information:

access-admin@127.0.0.1=PASSWORD

Now assign read/write permissions:

chmod 600 bootstrap.creds

And finally, restart the server.

Once that’s done, make the following POST call while still SSHed into the server:

curl --request PATCH \
  --url http://localhost:8081/artifactory/api/access/api/v1/users/admin \
  --header 'authorization: Basic <ENCODE access-admin:PASSWORD>' \
  --header 'content-type: application/json' \
  --data '{"password":"NEW_PASSWORD"}'

If you receive a response like the following, you’re all set:

{
  "username" : "admin",
  "realm" : "internal",
  "status" : "enabled",
  "allowed_ips" : [ "*" ],
  "created" : "2019-08-01T13:47:16.959Z",
  "modified" : "2019-09-04T15:25:02.398Z",
  "last_login_time" : "2019-08-21T13:46:55.061Z",
  "last_login_ip" : "204.194.141.30",
  "custom_data" : {
    "updatable_profile" : "true",
    "artifactory_admin" : "true"
  },
  "password_expired" : false,
  "password_last_modified" : 1567610702390,
  "groups" : [ ]
}