The Practical Administrator - Lessons of a Datacenter Administrator

Backups using Borg

2020-03-24 by Andrew Sullivan

I’m not going to lecture anyone about backups. I’m going to trust you already know you need backups and that you’re curious about how to set them up or you want to compare your setup to mine. Either way, you’re here now.

My setup is pretty straightforward. I repurposed an old NUC with an external SATA enclosure to act as the borg server. The external enclosure has four drives in a RAID-5, providing enough capacity for me to backup for a quite a while. In addition, I have the most important data replicated to object storage using rclone. This gives me multiple copies of data with multiple layers of protection, covering the 3-2-1 strategy for my most important data, and for the rest I am ok with the risk of loss.

Borg works off the principle of client-server, so the clients backup data to the “remote” borg server (it’s not actually remote, just at the other end of the house). If you aren’t familiar with borg, I highly encourage you to read the docs and have at least basic familiarity with it’s principle of operation.

My setup is loosely based on the configuration described here, so you may want to give that a read too if you like.

What and when to backup

I have a lot of things self-hosted in my home and on virtual machines in a couple of different services. I’m simply going to list a few of the things I backup, and leave you to evaluate what data you want backed up and where it’s at.

Private chat using Prosody

2020-03-222020-03-22 by Andrew Sullivan

There are a seemingly infinite number of chat protocols, services, and various other ways to interact with other people. Wanting to try something new and experiment a bit, I decided to deploy a Prosody server to a DigitalOcean virtual machine. This turned out to be more time consuming than I expected, despite everyone extolling how “quick, simple, and easy” it is, largely because the docs are lackluster and examples are hard to come by.

I currently use this Prosody instance to receive alerts and messages from the various things running in my lab (e.g. alerts about power outages sent using ntfy), as well as chatting with my wife. Yes, I could use WhatsApp, Telegram, Slack, IRC, Twitter, Signal, SMS, Rocket.Chat, Matrix, RCS, or about a thousand other things, but what fun would that be?

WireGuard VPN

2020-03-222020-03-22 by Andrew Sullivan

I travel somewhat frequently for work and end up using untrusted WiFi as a result. Being privacy conscious, and the primary IT helpdesk for my family, means that I want/need secure access to my resources. The homelab is super useful for demonstrating functionality and concepts, so I find that I use it fairly often when doing presentations. All of this is leading up to a theoretically simple solution: a VPN.

For years I used OpenVPN, which is generally considered a very good solution for most situations, even being used by some enterprises. However, over time I found the performance to be lackluster at best and the amount of time to establish connections was irritating.

A few months ago I had some spare time while on a work trip, so I made the switch to WireGuard. Yes, there’s lots of paranoia about it being validated, etc., etc. but it’s good enough for me. The switch has been surprisingly easy, even allowing me to use my Pi-Hole VM as both the DNS/DHCP and VPN host while providing excellent performance with fewer resources. There are countless helper scripts and other self-hosted GUIs for WireGuard, but honestly with only a few clients I haven’t found the need to use one…adding a client takes about 60 seconds manually.

My Pi-Hole configuration

2020-03-212020-03-21 by Andrew Sullivan

Pi-Hole has been a staple of my homelab for several years now. Even if you don’t want to use the ad-blocking feature, the reporting and logging I find to be very helpful. Anyway, it’s one of my favorite projects and I highly encourage anyone and everyone to check it out!

The default configuration is very good, particularly if you want to simply block the majority of ads. I’m a bit over zealous, so I like to block ads, trackers, malware, and many other things. Additionally, I use Pi-Hole for DHCP on my network, having made the change when I moved from a pfSense router to a USG.

I’m going to assume you are able to follow the Pi-Hole deployment instructions and get everything up and running in the default configuration. I use a virtual machine (running CentOS 7.7) to host my instance, but you can use a RaspberryPi, almost any old hardware, even a container on an existing Linux machine if you’d like. I’ll also assume that you’re capable of updating your network and/or clients to use the Pi-Hole.

Red Hat Virtualization All-in-One

2020-03-212020-03-21 by Andrew Sullivan

This was originally published as a Gist to help some peers, however since those are hard to discover I am reposting here.

The goal is to install Red Hat Virtualization onto a single host, for example a home lab. I’m doing this with Red Hat Enterprise Linux (7.7) and RHV, but will work just as well with CentOS and oVirt. If you’re using RHV, note that this is not supported in any way!

The Server

My home lab consists of a single server with adequate CPU and RAM to host a few things. I’m using an old, small HDD for the OS and a larger NVMe drive will be used for VMs. My host has only a single 1GbE link, which isn’t an issue since I’m not expecting to use remote storage and, being a lab, it’s ok if it doesn’t have network access due to a failure…it’s a lab!

You can use pretty much whatever hardware with whatever config you like for hosting your virtual machines. The only real requirement is about 6GB of RAM for the OS + hosted engine, but you’ll want more than that to host other VMs.

Install and configure

Before beginning, make sure you have forward and reverse DNS working for the hostnames and IPs you’re using for both the hypervisor host and RHV Manager. For example:

10.0.101.20 = rhvm.lab.com
10.0.101.21 = rhv01.lab.com

With that out of the way, let’s deploy and configure RHV!

NetApp PowerShell Toolkit 101: Volume Snapshots

2018-07-202016-07-20 by Andrew Sullivan

Snapshots are one of the core features of ONTAP, and something that many, many people rely on every day to protect their data from accidental (or malicious…) deletion and corruption. The NetApp PowerShell Toolkit can help us to manage the configuration of snapshot policies, the application of those policies to volumes, and creating/deleting/reverting snapshots too.

This post will cover:

Snapshots
- Management
- Reporting
- Snap Reserve
Snapshot Policies
Snapshot Autodelete
Recovering Data

NetApp PowerShell Toolkit: Authentication

2018-07-202016-07-18 by Andrew Sullivan

There are multiple ways to do authentication to NetApp systems when using the PowerShell Toolkit. This ranges from the simple and obvious one-time connection, to securely storing credentials for future use. Saving credentials can be useful when executing scripts from a host non-interactively, such as with scheduled tasks or triggered through another script.

Connecting to a Single Controller

The Connect-NcController is the standard method of connecting to a clustered Data ONTAP controller. Connect-NaController is the 7-mode equivalent and works identically. Additionally, the same credential rules apply for the Invoke-NcSsh and Invoke-NaSsh cmdlets as well.

Arguably the most common method of connecting to a controller is by simply providing the hostname:

# this will attempt to connect to the specified controller using stored credentials, or if none
# are found, will prompt for credentials.  it will also default to HTTPS, with a fallback to HTTP
Connect-NcController $myController

# this will attempt to connect to the specified controller using stored credentials, or if none

# are found, will prompt for credentials. it will also default to HTTPS, with a fallback to HTTP

Connect-NcController $myController

If you are connecting to an SVM’s management interface this will work as expected, though some cmdlets won’t work because of the limited scope. If you want to connect to an SVM by tunneling through the cluster management interface, use the -Vserver parameter.

Connect-NcController $clusterMgmtLif -Vserver $SvmName

1	Connect-NcController $clusterMgmtLif -Vserver $SvmName

However, there are a number of parameters which change the default behavior.

NetApp PowerShell Toolkit – Templates

2018-07-202016-02-11 by Andrew Sullivan

There’s one particular part of the NetApp PowerShell Toolkit which is not frequently used, but is extremely powerful. Templates can be created for many of the object types which are used to create a query for specific objects, or for limiting the amount of information returned from a particular cmdlet invocation.

To get started, we first need to initialize the object for our query or attribute limiting template. To do this we use the -Template parameter to our cmdlet.

# create an empty Aggregate object template
$aggrTemplate = Get-NcAggr -Template

1 2	# create an empty Aggregate object template $aggrTemplate = Get-NcAggr -Template

If we were to look at this object it is empty:

Many of the properties associated with an object, such as an aggregate, volume, or LUN, are objects themselves. If we want to use a property of a child object as the query filter then we need to initialize that property in the template object.

# create an empty Aggregate object template
$aggrTemplate = Get-NcAggr -Template

# initialize a property of the template
Initialize-NcObjectProperty -Object $aggrTemplate -Name AggrRaidAttributes

# alternatively, initialize all properties during template creation
$aggrTemplate = Get-NcAggr -Template -Full

# create an empty Aggregate object template

$aggrTemplate = Get-NcAggr -Template

# initialize a property of the template

Initialize-NcObjectProperty -Object $aggrTemplate -Name AggrRaidAttributes

# alternatively, initialize all properties during template creation

$aggrTemplate = Get-NcAggr -Template -Full

Our template object now has a property object which has been populated:

At this point we’re ready to use the template, let’s look at how to use it as a query or to limit the attributes returned.

NetApp PowerShell Toolkit: Aggregate Overcommitment Report

2018-07-202016-01-20 by Andrew Sullivan

I recently encountered a posting on the NetApp Community asking about, among other things, allocated capacity for an aggregate. As you can see in my response, I created a quick scriptlet which displays the information regarding total volume capacity allocated, but this is only part of the potentially thin provisioned capacity. LUNs can also be thin provisioned inside the volume. Additionally, some may find how much overcommitment exists with no storage efficiency applied as well (this can help with IOPS density calculations, for example).

To address this, I created a function which will display the total, used, saved, and committed capacity for an aggregate…

Show Overcommitment for Non-Root Aggregates

Get-NcAggr | ?{ $_.Name -notlike "*root" } | Get-NcAggrOvercommitReport | Format-Table -AutoSize

1	Get-NcAggr \| ?{ $_.Name -notlike "*root" } \| Get-NcAggrOvercommitReport \| Format-Table -AutoSize

Show Aggregates Which Have Overcommitment > 200%

Get-NcAggr | ?{ $_.Name -notlike "*root" } | Get-NcAggrOvercommitReport | ?{ $_.CommittedPercent -gt 200 }

1	Get-NcAggr \| ?{ $_.Name -notlike "*root" } \| Get-NcAggrOvercommitReport \| ?{ $_.CommittedPercent -gt 200 }

PowerShell: Convert String To Array

2018-07-202016-01-18 by Andrew Sullivan

In December I had fun solving the “riddles” at Advent Of Code and a large number of them (starting with number 1) had strings for input that needed to be parsed. So, posting this for myself, cause I can never remember it….

Convert a string to an array of single characters

# the input string
$string = "This is a string"

# convert it to a character array
$array = $string.ToCharArray()

# now we can iterate like any array
foreach ($char in $array) {
    $char
}

# the input string

$string = "This is a string"

# convert it to a character array

$array = $string.ToCharArray()

# now we can iterate like any array

foreach ($char in $array) {

$char

}

The output:

Convert a string into an array of words

# the input string
$string = "This is a string"

# create an array from an arbitrary separator, in this case a space
$array = $string.split(" ")

# this will also work
$array = $string -split " "

# the input string

$string = "This is a string"

# create an array from an arbitrary separator, in this case a space

$array = $string.split(" ")

# this will also work

$array = $string -split " "

Convert a string based on arbitrary characters

This also works with regular expressions…

$string = "a1b2c3d4e5f6"

# split on numbers - remember, this will remove the numbers from the output
$string -split "\d"

# split on letters - this will remove the letters from the output
$string -split "\D"

$string = "a1b2c3d4e5f6"

# split on numbers - remember, this will remove the numbers from the output

$string -split "\d"

# split on letters - this will remove the letters from the output

$string -split "\D"