Digestible DevOps

Kubernetes Lab Setup on Linux

Sat, 01 Feb 2025 06:11:46 +0000

This is a simple tutorial for setting up a 3 node Kubernetes lab on a Linux machine. This setup is intended for educational purposes and may not be suitable for production environments. This tutorial will specifically use Mutipass to create VMs on a Linux host. The host used for this tutorial is Ubuntu 24.04.

Conventions

k8s-cp: Control Plane node
k8s-worker1: Worker node 1
k8s-worker2: Worker node 2

Prerequisites

A Linux machine (Ubuntu 20.04 or later is recommended)
Root or sudo access

Step 1: Install Multipass

To install Multipass and then create the VMs, run the following commands in your terminal:

sudo snap install multipass --classic
multipass version
multipass launch --name k8s-cp --cpus 2 --memory 2G --disk 20G
multipass launch --name k8s-worker1 --cpus 2 --memory 2G --disk 20G
multipass launch --name k8s-worker2 --cpus 2 --memory 2G --disk 20G
multipass list

Since these are all using the same default Ubuntu image, you should expect that the first launch will take a while, as it downloads the full OS, but the subsequent launches will be much faster.

Step 2: Base VM Setup

Create a script file on the host named setup-vm.sh using this command:

cat <<EOF > setup-vm.sh
#!/usr/bin/env bash

# Update the apt package index and install packages needed to configure the k8s apt repo
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# Configure the apt repo for Kubernetes
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install kubelet, kubeadm, kubectl, and containerd
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl containerd
sudo apt-mark hold kubeadm kubelet kubectl

# Configure containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

# Disable swap (kubeadm init will fail if swap is enabled)
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Load necessary kernel modules for Kubernetes networking
sudo modprobe overlay
sudo modprobe br_netfilter
echo -e "overlay\nbr_netfilter" | sudo tee /etc/modules-load.d/k8s.conf
echo "br_netfilter" | sudo tee /etc/modules-load.d/k8s.conf

# Configure sysctl settings for Kubernetes networking
cat <<EOF2 | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF2
sudo sysctl --system

# Reboot the VM
if [ -f /var/run/reboot-required ]; then
    cat /var/run/reboot-required
    sudo reboot
fi
EOF

Then transfer and run the script on each VM:

for vm in k8s-cp k8s-worker1 k8s-worker2; do
    multipass transfer setup-vm.sh $vm:/home/ubuntu/setup-vm.sh
    multipass exec $vm -- bash -c "chmod +x /home/ubuntu/setup-vm.sh && /home/ubuntu/setup-vm.sh"
done

Step 3: Install Kubernetes Control Plane

Create a script file on the host named setup-k8s-cp.sh using this command:

cat <<EOF > setup-k8s-cp.sh
#!/usr/bin/env bash

# Start kubelet and run kubeadm init
sudo systemctl enable --now kubelet
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans $(hostname -I | awk '{print $1}')

# Configure kubectl for the current user
mkdir -p \$HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf \$HOME/.kube/config
sudo chown \$(id -u):\$(id -g) \$HOME/.kube/config

# Install Flannel CNI
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
EOF

Then transfer and run the script on the control plane node:

multipass transfer setup-k8s-cp.sh k8s-cp:/home/ubuntu/setup-k8s-cp.sh
multipass exec k8s-cp -- bash -c "chmod +x /home/ubuntu/setup-k8s-cp.sh && /home/ubuntu/setup-k8s-cp.sh"

Step 4: Join Kubernetes Workers to the Cluster

JOIN_COMMAND="$(multipass exec k8s-cp -- kubeadm token create --print-join-command)"
for vm in k8s-worker1 k8s-worker2; do
    multipass exec $vm -- bash -c "sudo $JOIN_COMMAND"
done

Step 5: Copy Kubeconfig to Host

First off, if you don’t have kubectl installed on your host, you can install it with:

sudo snap install kubectl --classic

You should also configure the Bash completion for kubectl:

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl
source /etc/bash_completion.d/kubectl

Then, copy the kubeconfig file from the control plane node to your host machine:

multipass exec k8s-cp -- bash -c 'sudo cp /etc/kubernetes/admin.conf /home/ubuntu/lab.conf; sudo chown $(id -u):$(id -g) /home/ubuntu/lab.conf'
mkdir -p ~/.kube
multipass transfer k8s-cp:/home/ubuntu/lab.conf ./lab.conf
mv ./lab.conf ~/.kube/

Step 6: Configure Host Proxy to Access the Cluster

If you want to access the cluster from a different machine, one option is that you can set up socat on your host machine to proxy the calls to the control plane VM:

K8S_CP_IP="$(multipass info k8s-cp | grep IPv4 | awk '{print $2}')"
sudo ufw allow 6443/tcp
sudo apt-get update && sudo apt-get install socat -y
multipass exec k8s-cp -- bash -c 'sudo apt-get update && sudo apt-get install socat'

mkdir -p ~/bin
cat <<EOF > ~/bin/forward-k8s-cp.sh
#!/usr/bin/env bash
multipass exec k8s-cp -- socat STDIO TCP4:127.0.0.1:6443
EOF

chmod +x ~/bin/forward-k8s-cp.sh

mkdir -p ~/.config/systemd/user
cat <<EOF > ~/.config/systemd/user/forward-k8s-cp.service
[Unit]
Description=Socat forward from host 6443 to k8s-cp VM
After=default.target

[Service]
# We run socat in user mode on port 6443
# No sudo needed, since 6443 > 1024
ExecStart=/usr/bin/socat TCP-LISTEN:6443,fork,reuseaddr SYSTEM:"${HOME}/bin/forward-k8s-cp.sh"

# Keep restarting on failure
Restart=always
RestartSec=5

[Install]
WantedBy=default.target
EOF

sudo loginctl enable-linger $(whoami)
systemctl --user daemon-reload
systemctl --user enable forward-k8s-cp
systemctl --user start forward-k8s-cp

echo "##############################################"
echo "# Use the config below to access the cluster #"
echo "##############################################"
sed "s/${K8S_CP_IP}/$(hostname -I | awk '{print $1}')/g" ~/.kube/lab.conf

Assuming that you copied the config shown into ~/.kube/lab.conf, now you can use this config to access the cluster from your host machine. To set your KUBECONFIG environment variable to use this config, run:

export KUBECONFIG=~/.kube/lab.conf

If you want to rename the context to something shorter, you can run:

kubectl config rename-context kubernetes-admin@kubernetes lab

And lastly, if you want to be able to use kubectl without specifying the KUBECONFIG environment variable, you can merge the config into your existing ~/.kube/config:

cp ~/.kube/config ~/.kube/config.bak
KUBECONFIG=~/.kube/config:~/.kube/lab.conf kubectl config view --flatten > ~/.kube/merged.conf
sleep 1 # I've seen some stuff...
mv ~/.kube/merged.conf ~/.kube/config
rm ~/.kube/lab.conf ~/.kube/merged.conf

Step 7: Verify the Setup

To verify that your Kubernetes cluster is up and running, you can run a few commands:

kubectl get nodes
kubectl get pods --all-namespaces

Step 8: Clean Up

When you’re done with the lab, you can delete the VMs using:

multipass delete k8s-cp k8s-worker1 k8s-worker2
multipass purge

Conclusion

This tutorial provided a step-by-step guide to setting up a 3 node Kubernetes lab on a Linux machine using Multipass. You can now experiment with Kubernetes and learn more about container orchestration.

Better PowerShell with PSReadLine

Sun, 15 Jan 2023 02:06:55 +0000

If you want an easy way to navigate your command history as you type in PowerShell, you may want to consider PSReadLine

To update to the latest version of PSReadLine, close all instances of PowerShell, including in VS Code, etc., then open an Administrator instance of Command Prompt (not PowerShell) and run:

pwsh -noprofile -command "Install-Module PSReadLine -Force -SkipPublisherCheck -AllowPrerelease"

Then back in a PowerShell instance, you can configure PSReadLine by running:

if ((Get-Content $PROFILE | Select-String "Import-Module PSReadLine").Length -eq 0) {
    Write-Output "Import-Module PSReadLine" | Add-Content "$PROFILE"
}
# Set-PSReadLineOption -PredictionSource HistoryAndPlugin
if ((Get-Content $PROFILE | Select-String "Set-PSReadLineOption -PredictionSource History").Length -eq 0) {
    Write-Output "Set-PSReadLineOption -PredictionSource History" | Add-Content "$PROFILE"
}
# Set-PSReadLineOption -PredictionViewStyle InlineView
if ((Get-Content $PROFILE | Select-String "Set-PSReadLineOption -PredictionViewStyle ListView").Length -eq 0) {
    Write-Output "Set-PSReadLineOption -PredictionViewStyle ListView" | Add-Content "$PROFILE"
}
if ((Get-Content $PROFILE | Select-String "Set-PSReadlineKeyHandler -Key Tab -Function MenuComplete").Length -eq 0) {
    Write-Output "Set-PSReadlineKeyHandler -Key Tab -Function MenuComplete" | Add-Content "$PROFILE"
}
if ((Get-Content $PROFILE | Select-String "Set-PSReadLineKeyHandler -Key UpArrow -Function HistorySearchBackward").Length -eq 0) {
    Write-Output "Set-PSReadLineKeyHandler -Key UpArrow -Function HistorySearchBackward" | Add-Content "$PROFILE"
}
if ((Get-Content $PROFILE | Select-String "Set-PSReadLineKeyHandler -Key DownArrow -Function HistorySearchForward").Length -eq 0) {
    Write-Output "Set-PSReadLineKeyHandler -Key DownArrow -Function HistorySearchForward" | Add-Content "$PROFILE"
}

Installing k3d on Windows

Wed, 24 Mar 2021 04:04:32 +0000

To install k3d, you will need Docker. If you don’t already have Docker installed, you can follow the instructions in the next section to get Docker Desktop (check the prerequisites here).

Installing Docker Desktop

Prerequisite: WSL2

Start a PowerShell instance as Administrator and run the following to install WSL2:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Reboot at this time, then come back and download and install WSL2 Linux kernel update package for x64 machines.

Now back in an administrative PowerShell instance:

# Set WSL2 as the default
wsl --set-default-version 2

Lastly, grab a Linux distro from the Microsoft Store. If you don’t know which to choose, just go with Ubuntu 20.04. You can always change it later. If you don’t have access to the Microsoft Store (e.g. it’s blocked on a work system), you can find manual install instructions here.

At this point you may as well launch and set up your distro, which will likely be nothing more than creating an account name and password. If you have a common username on other systems that you will likely be connecting to, it may be desirable to use that as your username in your Linux distro, but ultimately, you can use whatever username you want.

Docker Desktop

You can either go to the main product page to grab the latest installer (I would recommend this approach), or you can look for a specific version here. Whichever way you install it, make sure you select the option to install the WSL2 components (should be selected by default).

And now one more reboot and you should be good to go.

Tip: if you don’t plan on using Docker all the time, you may wish to adjust the settings to not have it start when you log in.

Installing k3d

The easiest way to get k3d running on Windows is with Chocolatey. To install Chocolatey you can run the following from an administrative PowerShell instance:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Now close PowerShell and open a new administrative instance and run the following to install k3d and a couple other useful tools:

choco install k3d -y
choco install jq -y
choco install yq -y
choco install kubernetes-helm -y

And now let’s configure tab completion for k3d:

# Create user profile file if it doesn't exist
if ( -not ( Test-Path $Profile ) ) { New-Item -Path $Profile -Type File -Force }
# Append the k3d completion to the end of the user profile
k3d completion powershell | Out-File -Append $Profile

Using k3d

Start a new PowerShell instance (doesn’t need to be administrative this time around). Now that you have the k3d binary on your path, you can create a cluster by running:

k3d cluster create localk8s

You can name the cluster whatever you want by replacing localk8s with whatever you would like.

Windows will ask you to approve a change to the firewall configuration. Definitely do not allow access to public networks.

With my current version of k3d (v4.3.0) I get an error about .kube/config failing to be overwritten, though it appears to be a pathing issue. To fix it, just move the file it created in .kube to be .kube/config:

Move-Item ~\.kube\config.k3d* ~\.kube\config -Force

You can peek at your running cluster by running some commands like the following:

# List the Kubernetes nodes
kubectl get nodes --output wide
# List several Kubernetes objects in the cluster
kubectl get all --all-namespaces

And since this is all running in Docker, you can look at the “real” containers there:

docker container ls

You should see two containers by default: a k3s instance and the k3d proxy. The k3d proxy is used to route traffic in to the API server, which you can see configured by looking at ~/.kube/config, where you might see something like server: https://0.0.0.0:52038, which would be the same port that Docker shows as routing to port 6443 of the k3d proxy container.

However, the main reason I want to use k3d instead of minikube is that I want to have multiple nodes so that I can realistically test out taints, tolerations, affinity rules, etc. To create a multi-node system, you can delete the previous cluster and recreate it, this time passing values for --servers and --agents. While we’re at it, we should also set up some port forwarding to be able to interact with we workloads:

# Delete the existing cluster
k3d cluster delete localk8s
# Create a new multi-node cluster
k3d cluster create localk8s --servers 3 --agents 3 --port "8888:80@loadbalancer" --port "8889:443@loadbalancer"
# Rename the config if needed
if ( Test-Path ~\.kube\config.k3d* ) { Move-Item ~\.kube\config.k3d* ~\.kube\config -Force }
# List the Kubernetes nodes
kubectl get nodes --output wide
# Look at the "real" containers again (you can see or new port forwarding)
docker container ls

Another you could do with k3d, though I’m not going to get into it, is to mount volumes to any combination of the nodes in your cluster. If this is something that you end up needing, a quick web search should set you on your way. And if your configuration starts to get too complex, you may want to consider using a config options file instead of command line arguments.

With the cluster running and ready to forward traffic, we can create a deployment, service, and ingress (from my example of making a multi-node cluster with Multipass):

echo "
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ui
  name: ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ui
  template:
    metadata:
      labels:
        app: ui
    spec:
      containers:
      - image: thiedebr/crud-ui:latest
        name: crud-ui
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ui
  name: ui
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: ui
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - image: thiedebr/crud-api:latest
        name: crud-api
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: api
  name: api
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8887
  selector:
    app: api
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: localhost
  annotations:
    ingress.kubernetes.io/ssl-redirect: 'false'
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ui
            port:
              number: 80
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 80
" | kubectl apply -f -

Using the echo alias for Write-Output actually allows this command to work the same in PowerShell and Bash. To test the result of what we’ve done, go to http://localhost:8888 in you browser and you should see “Hello Kubernetes” displayed.

When you are done using your cluster, it is probably best to stop it:

k3d cluster stop localk8s

And then to start it back up later:

k3d cluster start localk8s

If you happen to restart your machine without stopping the cluster, it may end up in an odd state. If this happens, you can try to stop then start the cluster. If that doesn’t work you may have to delete the cluster and start again.

Configuring Bash

As much as I actually prefer several of the features and principles in PowerShell over Bash, there are going to be a lot more examples using Bash scripts, and there will also be some apps that only run on Linux, therefore you will need to run them from WSL. Here is my recommendation for setting things up for WSL2 using Ubuntu 20.04. Just fire up an Ubuntu shell and follow along.

Use the Windows Kubernetes tooling

You can use the Windows binaries from WSL2, and you will want to just go ahead and use those instead of installing new Ubuntu variants. This will make sure that things run nicely both from Windows and Ubuntu, and it will significantly reduce your headaches when trying to proxy things through to your web browser. The one little hiccup is that the Bash tab completion is going to be based on an expected Linux command name, for example k3d, not the Windows command name that will end in .exe, .bat, or something else. Using Linux aliases, this isn’t a problem:

# Setup the alias for k3d in bash configuration (loaded every time you launch the a shell)
grep 'alias k3d=k3d.exe' ~/.bashrc || echo -e "# k3d setup\nalias k3d=k3d.exe" >>~/.bashrc
# Have bash completion get loaded for k3d
grep 'source <(k3d completion bash)' ~/.bashrc || echo "source <(k3d completion bash)" >>~/.bashrc
# Reload the configuration for the current shell
source ~/.bashrc

Now you will have an alias of k3d that will run k3d.exe, but will also provide you with the expected shell completion. Some of our commands are already set up as part of the Docker Desktop install, such that we have the “extensionless” name and tab completion. Here are the remaining things I recommend to get everything set up so far to behave as expected:

# Have bash completion get loaded for kubectl
grep 'source <(kubectl completion bash)' ~/.bashrc || echo -e "# kubectl setup\nsource <(kubectl completion bash)" >>~/.bashrc
# Setup the alias for helm
grep 'alias helm=helm.exe' ~/.bashrc || echo -e "# helm setup\nalias helm=helm.exe" >>~/.bashrc
# Have bash completion get loaded for helm
grep 'source <(helm completion bash)' ~/.bashrc || echo "source <(helm completion bash)" >>~/.bashrc
# Reload the configuration for the current shell
source ~/.bashrc

Lastly, since we are sharing the tooling across Windows and Ubuntu, we should share the config as well.

# Setup WINHOME environment variable for convenience
grep 'export WINHOME=' ~/.bashrc || echo -e '# Home directory in Windows\nexport WINHOME=$(cmd.exe /C "echo %USERPROFILE%" 2>/dev/null  | tr -d '\''\r\n'\'' | sed -E '\''s/([A-Z]+):\\\(.*)/\/mnt\/\L\1\/\2/; s/\\\/\//g'\'')' >>~/.bashrc
# Reload the configuration for the current shell
source ~/.bashrc

# Create a symlink so that ~/.kube pulls your Windows user config
rm -rf ~/.kube
ln -s "${WINHOME}"/.kube ~/.kube

And now you can verify everything with a list of what is currently in the cluster:

kubectl get all --all-namespaces

Resources

Docker Write Speeds on Mac

Thu, 18 Mar 2021 17:19:16 +0000

I was noticing some slowness while using bind mounts in Docker on my Mac. Here are some observations that I made.

Direct Bind Mount Interaction

$ time dd if=/dev/zero of=/shared-volume/speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 46.9982 s, 2.2 MB/s

real	0m47.005s
user	0m0.088s
sys	0m3.900s

Writing to Docker File System

$ time dd if=/dev/zero of=/tmp/speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.278643 s, 367 MB/s

real	0m0.281s
user	0m0.020s
sys	0m0.261s

Writing to Docker File System, then mv to Bind Mount

$ time bash -c "dd if=/dev/zero of=/tmp/speedtest bs=1024 count=100000 && mv /tmp/speedtest /shared-volume/speedtest"
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.280981 s, 364 MB/s

real	0m0.819s
user	0m0.014s
sys	0m0.395s

Summary

After these observations I will start doing as much I/O heavy work outside of any bind mounts and then move any files that need to be available outside of the container to the bind mount after the fact.

Configuring Apache as a Reverse Proxy on Mac

Mon, 22 Feb 2021 05:25:33 +0000

Sometimes it may be useful to use your own local certificate for certain testing, or you may want to present multiple sites as own domain to deal with cross origin resource sharing. These are some of the problems that can be solved with a reverse proxy. This explanation takes the perspective of a Mac user running the default Apache web server (instructions for basic setup). There will be three sections: Virtual Hosts, TLS, and Reverse Proxy.

Virtual Hosts

Virtual hosts, or vhosts, need to be enabled within the httpd.conf and then added to httpd-vhosts.conf. You will need to start editing /private/etc/apache2/httpd.conf as root, e.g. sudo vi /private/etc/apache2/httpd.conf, so that you can uncomment (remove the # at the start of the line) or add the following:

LoadModule log_config_module libexec/apache2/mod_log_config.so
LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so
Include /private/etc/apache2/extra/httpd-vhosts.conf

Now that the web server knows about /private/etc/apache2/extra/httpd-vhosts.conf you can add your own virtual host to the bottom of the file (though you probably want to remove the example entries as the DocumentRoot probably doesn’t exist):

<VirtualHost *:80>
    ServerName local.example.com
</VirtualHost>

Here I am leaving everything but the ServerName at the defaults. Go ahead and run sudo apachectl restart to pick up the new vhost and then you test it out by having curl resolve to that host name to your localhost:

curl http://local.example.com --resolve local.example.com:80:127.0.0.1

In order to allow your web browsers to know how to use this domain name, and to avoid having to manually tell curl what to do each time, you can add the following to the /private/etc/hosts file (edit it as root):

127.0.0.1 local.example.com

Now you should be able to go to http://local.example.com in your web browser.

TLS

In order to let your browser feel nice and cozy by having a TLS certificate, the first thing you need to do is create one. So let’s give it a home:

sudo mkdir -p /private/etc/apache2/tls
cd /private/etc/apache2/tls

For generating the actual certs, I used cfssl. To install it, you will need Go, at which point you can run:

go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson

With cfssl installed, now let’s create our root certificate authority:

cat <<EOF | cfssl genkey -initca - | sudo cfssljson -bare ca
{
    "CN": "localhost Root CA",
    "hosts": [],
    "key": { "algo": "rsa", "size": 4096},
    "names": [
        {
            "C": "US",
            "ST": "Michigan",
            "L": "Lansing",
            "O": "Digestible DevOps",
            "OU": "localhost"
        }
    ]
}
EOF

You should now see ca-key.pem and ca.pem in the /private/etc/apache2/tls folder. If you are curious you see the details of the resulting public key with the following:

openssl x509 -in ca.pem -text -noout

You can add trust to this cert to your Keychain Access certificates, or follow whatever method is needed to install the cert for your preferred browser in order to make things smoother when browsing you the domain you set up. For Keychain Access:

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /etc/apache2/tls/ca.pem

Now you need a server cert, signed by the root CA, and valid for 397 days (9528 hours):

cat > /tmp/ca-config.json <<EOF
{
  "signing": {
    "profiles": {
      "server": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "9528h"
      }
    }
  }
}
EOF

# Add any additional domain names here
HOSTNAME_LIST='"localhost", "*.example.com"'

SERVER_CSR="$(cat <<EOF | sudo cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=/tmp/ca-config.json -profile=server -
{
    "CN": "localhost",
    "hosts": [ ${HOSTNAME_LIST} ],
    "key": { "algo": "rsa", "size": 2048 },
    "names": [
        {
            "C": "US",
            "ST": "Michigan",
            "L": "Lansing",
            "O": "Digestible DevOps",
            "OU": "localhost"
        }
    ]
}
EOF
)"

echo "${SERVER_CSR}" | sudo cfssljson -bare server

Now you have server-key.pem and server.pem, which will be used directly by Apache, once we configure Apache to do so. Now the file /private/etc/apache2/httpd.conf needs to be modified to enable TLS. Once again you need to edit the file as root, making sure that the following line are present and uncommented (no # at the start of the line):

LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so
LoadModule ssl_module libexec/apache2/mod_ssl.so
Include /private/etc/apache2/extra/httpd-ssl.conf

Now we need to modify the file /private/etc/apache2/extra/httpd-ssl.conf, also as root, to set up the paths to the cert files so that you end up with the following lines:

Listen 443
SSLCertificateFile "/private/etc/apache2/tls/server.pem"
SSLCertificateKeyFile "/private/etc/apache2/tls/server-key.pem"

Test that the configuration is at least syntactically correct:

sudo apachectl configtest

If you see Syntax OK at the end of that, you are good, so go ahead and restart Apache

sudo apachectl restart

With the generic Apache config on Mac, you should be able to go to https://localhost/ (or the domain we set up earlier: https://local.example.com/) and see the “It Works!” banner. If not, e.g. you see “connection refused” or something like that, look at the error logs as you load the page:

sudo tail -f -n20 /var/log/apache2/error_log

Reverse Proxy

The last step here is setting up the reverse proxy. Sometimes this is referred to as just a proxy, but there are differences between a proxy and a reverse proxy. A network proxy is something you knowingly connect use in order to connect to something else, for example a corporate network may use a Squid proxy to only allow certain internet traffic, so you configure your system to use that server as a proxy so that you ask it to, for example, handle a request to github.com, whereas with a reverse proxy, you go to the site that you want, but that “edge” web server then decides to actually send your request to another web server. If you had several Node services running on various ports on a single server, you could set up a reverse proxy to send requests to specific services based on the URI path, etc., but the end user would see it all as one logical service.

To enable the reverse proxy capabilities in the Apache web server, we will again need to edit the /private/etc/apache2/httpd.conf file as root, this time uncommenting or adding the following:

LoadModule xml2enc_module libexec/apache2/mod_xml2enc.so
LoadModule proxy_html_module libexec/apache2/mod_proxy_html.so
LoadModule proxy_module libexec/apache2/mod_proxy.so
LoadModule proxy_connect_module libexec/apache2/mod_proxy_connect.so
LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
Include /private/etc/apache2/extra/proxy-html.conf

Now we will configure the actual revers proxy, but rather than mess with the proxy-html.conf file, we’ll modify httpd-vhosts.conf, by editing (as root) /private/etc/apache2/extra/httpd-vhosts.conf to change our previous:

<VirtualHost *:443>
    ServerName local.example.com
    SSLEngine On
    SSLCertificateFile /private/etc/apache2/tls/server.pem
    SSLCertificateKeyFile /private/etc/apache2/tls/server-key.pem
    SSLProxyEngine On
    ProxyRequests Off
    ProxyVia Off
    <Proxy *>
         Require all granted
    </Proxy>
    ProxyPass "/example"  "http://www.example.com/"
    ProxyPassReverse "/example"  "http://www.example.com/"
    ProxyPass "/domains/reserved" "https://www.iana.org/domains/reserved"
    ProxyPassReverse "/domains/reserved" "https://www.iana.org/domains/reserved"
</VirtualHost>

Now we restart the web server once again:

sudo apachectl restart

This change will cause the / path to behave the same as before, but now the /example and /domains/reserved paths will use the reverse proxy.

However… we see that the iana reverse proxy has some missing resources, because it assumes some absolute paths. This will not be uncommon. To figure out the paths that are needed, you can go to your local site in a browser and open the developer console and look at the errors. In this case, the following reverse proxies would need to be set up (just add them into the same VirtualHost element):

    ProxyPass "/_css" "https://www.iana.org/_css"
    ProxyPassReverse "/_css" "https://www.iana.org/_css"
    ProxyPass "/_js" "https://www.iana.org/_js"
    ProxyPassReverse "/_js" "https://www.iana.org/_js"
    ProxyPass "/_img" "https://www.iana.org/_img"
    ProxyPassReverse "/_img" "https://www.iana.org/_img"

Even though you can muscle your way through resolving conflicts like this, it will probably be easier by just mapping the reverse proxy from root, to root, when setting up a reverse proxy to what would have been an external site. Perhaps a better example of a setup that would use multiple reverse proxies would be the previously described scenario of having several microservices, each running on a different port, which could be represented by something like this:

<VirtualHost *:443>
    ServerName local.example.com
    SSLEngine On
    SSLCertificateFile /private/etc/apache2/tls/server.pem
    SSLCertificateKeyFile /private/etc/apache2/tls/server-key.pem
    SSLProxyEngine On
    SSLProxyVerify None
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off
    ProxyRequests Off
    ProxyVia Off
    <Proxy *>
         Require all granted
    </Proxy>
    ProxyPass "/"  "http://localhost:8080/"
    ProxyPassReverse "/"  "http://localhost:8080/"
    ProxyPass "/auth" "http://localhost:8081"
    ProxyPassReverse "/auth" "http://localhost:8081"
    ProxyPass "/api" "http://localhost:8082"
    ProxyPassReverse "/api" "http://localhost:8082"
    ProxyPass "/cache" "http://localhost:8083"
    ProxyPassReverse "/cache" "http://localhost:8083"
</VirtualHost>

Conclusion

This is a strategy that I have found many, many uses for. Hopefully you have found something useful here as well.

Using cert-manager in Kubernetes

Fri, 08 Jan 2021 00:19:29 +0000

TLS - What is it?

TLS stands for Transport Layer Security, and it represents one of the cornerstones of network security. Any web request that you make while specifying https as the protocol should be using TLS, in particular, TLS 1.2 or newer. Previously, web servers may have used SSL, Secure Socket Layer, in order to encrypt traffic, but SSL is deprecated, although you may still here people refer to SSL out of old habit.

TLS uses PKI, Public Key Infrastructure, which is a way of allowing a public key and a private key to be used separately to encrypt and decrypt information. There is a lot that goes on during the “TLS Handshake”, but the concept to keep in mind here is that in order to use TLS, you are going to need a public key, which anyone can have access to (‘server hello’ in a standard TLS Handshake is one way of distributing it, or you may pre-distribute it in certain scenarios), and a private key, that you protect dearly and never share. In fact, in order to make sure that even if your certificate did get accidentally leaked that it wouldn’t be valid forever, certs must have an expiration date no more than 398 days in the future, otherwise browsers may refuse to serve the content.

One last important concept regarding TLS is that of Certificate Authority, which is how you can not only know that your data is encrypted between you and the server, but that you have confidence that you are actually talking to the real server (see man-in-the-middle attack for more detail). In order for this to all work, there are certain special “Root” Certificate Authorities, or Root CAs, that will be part of every modern operating system and/or web browser, giving the building blocks of trust for every publicly valid cert in the world. It is possible, and sometimes necessary, to use certificates that are not signed by one of these CAs or their intermediates, but it will require some extra work in order to establish trust.

Why secure your traffic?

It may be obvious that you would want to encrypt sensitive information that a user is sending, such as a form POST containing credit card information (for legal requirements, look into PCI compliance), but there are other reasons to secure your traffic. For example, if you have a website that loads JavaScript via HTTP, it would be trivial for a bad actor to intercept the traffic and replace it with malicious code. If you whole site is being served over HTTP then it’s just that much easier for someone to, for example, mess with DNS and take over your entire site with no one even noticing. Other reasons for using TLS are for the comfort a user has seeing the green padlock in their browser (aesthetic may vary), improving search rank, and taking advantage of HTTP/2, which requires TLS. While the focus here is going to be on securing traffic that is coming out of a Kubernetes cluster, often called “north-south” traffic, keep in mind that there are good reasons for securing the “east-west” traffic that occurs within your cluster. Also be aware that securing east-west traffic with an internal/non-standard CA will likely require installing that CA for every container in your cluster.

TLS in Kubernetes

Manual Cert Management

Note: The extensions/v1beta1 and networking.k8s.io/v1beta1 versions of Ingress are deprecated as of Kubernetes 1.19 and are slated to have support removed at some point in Kubernetes 1.22 or later. For Kubernetes 1.19 and later you should use networking.k8s.io/v1 instead.

Perhaps the easiest way to apply TLS to your outbound traffic is by configuring it as part of an ingress configuration. To do this, you just need the following in you ingress definition:

spec:
  tls:
  - hosts:
      - manual.secure-example.com
    secretName: manual-tls

This will cause the ingress controller to use the secret (stored in teh same namespace) with the given name, “manual-tls” in the example. The secret will need to contain data for tls.crt and tls.key, and will be of type kubernetes.io/tls. The tls.crt will be the base64 encoded contents of the public key, and tls.key will be the base64 encoded contents of the private key. You can manually create a secret from key pair files using something like this:

# Note: piping the dry-run output of a create statement to an apply makes it easier to update the secret later, without having to delete and recreate
kubectl create secret tls manual-tls --cert=tls.crt --key=tls.key --dry-run=client -o yaml | kubectl apply --record=true -f -

You can see that I pipe the dry-run output of a create command to an apply command. This is to make it easier to update the secret, without having to delete and recreate the secret. Since it is in your best interest to have certs that are as short lived as is feasible, you will want to make sure these types of secrets are very easy to update.

Here is a full example of creating a self signed certificate (i.e. a certificate not signed by a trusted root CA), creating a secret from that certificate, and using it in an ingress:

mkdir -p ~/openssl-ca
cd ~/openssl-ca

##### Creating the CA
# Generate a private key for our self signed CA,
openssl genrsa -out ca.key 2048
# Use the private key to create a cert for our self signed CA, valid for 5 years (CAs can be longer lived than our TLS certs)
openssl req -x509 -new -nodes -key ca.key -subj "/CN=openssl-ca" -days 1825 -out ca.crt

##### Creating our TLS key pair
# Generate the private key for our TLS certificate
openssl genrsa -out tls.key 2048
# Create a CSR (certificate signing request) configuration
cat >csr.conf <<EOF
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
CN = manual.secure-example.com

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = secure-example
DNS.2 = manual.secure-example.com
EOF
# Create the CSR
openssl req -new -key tls.key -out tls.csr -config csr.conf
# Use the CSR and our CA to generate the public key for our TLS certificate, valid for 6 months
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 180 -extfile csr.conf

##### Store the key pair as a secret
# Note: piping the dry-run output of a create statement to an apply makes it easier to update the secret later, without having to delete and recreate
kubectl create secret tls manual-tls --cert=tls.crt --key=tls.key --dry-run=client -o yaml | kubectl apply --record=true -f -

##### Creating and exposing Nginx pod
cat <<EOF | kubectl apply --record=true -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: manual-secure-web
  name: manual-secure-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: manual-secure-web
  template:
    metadata:
      labels:
        app: manual-secure-web
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: manual-secure-web
  name: manual-secure-web
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: manual-secure-web
EOF

##### Creating ingress with TLS (kubernetes 1.19+)
cat <<EOF | kubectl apply --record=true -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: manual-secure-web
  name: manual-secure-web
spec:
  tls:
    - hosts:
        - manual.secure-example.com
      secretName: manual-tls
  rules:
    - host: manual.secure-example.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: manual-secure-web
              port:
                number: 80
EOF

Now in order to test this out, you can modify /etc/hosts (C:\Windows\System32\drivers\etc\hosts on Windows) to add an entry for something like:

172.29.97.225 manual.secure-example.com

Replace the IP address with whatever the IP address is for one of your worker nodes. You should be able to find this with kubectl get nodes -o wide, but only if the nodes are not behind a firewall and/or load balancer.

With /etc/hosts updated you can open a browser to https://manual.secure-example.com/, however you will notice a warning about the certificate being untrusted. This is because our CA is not a trusted root CA. If you choose the advanced options you can proceed any way. Either way, you can look at the cert information and see that it was issued by openssl-ca, the CA that we created earlier. You can see similar results with curl by ignoring cert errors

This is working, but it took a lot of work, and now we need to monitor our certs to make sure they get replaced before they expire. This is where cert-manager

cert-manager

cert-manager gets installed into your cluster and can automatically create/replace certificates to ensure that they never expire. First things first, let’s install cert-manager into the cluster:

kubectl apply --record=true -f https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml

You will now have a cert-manager namespace that should have some pods coming up. Wait until they are ready before proceeding.

With cert-manager up and running you are ready to install an Issuer, which is what will actually create or pull the certificates. There are multiple types of Issuers which can fit different scenarios. If your cluster will be able to be reached by the public, you may wish to use the ACME Issuer, as it can be free and gets you certs that are associated with a trusted root CA that basically any browser will trust right away. For this example, I will show a CA Issuer, which will work similarly to the prior example in that it will use the certificate authority that we generated from openssl. In a development environment, it could be feasible to have a self signed CA that is shared amongst everyone and added to each machine’s trusted certs in order to make things more seamless.

First, we will need to load the CA key pair into a secret:

cat <<EOF | kubectl apply --record=true -f -
apiVersion: v1
kind: Secret
metadata:
  name: ca-key-pair
data:
  tls.crt: $(cat ca.crt | base64 -w0)
  tls.key: $(cat ca.key | base64 -w0)
EOF

And now we create the Issuer:

cat <<EOF | kubectl apply --record=true -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: ca-key-pair
EOF

We now have an Issuer that can be used in the default namespace. Alternatively we could have created a ClusterIssuer, which would be available across the whole cluster.

With our new Issuer at the ready, you can now configure an ingress to use it by adding the following annotations:

metadata:
  annotations:
    cert-manager.io/issuer: ca-issuer

All the steps together:

mkdir -p ~/openssl-ca
cd ~/openssl-ca

##### Install cert-manager
kubectl apply --record=true -f https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml

##### Creating the CA
# Generate a private key for our self signed CA,
openssl genrsa -out ca.key 2048
# Use the private key to create a cert for our self signed CA, valid for 5 years (CAs can be longer lived than our TLS certs)
openssl req -x509 -new -nodes -key ca.key -subj "/CN=openssl-ca" -days 1825 -out ca.crt

##### Store the key pair as a secret
cat <<EOF | kubectl apply --record=true -f -
apiVersion: v1
kind: Secret
metadata:
  name: ca-key-pair
data:
  tls.crt: $(cat ca.crt | base64 -w0)
  tls.key: $(cat ca.key | base64 -w0)
EOF

##### Wait for cert-manager to be ready
echo "Waiting for cert-manager pods to come up (should take just a few seconds)"
while :; do
  [[ $(kubectl get po -n cert-manager --no-headers --field-selector='status.phase!=Running' 2>/dev/null | wc -l) -eq 0 ]] && break || sleep 1
done


##### Create our CA issuer
cat <<EOF | kubectl apply --record=true -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: ca-key-pair
EOF

##### Creating and exposing Nginx pod
cat <<EOF | kubectl apply --record=true -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cm-secure-web
  name: cm-secure-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cm-secure-web
  template:
    metadata:
      labels:
        app: cm-secure-web
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cm-secure-web
  name: cm-secure-web
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: cm-secure-web
EOF

##### Creating ingress with TLS (kubernetes 1.19+)
cat <<EOF | kubectl apply --record=true -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: ca-issuer
  labels:
    app: cm-secure-web
  name: cm-secure-web
spec:
  tls:
    - hosts:
        - cm.secure-example.com
      secretName: cm-tls
  rules:
    - host: cm.secure-example.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: cm-secure-web
              port:
                number: 80
EOF

Very quickly we should see a new secret created as cm-tls, which was created by cert-manager. We can see take a peek at the cert by base64 decoding it and passing it to openssl:

kubectl get secrets cm-tls -o jsonpath="{.data['tls\.crt']}" | base64 -d | openssl x509 -text -noout

Once again we can add an entry to /etc/hosts, this time for cm.secure-example.com, and then check out https://cm.secure-example.com in our web browser. As before, we will see the trust warnings about our certificate, which we can override if we feel like it. If you were going to use this type of setup for any real usage, e.g. as a persistent development environment, I would definitely suggest coming up with a strategy for trusting the CA that you are using, but keep in mind that anyone with access to the CA can use it to generate certificates that look just like your cluster’s certs, which would allow for an easy man-in-the-middle attack.

Wrap-up

Here I hopefully made the case for why you should use TLS for all of your Kubernetes clusters’ north-south traffic, and showed how you can simplify TLS certificate management in Kubernetes by using cert-manager. This post is far from exhaustive, so make sure you do more follow up specific to your needs, but it should get you started on your way.

Installing OpenStack in Azure

Sun, 16 Jun 2019 02:35:18 +0000

I recently started a new job where I am working with OpenStack and Kubernetes. I was curious about what it would take to stand up an OpenStack test environment, so I fired up a VM in Azure and went to town. The VM I created was a CentOS 7 based VM with 4 cores and 16GB of RAM (D4s_v3), and I gave it 120GB of standard SSD storage. In order to make things easier in the future, I set up a DNS label for the VM and then when I was ready to connect ot it I set up my ~/.ssh/config to use azure-openstack as an alias for my DNS name for the server, using the private key and username to align with what I set up for the VM.

Next I had to do some basic housekeeping on the VM to prepare for OpenStack. To make things easier, I ran sudo su - to elevate my priveleges and then ran the following:

# Make sure that only iptables will be used
systemctl disable firewalld NetworkManager
systemctl stop firewalld NetworkManager
systemctl start network
systemctl enable network

# Install the openstack and epel repositories
yum install -y epel-release centos-release-openstack-stein

# Upgrade all the things
yum upgrade -y

# And just to be on the safe side...
reboot

Once the VM came back up, it was time to get OpenStack up and running. To do this, I used PackStack, once again running as root:

yum install -y openstack-packstack

There is now a packstack command available. The next thing I did was to generate an answers file so that I could tweak the setup:

packstack --gen-answer-file=~/packstack.answers

Now I edited the file, changing the CONFIG_DEFAULT_PASSWORD (I used a 30 character alpha-numeric password generated by KeePass) along with with the following:

CONFIG_DEFAULT_PASSWORD=(30 character alpha-numeric password generated by KeePass)
CONFIG_KEYSTONE_ADMIN_PW=(30 character alpha-numeric password generated by KeePass)
CONFIG_KEYSTONE_DEMO_PW=(30 character alpha-numeric password generated by KeePass)
CONFIG_CINDER_VOLUMES_SIZE=4G
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-ex
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-ex:eth0

Now to actually intialize PackStack using the answers file:

packstack --answer-file=~/packstack.answers

This ran for quite some time (~20 minutes). Once it was done I changed the virtualization type for Nova to KVM by uncommenting virt_type=kvm in /etc/nova/nova.conf and commenting the line that says virt_type=qemu. To pick up the change I had to run:

systemctl restart openstack-nova-compute.service

Now there should be a nice shiny dashboard waiting for you, in my case, at http://10.0.1.4/dashboard. Except, that’s internal to Azure’s network, so I needed to enable SSH tunneling. I disconnected my SSH session and started a new one with tunneling:

ssh azure-openstack -L 5000:localhost:80

Now I was able to get to my dashboard at http://localhost:5000 and login as “demo” with the password I had set in my answer file, which was enough to verify the basic installation was valid.

Messing Around with Broadcast Channel API

Mon, 08 Apr 2019 18:19:51 +0000

I had previously written about the Boradcast Channel API specifically in context of using it with Reveal.js, but some folks had asked me about Broadcast Channel more generically, so I create a demo (click here to see it). Source code is also available.

For this example, I chose to use multiple iframes in a single window, not because I think it’s a good idea, but because it’s easier to demonstrate.

The Broadcast Channel API

Currently available in Chrome and Firefox, this API uses Web Workers behind the scenes, and can allow communication between different windows, tabs, frames, and iframes with the same origin, which are situations you might handled through AJAX requests in the past.

Basic Usage

There are really only four things you can do with the Broadcast Channel API: create/join a channel, post a message, listen for messages, and leave a channel.

To join a broadcast channel, just call the constructor. All you need to know is the name of the channel (in this case, crossFrameCommunication), and then just call the constructor like this:

const broadcastChannel = new BroadcastChannel('crossFrameCommunication');

From this point you can start posting and receiving messages.

To post a message, just call the postMessage method on the BroadcastChannel object and pass in any object (JSON, DOMString, etc.). Here’s an example of passing in a JSON value:

broadcastChannel.postMessage({
    user: 'brendon',
    product: 'banana',
    quantity: 8
});

To receive messages, add a listener for the onmessage event. When you respond to the event, the message will be in the data property of the event. Here a simple listener:

broadcastChannel.onmessage = function (event) {
    console.log(`${event.data.user} wants to buy ${event.data.quantity} ${event.data.product}`);
};

Note that any messages posted by an instance (window, tab, frame, or iframe) will not trigger its own onmessage event.

Things to Keep in Mind

First and foremost, this API only works with the same origin, so for different origins (different domain, a sub-domain, etc.), the Broadcast Channel API won’t work.

The example I created is obviously very contrived, but I found it to be a simple way of showing the power of this API. That said, the way that I created this example has what are essentially four different web pages, three being loaded into a fourth, and as such, their resources are completely separate, meaning that, for example, the main.css is downloaded four times, since each page references it. If you want to have more control over shared resources, you should look into the SharedWorker API, where you can have a lot more control.

The reason I found this API in the first place was that I wanted to synchronize multiple windows of the same thing, in particular, a Reveal.js presentation, for which I found the Broadcast Channel API to be very effective. You can see a post I made about it here: Presenter Mode for Reveal.js

Switching Text Delimiters in PowerShell

Sat, 06 Apr 2019 18:37:23 +0000

I am a huge fan of bcp for loading seed data into a Microsoft SQL Server database for use in a Continuous Integration pipeline, however manipulating a flat text file can be a little cumbersome for people who want to add new data to be used during testing. One naive approach would be to use either tab or comma delimiters for bcp’s export and just slap a csv extension on the file. In theory, you should end up with a file that Excel can open. But what if there are commas or tabs in your data? You could try doing a more complex delimiter like ",", but what about quotes in your data?

A coworker and I wanted to allow people to use Excel in order to add or change the seed data, but we couldn’t figure out how to have bcp export the data in a format that could be guaranteed to work with Excel. Thank goodness for the Export-Csv cmdlet!

Export-Csv

The Export-Csv cmdlet is pretty straightforward. You provide it with a collection of objects and it will create a CSV file based on the objects, creating a header row with values puled from the property names of the object collection. For some reason, the default behavior is to have a row at the top of the exported CSV file with information about the PowerShell objects that were used to create the CSV, but this can easily be avoided by passing the NoTypeInformation flag. All we had to do to leverage the cmdlet was to convert my bcp formatted file into a collection of objects.

Generating the BCP Export

The one thing we had to do first, was to pick a delimiter to use with bcp. Thankfully, it can handle multi-character delimiters, so we picked !~! as something that never appeared in our actual data. To create the export, we did it in two steps, one grabbing the headers and the other grabbing the actual data. We stored these pieces as two separate files, let’s call them export_columns.dmp and export.dmp. To generate these files we used a script like this:

$SourceServer = "(LocalDB)\MSSQLLocalDB"
$SourceSchemaName = "dbo"
$SourceDatabaseName = "MyDatabase"
$Table = "MyTable"
$ExportFileName = "export.dmp"
$ExportFileColumnsName = "export_columns.dmp"

# Query to get a delimited list of column names
$Query = "Select Stuff(
        (
        Select '!~!' + C.name
        From $SourceDatabaseName.sys.COLUMNS As C
        Where c.object_id = t.object_id
        Order By c.column_id
        For Xml Path('')
        ), 1, 3, '') As Columns
From $SourceDatabaseName.sys.TABLES As T
WHERE t.NAME = '$Table'"

bcp.exe $query queryout $ExportFileColumnsName -S "$SourceServer" -T -c -t'!~!' -r\n
bcp.exe "$SourceDatabaseName.$SourceSchemaName.$Table" out $ExportFileName  -S "$SourceServer" -T -c -t'!~!' -r\n

Quick notes on the bcp parameters used:

-T uses “integrated security”, logging you into the database using the current network login
-c treats all columns as char columns
-t specifies the field delimiter, !~!
-r specifies the line terminator, \n

Reading the Export

The next step was to read in the bcp export and split the rows by the delimiter. The test file we were playing with was a few gigabytes in size, so we knew we’d have to do some batching of he input So far things were pretty straightforward, and we ended up with something like the following pseudo-code:

$RecordDelimiter = "!~!"
$ExportFileName = "export.dmp"
$CsvFileName = "export.csv"
$ReadBatchSize = 1000
Get-Content "$ExportFileName" -Read $ReadBatchSize -Encoding ASCII | ForEach-Object {
    foreach ($row in $_) {
        $csvValues = $row -split "$RecordDelimiter"
        # Convert the row to an object and add it to the collection
        ...
    }
    $newCsvRecords | Export-Csv -Path $CsvFileName -NoTypeInformation -Append -Encoding ASCII
};

This part actually stayed pretty much the same through all the iterations of this experiment. Creating the objects, on the other hand, went through many improvement cycles before we got a solution that we found to be sufficient.

Object Creation

Because we wanted to make this process be able to handle any table’s data, with any column names and any number of columns, we needed to dynamically define my PowerShell objects. I thought I had a decent approach by messing with NoteProperty values, like this original (and very slow) approach:

$csvValues = $_[$row] -split "$RecordDelimiter"
$csvRecord = New-Object -TypeName psobject
for ($col = 0; $col -lt $columnHeaders.Count; $col++) {
    $csvRecord | Add-Member -MemberType NoteProperty -Name $columnHeaders[$col] -Value $csvValues[$col]
}
$newCsvRecords.Add($csvRecord) | Out-Null

As it turns out, all this messing with objects was very, very inefficient. After a lot of messing around, we discovered that you can very quickly and easily create a hash table and then simply cast it as a pscustomobject, making the following code an order of magnitude faster:

$csvValues = $row -split "$RecordDelimiter"
$csvRecord = [ordered]@{}
for ($col = 0; $col -lt $columnHeaders.Count; $col++) {
    $csvRecord += @{ $columnHeaders[$col] = $csvValues[$col] }
}
$newCsvRecords.Add([PSCustomObject]$csvRecord) | Out-Null

Note that this is an “ordered” hash table, which makes sure that the columns in the CSV stay in the same order as in the bcp export.

Side Quest: Collections

In the sample code in the previous section I’m showing the Add method of an ArrayList, but that was not the original strategy. In the first iteration we just used a standard PowerShell array. We came across a good article, How To Create Arrays for Performance in PowerShell, that explains why you should definitely prefer an ArrayList when needing to append lots of objects. This is mainly due to the fact that when you append to a PowerShell array by using += a full copy of the array is made with the new element appended to it.

PowerShell Array: The Bad Way

$psArray = @()
foreach ($thing in $lotsOfThings) {
    $psArray += $thing
}

ArrayList: The Better Way

[System.Collections.ArrayList]$theList = @()
foreach ($thing in $lotsOfThings) {
    $theList.Add($thing)
}

I learned quite a few things about PowerShell during this venture, and hopefully this post helps some others kick start their PowerShell journey. I’m modifying the source code to make it more generic/general purpose and putting it on GitHub as bcp-to-csv-to-bcp.

Presenter Mode for Reveal.js

Tue, 02 Apr 2019 19:40:27 +0000

I love using Reveal.js. A few of the itches it scratches for me are that it allows me to use version control easily, it supports putting code in your slides, and it can be hosted on a webpage. If you want to see some of the presentations that I’ve written using Reveal.js, you can go to my presentations page.

There was, however, one thing that I really wanted to do, was to be able to display the presentation on one screen for the audience, but have a different view on my own screen, while still having the controls match up. I know I can do this using slides.com, along with many other very cool things, but I wanted to keep things all running on my laptop. After some looking around I learned about the Broadcast Channel API that I can use from Chrome. The concept is quite simple: you create a channel and then either post to the channel or react to an onmessage event. I decided that what I was going to do was to create a presenter tab (just adding a query parameter to the URL of the existing slide deck) and have any “non-presenter” instances move to one slide prior to the slide of the presenter on the slidechanged event of the presenter. This turned out to be very simple. I just added the following code to my presentation’s javascript after Reveal.initialize was called:

    Reveal.initialize({
      dependencies: [
        {
          src: 'plugin/highlight/highlight.js',
          async: true,
          callback: function () { hljs.initHighlightingOnLoad(); }
        }
      ],
      history: true
    });

    const controlChannel = new BroadcastChannel('controller');
    const urlParams = new URLSearchParams(window.location.search);
    const mode = urlParams.get('mode');

    if (mode === 'presenter') {
      Reveal.addEventListener('slidechanged', function (event) {
        controlChannel.postMessage({
          "presenterSlide": event.indexh
        });
      });
    } else {
      controlChannel.onmessage = function (ev) {
        Reveal.slide(ev.data.presenterSlide - 1);
      }
    }

A major assumption here is that you will only use horizontal slide progressions (I personally do this almost exclusively to make it easier to use a “clicker” device). Another flaw in this approach was that I couldn’t show the last slide to the audience. I initially just added a dummy slide to the end so that the presenter would land on that and the audience would see the slide prior, however I didn’t like that the progress indicator showed the audience that there was something else. In the end I decided to add a class of “presenter” to slides that should only be in the presenter’s view and I just removed those from the DOM prior to calling Reveal.initialize. This resulted in the following code:

    const controlChannel = new BroadcastChannel('controller');
    const urlParams = new URLSearchParams(window.location.search);
    const mode = urlParams.get('mode');

    if (mode !== 'presenter') {
      document.querySelectorAll('.presenter').forEach((elem) => { elem.parentNode.removeChild(elem); });
    }

    Reveal.initialize({
      dependencies: [
        {
          src: 'plugin/highlight/highlight.js',
          async: true,
          callback: function () { hljs.initHighlightingOnLoad(); }
        }
      ],
      history: true
    });

    if (mode === 'presenter') {
      Reveal.addEventListener('slidechanged', function (event) {
        controlChannel.postMessage({
          "presenterSlide": event.indexh
        });
      });
    } else {
      controlChannel.onmessage = function (ev) {
        Reveal.slide(ev.data.presenterSlide - 1);
      }
    }

Here’s a sample of what it looks like in action, where the presenter view (bottom) is using the overview feature of Reveal.js:

If you want to try it out yourself you can open this as the presenter tab and in another tab open, this as the audience view. If you change the slide in the presenter tab you should see that the audience tab gets updated to be one prior.

To view the full source for how I use my presentations from Jekyll, you can look at my GitHub repo.

Edit

After just one use, I realized that I did not actually want to have the presenter to be a slide ahead. This actually greatly simplifies things. I technically got it working with just a very naive approach of just:

    const controlChannel = new BroadcastChannel('controller');

    Reveal.initialize({
      dependencies: [
        {
          src: 'plugin/highlight/highlight.js',
          async: true,
          callback: function () { hljs.initHighlightingOnLoad(); }
        }
      ],
      history: true
    });

    Reveal.addEventListener('slidechanged', function (event) {
      controlChannel.postMessage({
        "indexh": event.indexh,
        "indexv": event.indexv
      });
    });
    controlChannel.onmessage = function (event) {
      Reveal.slide(event.data.indexh, event.data.indexv);
    }

Notice that this will allow for presentations that have vertical slides as well, so that’s cool, but there is an extra event that happens where the audience view will also trigger a message post to the broadcast channel. Like I said, this technically worked, but I wanted to be a little smarter, so I added a kind of tracker to let me check where I had gone at the behest of a remote change. I also thought it would be good to allow changes to be synced both ways. Where I ended up was with this:

    const controlChannel = new BroadcastChannel('controller');
    let remoteSlide = { indexh: -1, indexv: -1 };

    Reveal.initialize({
      dependencies: [
        {
          src: 'plugin/highlight/highlight.js',
          async: true,
          callback: function () { hljs.initHighlightingOnLoad(); }
        }
      ],
      history: true
    });

    Reveal.addEventListener('slidechanged', function (event) {
      if (remoteSlide.indexh !== event.indexh || remoteSlide.indexv !== event.indexv) {
        remoteSlide = { indexh: -1, indexv: -1 };
        controlChannel.postMessage({
          "indexh": event.indexh,
          "indexv": event.indexv
        });
      }
    });
    controlChannel.onmessage = function (event) {
      remoteSlide = event.data;
      Reveal.slide(event.data.indexh, event.data.indexv);
    }

A remtoteSlide value of { indexh: -1, indexv: -1 } means “the last slide change was done in this tab”, whereas any other value is checked against the current slide, and if they are the same values for indexh and indexv, the tab assumes it ended up there because a remote tab told it to change, otherwise, the tab again assumes it made the change, and therefore posts a message for any other tabs to change as well.