Ever since switching to orgmode as my GTD system, I've been trying to (slowly) optimize my capture system for things coming to me from different places. One of the things that I was not satisfied with was capturing and processing inputs from Slack.

Slack messages can easily get lost in the fast-paced stream of conversations. Manually copying and pasting messages introduces too much friction, and also removes some context (backlinks), unless a link to the original message is also manually copied. There are ways to save messages in Slack itself, but it just makes it another Inbox to check, and I'm trying to reduce that.

I started by using the saved messages feature to save messages in a single place, and later either shifting them to my org inbox manually, or directly working on them and completing them. Then, I shifted to using the Slack Todoist integration to save slack messages to Todoist, and pulling them from Todoist into my org Inbox.

I've now found a better mechanism that allows me to seamlessly capture Slack message with it's context into orgmode. Here's a demo:

Demo breakdown

This method uses userscripts to add a custom button to the hover menu that comes up for any message in slack, and triggers org protocol capture with the message details when the button is clicked. The message details include the sender, message text, and a direct link back to the message. I've set up this protocol handler to ask me to enter the heading of the capured item, but it can be as easily set up to directly capture the message without user input.

Setup and Implementation

Prerequisites

1. Slack running in a browser (instead of a desktop app)
2. Browser extention for userscripts (Tampermonkey, Violentmonkey, Greasemonkey, etc)
3. Emacs with orgmode installed
4. Org protocol setup

I didn't find a good way to inject userscripts into the Slack destop app, so for now, this method requires using Slack in a browser. It also works when Slack is installed using the "Install Page as App" feature of the browser.

Update: I got a comment that using this in the Slack app is also possible, though I've not evaluated this yet. You can check this comment for details.

Setting up Org Protocol Capture

Setting up org protocol capture involves two steps: configuring your OS to use Emacs to open org-protocol:// links, and configuring Emacs to save the captured data as you want.

For OS setup, please check the guides for your OS here: https://orgmode.org/worg/org-contrib/org-protocol.html#orge00964c

On Emacs side, this is the minimal config required:

(server-start)
(setq-default org-agenda-files '("~/org"))
(setq-default my-org-inbox
    (expand-file-name "inbox.org" "~/org"))
(setq-default org-capture-templates
      '(    
    ("i" "Inbox" entry (file my-org-inbox)
         "* %?\n%i\n%U"
         :kill-buffer t)
    ("l" "Inbox with link" entry (file my-org-inbox)
         "* %?\n%i\n%a\n%U"
         :kill-buffer t)))
(setq-default org-protocol-default-template-key "l")
(require 'org-protocol)

An optional enhancement that can be seen in the demo is: open a new emacs frame to capture this message, then close it automatically after the capture has been done. For this, I use the config snippet from prot's excellent post: https://protesilaos.com/codelog/2024-09-19-emacs-command-popup-frame-emacsclient/

To run emacsclient with the popup frame parameter, I use:

emacsclient --create-frame -F \
    '((prot-window-popup-frame . t) (name . "org-protocol-capture") (width . 80))' \
    -- %u

Emacsclient's args can be set in the desktop entry file (linux) or org-protocol.app file (OSX) when setting up org-protocol.

The userscript

For the userscript, I searched for an existing published userscript for Slack that adds a button, then tweaked it a bit to configure the button according to my needs.

I've published the userscript here: https://greasyfork.org/en/scripts/521908-slack-org-protocol-capture

From this, I learned about an interesting API called MutationObserver that seems very useful for userscripts.

Notes

Using emacs-slack

Another simple approach for this can be to use the emacs-slack package to directly use Slack from Emacs. I tried this, and it does not work very well for me because:

1. My org limits Slack session authentication to 1 week, and authenticating in emacs-slack is a little cumbersome.
2. We also use Slack huddles, and it does not work well with emacs-slack.

Possible Improvements

1. Make org capture template configurable
2. Add tooltip to the button
3. Pass even more metadata like message timestamp, channel name, emojis, etc.
4. Maybe some keyboard shortcuts to trigger the capture?

Limitations / Drawbacks

Since this is dependent on the HTML structure of the slack web app, it can be fragile and can break easily if there are changes in the app.

This userscript uses the MutationObserver API to observe DOM changes in the whole slack workspace. So, it has some impact on performance. However, I've been using this daily for the last several months and I have not noticed any issues.

The first Emacs 30.0.91 pretest for what will be the 30.1 release of Emacs was announced on the mailing list a few days ago. I was going through the NEWS file, and found something that I've wanted in Emacs for a while now.

One of the niggles I had with Emacs was its tab behavior. It worked differently from how other applications that I'm used to work (firefox, kitty, etc).

In emacs, tab-per-buffer can be achieved by using the tab-line-mode. But, before now, it had the following problems:

1. Since tab-line-mode listed buffers sorted by recently visited, so the order or tabs kept changing
2. There was no wrap-around when trying to go to the next tab from the last tab

Here's a video showing the old behavior:

To solve this, there is package called intuitive-tab-line-mode that solves the above two problems. But, I had problems with it because it did not work well with the beframe package that I also use.

Now, with Emacs 30.1, this behavior is what comes out-of-the-box with Emacs. The only config needed is to enable global-tab-line-mode. And because it's built-in, it works with other modes like beframe.

Here's my config for minimal intuitive per-buffer tabs in Emacs >= 30.0.91:

(use-package tab-line
  :demand t
  :bind
  (("C-<iso-lefttab>" . tab-line-switch-to-prev-tab)
   ("C-<tab>" . tab-line-switch-to-next-tab))
  :config
  (global-tab-line-mode 1)
  (setq
   tab-line-new-button-show nil
   tab-line-close-button-show nil))

Here, I've also added keybindings to switch to next/prev tab using C-TAB and C-S-TAB keys.

More details about this change can be found in it's bug report mail thread.

Aside: why show buffers as tabs

One of the common questions is the need to show buffers as tabs at all. After all, in a long running Emacs session, there might be hundreds of buffers, and showing all of them as tabs becomes useless.

For me, I find that I usually work on a 2 to 5 buffers for a "purpose" at a time. These might be some files on a project I'm working on, or org-agenda + a couple of org files, or mu4e-main + headers + one or two emails and a reply. When in this mode, I think looking at all of these buffers in the tab bar gives me a good understanding of where I am in the project and makes it easy to switch to the next or previous buffer easily.

I also use frames and beframe to make sure that only a single "project" is showing in a frame at a time. So, even if there are hundreds of buffers in my Emacs session, only a handful are shown as tabs. This makes it useful to me without overwhelming me with too many tabs.

Syndicated to:

• https://bsky.app/profile/srijan4.bsky.social/post/3l36vuzxpbp24

I've been using Emacs' orgmode as my GTD system for the last several months. I migrated from Todoist, and one of the things I missed was integration with other services that make things easy to capture tasks into the inbox.

There are ways to capture org data via email, and this can be a good enough alternative, because most (though not all) services allow some kind of email forward to capture tasks. But, this depends on a complex email setup and would probably work on a single machine.

The main integrations/features I wanted to use were:

1. Slack: Todoist has a native app for Slack using which any Slack message can be captured into Todoist as a task.
2. Google Assistant: Todoist has integration with Google Assistant which can be used to capture tasks by talking to the google assistant.
3. Todoist quick entry on mobile with date recognition: The Todoist apps have a more polished mobile capture system that can be triggered from a widget and can recognize dates using natural language entry.

The workflow would look something like this:

I found an existing Todoist integration for emacs, but it's more suitable for using Todoist as the source of truth for tasks, and keeping a local buffer for operations on it in Emacs.

But, I was able to use it's functions to achieve what I wanted. Here's my elisp:

(use-package todoist)

(defun fetch-todoist-inbox ()
  (interactive)
  (let ((tasks (todoist--query "GET" "/tasks?project_id=<project_id>")))
    (mapcar (lambda (task)
              (todoist--insert-task task 1 t)
              (todoist--query
                "DELETE"
                (format "/tasks/%s" (todoist--task-id task))))
            tasks)))

Here, the project_id is the Todoist project id of the project from which tasks have to be imported. This can be found by opening the project in the todoist web app - the project id is the last part of the URL.

The elisp function fetch-todoist-inbox can be called when in any org buffer (or any buffer actually). It will fetch all tasks in the specified project, insert then into the current buffer, and delete them from Todoist. It can be bound to any keybinding for easy access. Note that it requires setting the todoist token using elisp or an environment variable.

Improvements Ideas

• Show number of un-fetched items in status bar
• Fetch comments and attachments
• Fetch task labels and show as orgmode tags

#GTD #Emacs #OrgMode

I recently updated my Install docker and docker-compose using ansible post and wanted to test it against multiple target OSes and OS versions. Here's a way I found to do it easily using Vagrant.

Here's the Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :

targets = [
  "debian/bookworm64",
  "debian/bullseye64",
  "debian/buster64",
  "ubuntu/jammy64",
  "ubuntu/bionic64",
  "ubuntu/focal64"
]

Vagrant.configure("2") do |config|
  targets.each_with_index do |target, index|
    config.vm.define "machine#{index}" do |machine|
      machine.vm.hostname = "machine#{index}"
      machine.vm.box = target
      machine.vm.synced_folder ".", "/vagrant", disabled: true

      if index == targets.count - 1
        machine.vm.provision "ansible" do |ansible|
          ansible.playbook = "playbook.yml"
          ansible.limit = "all"
          ansible.compatibility_mode = "2.0"
          # ansible.verbose = "v"
        end
      end
    end
  end
end

The targets variable defines what Vagrant boxes to target. The possible list of boxes can be found here: https://app.vagrantup.com/boxes/search

In the Vagrant.configure section, I've defined a machine with an auto-generated machine ID for each target.

The machine.vm.synced_folder line disables the default vagrant share to keep things fast.

Then, I've run the ansible provisioning once at the end instead of for each box separately (from: https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#tips-and-tricks).

The test can be run using:

$ vagrant up

If the boxes are already up, to re-run provisioning, run:

$ vagrant provision

This code can also be found on GitHub: https://github.com/srijan/ansible-install-docker

This new packaging from BlueTokai looks nice. And the coffee tastes amazing. Chocolatey flavour with very little bitterness.

Background

I use mbsync to sync my mailbox from my online provider (FastMail - referer link) to my local system to eventually use with mu4e (on Emacs).

For periodic sync, I have a systemd service file called mbsync.service defining a oneshot service and a timer file called mbsync.timer that runs this service periodically. I can also activate the same service using a keybinding from inside mu4e.

[Unit]
Description=Mailbox synchronization service
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync fastmail-all
ExecStartPost=bash -c "emacsclient -s srijan -n -e '(mu4e-update-index)' || mu index"

[Install]
WantedBy=default.target

[Unit]
Description=Mailbox synchronization timer
BindsTo=graphical-session.target
After=graphical-session.target

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service

[Install]
WantedBy=graphical-session.target

Also, for instant download of new mail, I have another service called goimapnotify configured that listens for new/updated/deleted messages on the remote mailbox using IMAP IDLE, and calls the above mbsync.service when there are changes.

This has worked well for me for several years.

The Problem

I recently split my (huge) archive folder into yearly archives so that I can keep/sync only the recent years on my phone. [ Aside: yearly refile in mu4e snippet ]. This lead to an increase in the number of folders that mbsync has to sync, and this increased the time taken to sync because it syncs the folders one by one.

It does have the feature to sync a subset of folders, so I created a second systemd service called mbsync-quick.service and only synced my Inbox from this service. Then I updated the goimapnotify config to trigger this quick service instead of the full service when it detects changes.

But, this caused a problem: these two services can run at the same time, and hence can cause corruption or sync conflicts in the mail files. So, I wanted a way to make sure that these two services don't run at the same time.

Ideally, whenever any of these services are triggered and the other service is already running, then it should wait for the other service to stop before starting, essentially forming a queue.

Solution 1: Using systemd features

Systemd has a way to specify conflicts in the unit section. From the docs:

If a unit has aConflicts=setting on another unit, starting the former will stop the latter and vice versa.
[...] to ensure that the conflicting unit is stopped before the other unit is started, anAfter=orBefore=dependency must be declared.

This is different from our requirement that the conflicting service should be allowed to finish before the triggered service starts, but maybe a good enough way to at least prevent both running at the same time.

To test this, I added Conflicts= in both the services with the other service as the conflicting service, and it works. The only problem is that when a service is triggered, the other service is SIGTERMed. This itself might not cause a corruption issue, but if this happens with the mbsync-quick service, then there might be a delay getting the mail.

This is the best way I found that uses built-in systemd features without any workarounds or hacks. Other solutions below involve some workarounds.

Solution 2: Conflict + stop after sync complete

This is a variation on solution 1 - add a wrapper script to trap the SIGTERM and only exit when the sync is complete. This also worked.

But, the drawback with this method is that anyone calling stop on these services (like the system shutting down) will have to wait for this to finish (or till timeout of 90s). This can cause slowdowns in system shutdown that are hard to debug. So, I don't prefer this solution.

Solution 3: Delay start until the other service is finished

This is also a hacky solution - use ExecStartPre to check if the other service is running, and busywait for it to stop before starting ourselves.

[Unit]
Description=Mailbox synchronization service (quick)
After=network-online.target

[Service]
Type=oneshot
ExecStartPre=/bin/sh -c 'while systemctl --user is-active mbsync.service | grep -q activating; do sleep 0.5; done'
ExecStart=/usr/bin/mbsync fastmail-inbox
ExecStartPost=bash -c "emacsclient -s srijan -n -e '(mu4e-update-index)' || mu index"

Here, we use systemctl is-active to query the status of the other service, and wait until the other service is not in activating state anymore. The state is called activating instead of active because these are oneshot services that go from inactive to activating to inactive without ever reaching active.

To not make this an actual busywait on the CPU, I added a sleep of 0.5s.

This worked the best for my use case. When one of the services is triggered, it checks if the other service is running and waits for it to stop before running itself. It also does not have the drawback of solution 2 of trapping exits and delaying a stop command.

But, after using it for a day, I found there is a race condition (!) that can cause a deadlock between these two services and none of them are able to start.

The reason for the race condition was:

• A service is marked as activating when it's ExecStartPre command starts
• I added a sleep of 0.5 seconds

So, if the other service is triggered again in between those 0.5 seconds, both services will be marked as activating and they will indefinitely wait for each other to complete. This is what I get for using workarounds.

Solution 4: One-way conflict, other way delay

So, the final good-enough solution I came up with was to break this cyclic dependency by doing a hybrid of Solution 1 and Solution 3. I was okay with the mbsync.service being stopped for the (higher priority) mbsync-quick.service.

So, I added mbsync.service in Conflicts section of mbsync-quick.service, and used the ExecStartPre method in mbsync.service.

💡Let me know if you know a better way to achieve this.

References

• https://unix.stackexchange.com/questions/503719/how-to-set-a-conflict-in-systemd-in-one-direction-only
• https://unix.stackexchange.com/questions/465794/is-it-possible-to-make-a-systemd-unit-wait-until-all-its-conflicts-are-stopped/562959

I have an Arch Linux system with an unencrypted root partition that I wanted to encrypt. I've documented the steps I followed to achieve this here.

I selected the "LUKS on a partition" option from here. I don't have an LVM setup on this system and didn't need to encrypt the boot partition.

The first step was to have a backup so that if something failed, I could at least recover my critical files. I don't have filesystem-level backups configured, so I used kopia to back up my home folder. Details on this might be in a future blog post.

Process

1. To begin, I set up a USB flash installation medium so that I could boot into a live environment to perform the actual actions. Since I needed to encrypt the root partition, this could not be performed from inside the system running off that partition.

2. After booting into the live environment using the above USB medium, I first shrank the existing filesystem by 32MiB to make space for the LUKS encryption header, which is always stored at the beginning of the device. My filesystem size is exactly 500GiB, so I set the new size to 511968M.

# echo "Check the filesystem"
# e2fsck -f /dev/nvme0n1p7

# echo "Resize"
# resize2fs -p /dev/nvme0n1p7 511968M

3. Now, I encrypted it using the default cipher. This took 37 minutes on my 500GiB partition, which was about 55% full.

# cryptsetup reencrypt --encrypt --reduce-device-size 16M /dev/nvme0n1p7

WARNING!

========

This will overwrite data on LUKS2-temp-12345678-9012-3456-7890-123456789012.new irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for LUKS2-temp-12345678-9012-3456-7890-123456789012.new: 
Verify passphrase:

4. Next, I extended the original ext4 file system to occupy all available space again on the now encrypted partition:

# cryptsetup open /dev/nvme0n1p7 root
Enter passphrase for /dev/nvme0n1p7: 

# resize2fs /dev/mapper/root

5. Now, I mounted the filesystem and chrooted into it:

# mount /dev/mapper/root /mnt
# mount /dev/nvme0n1p1 /mnt/boot
# arch-chroot /mnt

6. Since I have a systemd-based initramfs, I added keyboard, sd-vconsole, and sd-encrypt hooks in the HOOKS section of /etc/mkinitcpio.conf:

HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block sd-encrypt filesystems fsck)

7. Next, I regenerated the initramfs ( -P regenerates it for all presets ):

# mkinitcpio -P

8. Next, I configured the boot loader by adding to kernel parameters:

rd.luks.name=<device-UUID>=root root=/dev/mapper/root

I found the device UUID using: sudo blkid -s UUID -o value /dev/nvme0n1p7. Surprisingly (for me), the UUID had changed after encrypting the partition.

My final bootloader conf file looked like this:

title    Arch Linux
linux    /vmlinuz-linux
initrd   /amd-ucode.img
initrd   /initramfs-linux.img
options  rd.luks.name=1df8ea89-4274-4ef9-a670-76c13e612901=root root=/dev/mapper/root rw

9. Lastly, I updated /etc/fstab:

/dev/mapper/root  /  ext4  rw,relatime  0 1

10. All done. To test it out, I logged out of the chroot environment and rebooted the system.

It asked me for the disk encryption password. After entering the password selected in step 3, the system booted up as usual, and everything looked to be working.

Final Thoughts

This was surprisingly easy to do and did not take much time. The ArchWiki was helpful, even if the information was spread over multiple pages/sections. Taking a backup before starting also made me feel safe about the process.

I did not like the design of the decryption password prompt at bootup. Maybe there's a way to customize it to look better. Update: I found a way. Details here.

Here's how the overall flow will look like:

We want this roundup post triggered on some schedule (maybe daily), so the Cloud Scheduler is required to send an event to a Google Pub/Sub topic that triggers our cloud function, which queries the slack API to get channels details, filter recently created, and post it back to a slack channel. Secret Manager is used to securely store slack's bot token and signing secret.

Note that the credentials shown in any screenshots below are not valid.

Create the slack app

The first step will be to create the slack app. Go to https://api.slack.com and click on "Create an app". Choose "From scratch" in the first dialog; enter an app name and choose a workspace for your app in the second dialog. In the next screen, copy the "Signing Secret" from the "App Credentials" section and save it for later use.

Next, go to the "OAuth and Permissions" tab from the left sidebar, and scroll down to "Scopes" -> "Bot Token Scopes". Here, add the scopes:

• channels:read: required to query public channels and find their creation times
• chat:write: required to write to a channel (where the bot is invited)

Next, scroll up on the same screen and click "Install to Workspace" to install to your workspace. Click "Allow" in the next screen to allow the installation. Next, copy the "Bot User OAuth Token" from the "OAuth Tokens for Your Workspace" section on the same page and save it for later use.

💡Keep track of the Bot User OAuth Token and Signing Secret you copied above.

Post to a Slack channel from a Google Cloud Function

Next, we will try to use the credentials copied above to enable a Google Cloud Function to send a message to a Slack channel.

Google Cloud Basic Setup

We will use gcloud cli for the following sections, so install and initialize the Google Cloud CLI if not done yet. If you already have gcloud cli, run gcloud components update to update it to the latest version.

Create a new project for this if required, or choose an existing project, set it as default, and export the project id as a shell environment for using later. Also export the region you want to use.

export PROJECT_ID=slackbot-project
export REGION=us-central1

gcloud config set project ${PROJECT_ID}

You will have to enable billing for this project to be able to use some of the functionality we require.

You may also have to enable the Secret Manager, Cloud Functions, Cloud Build, Artifact Registry, and Logging APIs if this is the first time you're using Functions in this project. Note that some services like Secret Manager need billing to be setup before they can be enabled.

gcloud services enable --project slackbot-project \
        secretmanager.googleapis.com \
        cloudfunctions.googleapis.com \
        cloudbuild.googleapis.com \
        artifactregistry.googleapis.com \
        logging.googleapis.com

Create a service account

By default, Cloud Functions uses a default service account as its identity for function execution. These default service accounts have the Editor role, which allows them broad access to many Google Cloud services. Of course, this is not recommended for production, so we will create a new service account for this and grant it the minimum permissions that it requires.

SA_NAME=channelbot-sa
SA_EMAIL=${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

gcloud iam service-accounts create ${SA_NAME} \
    --description="Service Account for ChannelBot slackbot" \
    --display-name="ChannelBot SlackBot SA"

Store secrets and give permissions to service account

First, we need to store the secrets in Secret Manager.

printf $SLACK_BOT_TOKEN | gcloud secrets create \
    channelbot-slack-bot-token --data-file=- \
    --project=${PROJECT_ID} \
    --replication-policy=user-managed \
    --locations=${REGION}

printf $SLACK_SIGNING_SECRET | gcloud secrets create \
    channelbot-slack-signing-secret --data-file=- \
    --project=${PROJECT_ID} \
    --replication-policy=user-managed \
    --locations=${REGION}

And give our service account the roles/secretmanager.secretAccessor role on these secrets.

gcloud secrets add-iam-policy-binding \
    projects/${PROJECT_ID}/secrets/channelbot-slack-bot-token \
    --member serviceAccount:${SA_EMAIL} \
    --role roles/secretmanager.secretAccessor

gcloud secrets add-iam-policy-binding \
    projects/${PROJECT_ID}/secrets/channelbot-slack-signing-secret \
    --member serviceAccount:${SA_EMAIL} \
    --role roles/secretmanager.secretAccessor

Create and deploy the function

Here's a simple HTTP function that sends a message to slack on any HTTP call:

import functions_framework
from slack_bolt import App

# process_before_response must be True when running on FaaS
app = App(process_before_response=True)

print('Function has started')

@functions_framework.http
def send_to_slack(request):
    print('send_to_slack triggered')
    channel = '#general'
    text = 'Hello from Google Cloud Functions!'
    app.client.chat_postMessage(channel=channel, text=text)
    return 'Sent to slack!'

functions-framework
slack_bolt

Assuming main.py and requirements.txt are present in src-v1 folder, deploy using:

gcloud beta functions deploy channelbot-send-to-slack \
    --gen2 \
    --runtime python310 \
    --project=${PROJECT_ID} \
    --service-account=${SA_EMAIL} \
    --source ./src-v1 \
    --entry-point send_to_slack \
    --trigger-http \
    --allow-unauthenticated \
    --region ${REGION} \
    --memory=128MiB \
    --min-instances=0 \
    --max-instances=1 \
    --set-secrets 'SLACK_BOT_TOKEN=channelbot-slack-bot-token:latest,SLACK_SIGNING_SECRET=channelbot-slack-signing-secret:latest' \
    --timeout 60s

💡We're using --allow-unauthenticated here just to test it out. It will be removed in later sections.

Test it out

Once the deployment is complete, we can view the function logs using:

gcloud beta functions logs read channelbot-send-to-slack \
	--project ${PROJECT_ID} --gen2

If everything was successful above, once of the recent log statements should say: Function has started.

Next, add the bot to the #general channel using /invite @ChannelBot in the general channel on your slack workspace.

Next, find the service endpoint using:

gcloud functions describe channelbot-send-to-slack \
    --project ${PROJECT_ID} \
    --gen2 \
    --region ${REGION} \
    --format "value(serviceConfig.uri)"

This will give a URL like https://channelbot-send-to-slack-ga6Ofi9to0-uc.a.run.app.

To trigger the channel post, just do curl ${SERVICE_URL}. This should result in a test message from ChannelBot to the #general channel.

Trigger via Google Pub/Sub

Now, instead of an unauthenticated HTTP trigger, we would like to trigger this via Google Pub/Sub. We would also like to pass the channel name and the message to post in the event.

Google Pub/Sub basics

Pub/Sub enables you to create systems of event producers and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events. Some core concepts:

• Topic. A named resource to which messages are sent by publishers.
• Subscription. A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application.
• Message. The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
• Publisher. An application that creates and sends messages to a single or multiple topics.

In this section, we will create a topic, create a subscription for our cloud function to listen to messages to that topic, and produce messages manually to that topic using gcloud cli. The message will contain the channel name and message to post, and the cloud function will post that message to the specified slack channel.

Create pub/sub topic

First, we need to create a topic.

export PUBSUB_TOPIC=channelbot-pubsub
gcloud pubsub topics create ${PUBSUB_TOPIC} \
    --project ${PROJECT_ID}

Grant permissions to the service account

Next, we need to give the roles/pubsub.editor role to the service account we're using for the function execution so that it can create a subscription to this pub/sub topic.

gcloud pubsub topics add-iam-policy-binding ${PUBSUB_TOPIC} \
    --project ${PROJECT_ID} \
    --member serviceAccount:${SA_EMAIL} \
    --role roles/pubsub.editor

Update the function code

Here's the main.py we'll need to listen to pub/sub events, extract channel and text, and sent it to slack:

import base64
import json
import functions_framework
from slack_bolt import App

# process_before_response must be True when running on FaaS
app = App(process_before_response=True)

print('Function has started')

# Triggered from a message on a Cloud Pub/Sub topic.
@functions_framework.cloud_event
def pubsub_handler(cloud_event):
    try:
        data = base64.b64decode(
            cloud_event.data["message"]["data"]).decode()
        print("Received from pub/sub: %s" % data)
        event_data = json.loads(data)
        channel = event_data["channel"]
        text = event_data["text"]
        app.client.chat_postMessage(channel=channel, text=text)
    except Exception as E:
        print("Error decoding message: %s" % E)

Before deploying, we also need to enable the Eventarc API in this project.

gcloud services enable --project ${PROJECT_ID} \
    eventarc.googleapis.com

Deploy and Test

Now, there's a slightly modified version of the deploy command to deploy this:

gcloud beta functions deploy channelbot-send-to-slack \
    --gen2 \
    --runtime python310 \
    --project ${PROJECT_ID} \
    --service-account ${SA_EMAIL} \
    --source ./src-v2 \
    --entry-point pubsub_handler \
    --trigger-topic ${PUBSUB_TOPIC} \
    --region ${REGION} \
    --memory 128MiB \
    --min-instances 0 \
    --max-instances 1 \
    --set-secrets 'SLACK_BOT_TOKEN=channelbot-slack-bot-token:latest,SLACK_SIGNING_SECRET=channelbot-slack-signing-secret:latest' \
    --timeout 60s

The main changes are:

• Changed entry-point to the new function pubsub_handler
• Replaced --trigger-http with --trigger-topic
• Removed --allow-unauthenticated

Before sending a pub/sub message, we will also need to give the roles/run.invoker role to our service account to be able to trigger our newly deployed function.

gcloud run services add-iam-policy-binding channelbot-send-to-slack \
    --project ${PROJECT_ID} \
    --region ${REGION} \
    --member=serviceAccount:${SA_EMAIL} \
    --role=roles/run.invoker

To test this out, we can send a pub/sub message using gcloud cli:

gcloud pubsub topics publish ${PUBSUB_TOPIC} \
    --project ${PROJECT_ID} \
    --message '{"channel": "#general", "text": "Hello from Cloud Pub/Sub!"}'

Post new channels roundup using cloud scheduler

Manually post recently created channels

Now that we have gained the capability to trigger a message from pub/sub to slack, we can add some logic to fetch the recently created channels from slack and post it as a message on this trigger.

Here's the modified main.py to do this:

import base64
import json
import time
import functions_framework
from slack_bolt import App

# process_before_response must be True when running on FaaS
app = App(process_before_response=True)

print('Function has started')

# Triggered from a message on a Cloud Pub/Sub topic.
@functions_framework.cloud_event
def pubsub_handler(cloud_event):
    try:
        data = base64.b64decode(
            cloud_event.data["message"]["data"]).decode()
        print("Received from pub/sub: %s" % data)
        event_data = json.loads(data)
        max_days = event_data["max_days"] # Max age of channels
        channel = event_data["channel"]
        recent_channels = get_recent_channels(app, max_days)
        if len(recent_channels) > 0:
            blocks, text = format_channels(recent_channels, max_days)
            app.client.chat_postMessage(channel=channel, text=text,
                                        blocks=blocks)
        else:
            print("No recent channels")
    except Exception as E:
        print("Error decoding message: %s" % E)


def get_recent_channels(app, max_days):
    max_age_s = max_days * 24 * 60 * 60
    result = app.client.conversations_list()
    all = result["channels"]
    now = time.time()
    return [ c for c in all if (now - c["created"] <= max_age_s) ]

def format_channels(channels, max_days):
    text = ("%s channels created in the last %s day(s):" %
            (len(channels), max_days))
    blocks = [{
        "type": "header",
        "text": {
            "type": "plain_text",
            "text": text
        }
    }]
    summary = ""
    for c in channels:
        summary += "\n*<#%s>*: %s" % (c["id"], c["purpose"]["value"])
    blocks.append({
        "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": summary
        }
    })
    return blocks, text

After deploying this with the same command above (just change --source ./src-v2 to --source ./src-v3), we can send a pub/sub event to trigger it:

gcloud pubsub topics publish ${PUBSUB_TOPIC} \
    --project ${PROJECT_ID} \
    --message '{"channel": "#general", "max_days": 7}'

And it posts a message like this:

Create schedule

Next, we want to periodically schedule this message. For this, we will configure a cron job in Google Cloud Scheduler to send a Pub/Sub event with the required parameters periodically.

Before we create a schedule, we will have to enable the Cloud Scheduler API

gcloud services enable --project ${PROJECT_ID} \
    cloudscheduler.googleapis.com

To schedule the Pub/Sub trigger at 1600 hours UTC time every day:

gcloud scheduler jobs create pubsub channelbot-job \
    --project ${PROJECT_ID} \
    --location ${REGION} \
    --schedule "0 16 * * *" \
    --time-zone "UTC" \
    --topic ${PUBSUB_TOPIC} \
    --message-body '{"channel": "#general", "max_days": 1}'

After this, a Pub/Sub event should be fired to the channelbot-pubsub topic every day, which should result in a slack message to #general with a list of channels created in the last day.

Closing Thoughts

Full code samples for this can be found in this github repo. I've also included a Makefile with targets split into sections associated with the different steps in this post.

I also plan to follow this up with a part 2 where we will use slack's slash commands to allow the end-user of this bot to setup the channel and frequency of posting of the recent channels list, and even configure multiple schedules. Please comment below if this is something you will be interested in.

One of the problems I've faced working from home for the last one year is the rigidity of the VPN software used at my work. If we were using something like OpenVPN, then I could modify the client config to setup any overrides I wanted to the network routing table or DNS, but we use FortiClient SSL VPN, which does not have such a functionality. Also, I've been using Windows 10 on my work setup for some time now because WSL works very well for me.

But first, why did I even need to modify the routing table or DNS at all?

1. I use a slightly non-standard network setup for my home, and one of my home subnets actually clashes with one of the routed work subnets (which I don't need). So, the easy solution for me is to change this routing table entry to what works for me.
2. I use diversion on my home router to do central ad-blocking, and wanted to leverage that even when connected to the work VPN.

Here is the PowerShell script that does the changes I want:

Start-Transcript -Append -Path "C:\Users\srijan\Apps\network-post-connect.log"

if( (Get-NetConnectionProfile -InterfaceAlias Wi-Fi).Name -eq "Home Wifi" ) {
    echo "Home Wifi is connected"
    $FortinetAdapter = Get-NetAdapter -InterfaceDescription "Fortinet SSL*"
    if($FortinetAdapter.Status -eq "Up") {
        echo "Work VPN is connected"
        $FortinetAdapter | Set-DnsClientServerAddress -ServerAddresses ("192.168.2.1", "8.8.8.8")
        echo "[OK] DNS server set to 192.168.2.1,8.8.8.8"
        Get-NetRoute -DestinationPrefix 192.168.2.0/24 -RouteMetric 0 | Set-NetRoute -RouteMetric 500
        echo "[OK] 192.168.2.0/24 routed locally"
    }
    else {
        echo "Work VPN is not connected. Doing nothing."
    }
}
else {
    "Home Wifi is not connected. Doing nothing."
}

Stop-Transcript

Explanation of what it does:

1. Uses Start-Transcript and Stop-Transcript to log the output to a file.
2. Checks if the system is connected to SSID "Home Wifi".
3. If so, checks if the adapter with description with the pattern "Fortinet SSL*" is up
4. If so, changes the DNS server address
5. In the routing table, it increases the route metric of 192.168.2.0/24 with metric 0 - sets it to 500. This route is added by FortiClient, which I wanted to de-prioritize.

From the Set-NetRoute docs:

The computer selects the route with the lowest combined value.

Now, we just need to setup some automation to run this whenever the VPN is connected. For this, I used Windows Task Scheduler. Whenever windows activates any network profile, a Microsoft-Windows-NetworkProfile/Operational log is generated with Event Id 10000. Windows Task Scheduler has the capability to schedule a task to be run whenever any event is triggered.

Here are some screenshots of the task configuration:

Here's the exported XML of the task:

<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2021-01-19T23:24:57.7398228</Date>
    <Author>srijan</Author>
    <Description>Currently:
1. Set local DNS
2. Route 192.168.2.0/24 locally</Description>
    <URI>\Network post connect automation</URI>
  </RegistrationInfo>
  <Triggers>
    <EventTrigger>
      <Enabled>true</Enabled>
      <Subscription>&lt;QueryList&gt;&lt;Query Id="0" Path="Microsoft-Windows-NetworkProfile/Operational"&gt;&lt;Select Path="Microsoft-Windows-NetworkProfile/Operational"&gt;*[System[Provider[@Name='Microsoft-Windows-NetworkProfile'] and EventID=10000]]&lt;/Select&gt;&lt;/Query&gt;&lt;/QueryList&gt;</Subscription>
      <Delay>PT10S</Delay>
    </EventTrigger>
  </Triggers>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>false</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>PT1H</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>powershell</Command>
      <Arguments>-File C:\Users\srijan\Apps\network-post-connect.ps1 -WindowStyle Hidden</Arguments>
    </Exec>
  </Actions>
</Task>

I have gotten used to the ease of setting up things like this for Linux, but was pleasantly surprised that it's easy enough for Windows as well. Windows Task Scheduler actually supports a lot of different conditionals for tasks as well. For example, only starting the task if the computer has been idle for some time, or only starting if connected to AC power, etc..

Let me know in the comments if you think there is an easier way, or if you have any improvement suggestions for the above.

Dialyzer is a static analysis tool for Erlang that identifies software discrepancies, such as definite type errors, code that has become dead or unreachable because of programming errors, and unnecessary tests, in single Erlang modules or entire (sets of) applications.

Dialyzer is integrated with rebar3 (a build tool for Erlang), and its default output looks like this:

This is a good starting point, but it's not very useful in some cases:

1. If you have lots of warnings, this output covers several screens, and it becomes difficult to parse through everything.
2. If you run this in some sort of continuous integration (like Jenkins), then the console output is not very friendly.

One way to improve this is to generate an HTML report which can be published/emailed/opened in the browser.

So, I build a rebar3 plugin that generates a nicely formatted color HTML report from the dialyzer output. The plugin can be found on hex.pm, or on github.

Usage

Make sure you're using rebar3 version 3.15 or later.

1. Add the plugin to your rebar.config:

{plugins, [
    %% from hex
    {rebar3_dialyzer_html, "0.2.0"}
    
    %% or, latest from git
    {rebar3_dialyzer_html, {git, "https://github.com/srijan/rebar3_dialyzer_html.git", {branch, "main"}}}
]}.

2. Select raw format for the dialyzer warnings file generated by rebar3 (this is a new flag available from rebar 3.15):

{dialyzer, [
    {output_format, raw}
]}.

3. Run the dialyzer_html rebar3 command:

$ rebar3 dialyzer_html          
===> Generating Dialyzer HTML Report
===> HTML Report written to _build/default/dialyzer_report.html

Here's how the report looks:

How I built the plugin

rebar3 dialyzer

The rebar3 built-in dialyzer plugin does the following:

1. Runs dialyzer with the options configured in rebar.config
2. Converts the output to ANSI color format and write it to console (it has a custom function for this formatting)
3. Converts the output to basic format (using built-in dialyzer:format/2) and write it to a dialyzer_warnings file.

I wanted to find out the easiest way to get a nicely formatted HTML report, ideally without forking the rebar3 project itself.

The first thing I needed was a way to save the raw (machine parse-able) dialyzer output to the warnings file instead of the default formatted output. For this, I submitted a new feature to the rebar3 project, and it introduces a new config to enable this. So, this plugin needs rebar3 version 3.15 or later.

Plugin vs Escript

Next, to actually parse and output the HTML file, I would need to run some Erlang code. There are two options I considered:

1. Escript called from Makefile/wrapper
   This option works okay, but we cannot re-use any rebar3 internal function or State. I wanted to use rebar3's own custom function for formatting the dialyzer warnings, so decided to not go with this option.
   
2. Custom rebar3 plugin
   Doing it this way makes it easy for anyone to use, and I can re-use things already implemented in rebar3 itself. So, I decided to use this option.
   

HTML output

Now, in the custom rebar3 plugin, I needed to convert the ANSI color-coded output given by rebar_dialyzer_format:format_warnings/2 into something for HTML.

I thought of the following options:

1. rebar3 uses the cf library to convert tagged strings to ANSI color codes. I can use something like dependency injection to replace the cf module with my own module so that the tagged strings are directly converted to HTML without even going to the intermediate ANSI color-coded format.
   
   This method seemed very hacky, so I decided not to pursue it. But, if rebar3 makes the dialyzer format interface configurable, I can reevaluate this approach.
   
2. Convert by writing a library in Erlang for ANSI code to HTML tags conversion.
   There is a library called ansi_to_html in elixir - but didn't want to add a huge dependency like that.
   But writing a new Erlang library to do this can be a future optimization.
   
3. Convert using a JS library after page load. I found a javascript library called ansi_up which can convert ANSI codes to HTML color tags, or it can add CSS classes that can be styled as required.
   

I opted for approach #3 because it was the easiest. I also grouped the warnings by app name so that all warnings for a single app are in one place, and the report includes the number of warnings per app.

Also, if the JS library could not be loaded (for example due to no internet, or any security headers), then it will still show the basic formatted output using dialyzer:format/2.

Future Improvements

1. I want to remove the dependency on Javascript, and want to write/use a pure Erlang library that can convert the ANSI codes to HTML.
2. Ideally, rebar3 itself can separate the dialyzer warning parsing and formatting into different functions, and make it possible to override the formatting function so that any plugin can pass its own formatting function into the dialyzer plugin.
3. This can even run git commands in the shell to figure out if any lines changed in the most recent commit involve a warning, and maybe highlight them especially in the report. This can be useful CI reports on pull requests.
4. Maybe make the format plug-able so the report can be saved in any JSON / XML or any custom format.

Let me know in the comments below, or on twitter/github if you have any suggestions for this plugin.

At work, we use the multi-app project pattern to organize our codebase. This lets us track everything in a single repository but still keep things isolated.

For isolation, we wanted to restrict apps to only be able to call the public interfaces of other apps (similar to facade pattern). However, since everything in Erlang is in a global namespace, nothing prevents code in one app to call the (exported) functions from another app.

Next best solution—detect the above scenario and raise warnings during code review/CI.

Xref to the rescue:

Xref is a cross reference tool that can be used for finding dependencies between functions, modules, applications and releases.

Xref includes some predefined analysis patterns that perform some common tasks like searching for undefined functions, deprecated function calls, unused exported functions, etc.

How it works: when xref server is started and some modules/applications/releases are added for analysis, it builds a Call Graph: a directed graph data structure containing the calls between functions, modules, applications or releases. It also creates an Inter Call Graph which holds information about indirect calls (chain of calls). It exposes a very powerful query language, which can be used to extract any information we want from the above graph data structures.

To demonstrate this, I created a sample multi-app repository: library_sample. There are some cross-app function calls in this code that we want to detect.

This repo is supposed to represent the functionality of a physical Library. It has four apps: library, library_api, library_catalog, and library_inventory. library_catalog has metadata about the books in the library, library_inventory has information about the availability of books, return dates, etc., library_api has HTTP handlers which call the above, and library is the main app which brings it all together.

Let’s say we want that library_api can call library_catalog and library_inventory functions, but catalog and inventory cannot call each other directly.

First, we clone the repo and run rebar3 shell:

$ git clone https://github.com/srijan/library_sample
Cloning into 'library_sample'...
remote: Enumerating objects: 29, done.
remote: Counting objects: 100% (29/29), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 29 (delta 3), reused 29 (delta 3), pack-reused 0
Unpacking objects: 100% (29/29), 910.62 KiB | 2.53 MiB/s, done.

$ cd library_sample

$ ./rebar3 shell
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling library_inventory
===> Compiling library_catalog
===> Compiling library
===> Compiling library_api
Erlang/OTP 23 [erts-11.1.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V11.1.7  (abort with ^G)
1>

Then, we start xref and add our build directory for analysis:

1> xref:start(s).
{ok,<0.185.0>}

2> xref:add_directory(s, "_build/default/lib", [{recurse, true}]).
{ok,[library_api,library_app,library_catalog,
     library_inventory,library_sample_app,library_sample_sup,
     library_sup]}

Using xref:q/2 for querying the constructed call graph:

3> xref:q(s, "E | library_inventory || library_catalog").
{ok,[]}

4> xref:q(s, "E | library_catalog || library_inventory").
{ok,[{{library_catalog,get_by_id,1},
      {library_inventory,get_available_copies,1}}]}

This means that there are no direct calls from the library_inventory application to the library_catalog application. But, there is a direct call from library_catalog:get_by_id/1 to library_inventory:get_available_copies/1.

The query E | library_catalog || library_inventory can be read as:

• E = All Call Graph Edges
• | = The subset of calls from any of the vertices. So | library_catalog creates a subset which contains calls from the library_catalog app.
• || = The subset of calls to any of the vertices. So, || library_inventory further creates a subset of the previous subset which contains calls to the library_inventory app.

To get both direct and indirect calls, closure E has to be used:

5> xref:q(s, "closure E | library_catalog || library_inventory").
{ok,[{{library_catalog,get_by_id,1},
      {library_inventory,get_all,0}},
     {{library_catalog,get_by_id,1},
      {library_inventory,get_available_copies,1}}]}

This tells us that there is an indirect direct call from library_catalog:get_by_id/1 to library_inventory:get_all/0.

The query language is very powerful, and there are more interesting examples in the xref user’s guide.

But this only runs the required queries manually in Erlang shell. We want to be able to run it in continuous integration. Luckily, rebar3 comes with a way to specify custom xref queries to run when running ./rebar3 xref, and to raise an error if they don’t match against the expected value defined.

Here’s the xref section from my rebar.config:

{xref_queries, [
                {"closure E | library_catalog || library_inventory", []},
                {"closure E | library_inventory || library_catalog", []}
               ]}.

This performs the two queries I want and matches them against the the target value of []. Sample output:

$ ./rebar3 xref
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling library_inventory
===> Compiling library_catalog
===> Compiling library
===> Compiling library_api
===> Running cross reference analysis...
===> Query closure E | library_catalog || library_inventory
 answer []
 did not match [{{library_catalog,get_by_id,1},{library_inventory,get_all,0}},
                {{library_catalog,get_by_id,1},
                 {library_inventory,get_available_copies,1}}]

So, now this is ready for automation.

Introduction

This post will go through my experience with setting up some advanced monitoring for PostgreSQL database using Telegraf, InfluxDB, and Grafana (also known as the TIG stack), the problems I faced, and what I ended up doing at the end.

What do I mean by advanced? I liked this Datadog article about some key metrics for PostgreSQL monitoring. Also, this PostgreSQL monitoring template for Zabbix has some good pointers. I didn’t need everything mentioned in these links, but they acted as a good reference. I also prioritized monitoring for issues which I’ve myself faced in the past.

Some key things that I planned to monitor:

• Active (and idle) connections vs. max connections configured
• Size of databases and tables
• Read query throughput and performance (sequential vs. index scans, rows fetched vs. returned, temporary data written to disk)
• Write query throughput and performance (rows inserted/updated/deleted, locks, deadlocks, dead rows)

There are a lot of resources online about setting up the data collection pipeline from Telegraf to InfluxDB, and creating dashboards on Grafana. So, I’m not going into too much detail on this part. This is what the pipeline looks like:

And here’s what my final Grafana dashboard looks like

Research on existing solutions

I found several solutions and articles online about monitoring PostgreSQL using Telegraf:

1. Telegraf PostgreSQL input plugin

Telegraf has a PostgreSQL input plugin which provides some built-in metrics from the pg_stat_database and pg_stat_bgwriter views. But this plugin cannot be configured to run any custom SQL script to gather the data that we want. And the built-in metrics are a good starting point, but not enough. So, I rejected it.

2. Telegraf postgresql_extensible input plugin

Telegraf has another PostgreSQL input plugin called postgresql_extensible. At first glance, this looks promising: it can run any custom query, and multiple queries can be defined in its configuration file.

However, there is an open issue due to which this plugin does not run the specified query against all databases, but only against the database name specified in the connection string.

One way this can still work is to specify multiple input blocks in the Telegraf config file, one for each database.

[[inputs.postgresql_extensible]]
  address = "host=localhost user=postgres dbname=database1"
  [[inputs.postgresql_extensible.query]]
    script="db_stats.sql"

[[inputs.postgresql_extensible]]
  address = "host=localhost user=postgres dbname=database2"
  [[inputs.postgresql_extensible.query]]
    script="db_stats.sql"

But, configuring this does not scale, especially if the database names are dynamic or we don’t want to hardcode them in the config.

But I really liked the configuration method of this plugin, and I think this will work very well for my use case once the associated Telegraf issue gets resolved.

3. Using a monitoring package like pgwatch2

Another method I found was to use a package like pgwatch2. This is a self-contained solution for PostgreSQL monitoring and includes dashboards as well.

Its main components are

1. A metrics collector service. This can either be run centrally and “pull” metrics from one or more PostgreSQL instances, or alongside each PostgreSQL instance (like a sidecar) and “push” metrics to a metrics storage backend.
2. Metrics storage backend. pgwatch2 supports multiple metrics storage backends like bare PostgreSQL, TimescaleDB, InfluxDB, Prometheus, and Graphite.
3. Grafana dashboards
4. A configuration layer and associated UI to configure all of the above.

I really liked this tool as well, but felt like this might be too complex for my needs. For example, it monitors a lot more than what I want to monitor, and it has some complexity to handle multiple PostgreSQL versions and multiple deployment configurations.

But I will definitely keep this in mind for a more “batteries included” approach to PostgreSQL monitoring for future projects.

My solution: custom Telegraf plugin

Telegraf supports writing an external custom plugin, and running it via the execd plugin. The execd plugin runs an external program as a long-running daemon.

This approach enabled me to build the exact features I wanted, while also keeping things simple enough to someday revert to using the Telegraf built-in plugin for PostgreSQL.

The custom plugin code can be found at this Github repo. Note that I’ve also included the line_protocol.py file from influx python sdk so that I would not have to install the whole sdk just for line protocol encoding.

What this plugin (and included configuration) does:

1. Runs as a daemon using Telegraf execd plugin.
2. When Telegraf asks for data (by sending a newline on STDIN), it runs the queries defined in the plugin’s config file (against the configured databases), converts the results into Influx line format, and sends it to Telegraf.
3. Queries can be defined to run either on a single database, or on all databases that the configured pg user has access to.

This plugin solves the issue with Telegraf’s postgresql_extensible plugin for me—I don’t need to manually define the list of databases to be able to run queries against all of them.

This is what the custom plugin configuration looks like

[postgresql_custom]
address=""

[[postgresql_custom.query]]
sqlquery="select pg_database_size(current_database()) as size_b;"
per_db=true
measurement="pg_db_size"

[[postgresql_custom.query]]
script="queries/backends.sql"
per_db=true
measurement="pg_backends"

[[postgresql_custom.query]]
script="queries/db_stats.sql"
per_db=true
measurement="pg_db_stats"

[[postgresql_custom.query]]
script="queries/table_stats.sql"
per_db=true
tagvalue="table_name,schema"
measurement="pg_table_stats"

Any queries defined with per_db=true will be run against all databases. Queries can be specified either inline, or using a separate file.

The repository for this plugin has the exact queries configured above. It also has the Grafana dashboard JSON which can be imported to get the same dashboard as above.

Future optimizations

• Monitoring related to replication is not added yet, but can be added easily
• No need to use superuser account in PostgreSQL 10+
• This does not support running different queries depending on version of the target PostgreSQL system.

Let me know in the comments below if you have any doubts or suggestions to make this better.

Jenkins is a free and open source automation server, which is used to automate software building, testing, deployment, etc.

I wanted to have a quick and easy way to run Jenkins inside docker, but also use docker containers to run jobs on the dockerized Jenkins. Using docker for jobs makes it easy to encode job runtime dependencies in the source code repo itself.

The official document on running Jenkins in docker is pretty comprehensive. But, I wanted a version using docker-compose (on Linux).

So, I started with a basic compose file:

version: '3.7'
services:
  jenkins:
  	image: jenkins/jenkins:alpine
    ports:
      - 8081:8080
    container_name: jenkins
    volumes:
      - ./home:/var/jenkins_home

When using this ( docker-compose up -d ), things came up properly, but Jenkins did not have access to the docker daemon running on the host. Also, the docker cli binary is not present inside the container.

The way to achieve this was to mount the docker socket and cli binary to inside the container so that it can be accessed. So, we come to the following compose file:

version: '3.7'
services:
  jenkins:
    image: jenkins/jenkins:alpine
    ports:
      - 8081:8080
    container_name: jenkins
    volumes:
      - ./home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/local/bin/docker

But, when trying to run docker ps inside the container with the above compose file, I was still getting the error: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock. This is because the Jenkins container is running with the jenkins user, which does not have access to use that socket.

From my research, the commonly recommended ways to solve this problem were:

• Run the container as root user
• chmod the socket file to 777
• Install sudo inside the container and give the jenkins user access to sudo without needing to enter password.

A more secure way is to create the docker group inside the container, and add the jenkins user to that group. But, this requires us to build a custom image.

Also, the group id of the docker group inside and outside the container have to be the same, so I had to add an extra check which deletes any existing group inside the container which uses the same group id, then creates the new docker group with the passed group id, and then adds the jenkins user to the docker group.

So, the final Dockerfile is:

FROM jenkins/jenkins:alpine
ARG docker_group_id=999

USER root
RUN old_group=$(getent group $docker_group_id | cut -d: -f1) && \
    ([ -z "$old_group" ] || delgroup "$old_group") && \
    addgroup -g $docker_group_id docker && \
    addgroup jenkins docker

USER jenkins

And the final docker-compose.yml file is:

version: '3.7'
services:
  jenkins:
    build:
      context: .
      args:
        docker_group_id: 999
    ports:
      - 8081:8080
    container_name: jenkins
    volumes:
      - ./home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/local/bin/docker

The docker_group_id argument can be edited in the compose file. Command to get the group id of docker:

$ getent group docker | cut -d: -f3

With the above, everything works:

$ docker-compose up -d
Creating network "jenkins_test_default" with the default driver
Building jenkins
Step 1/6 : FROM jenkins/jenkins:alpine
alpine: Pulling from jenkins/jenkins
801bfaa63ef2: Pull complete
2b72e22c6786: Pull complete
8d16efe80b55: Pull complete
682cd8857a9a: Pull complete
29c6010e8988: Pull complete
fa466f5d199d: Pull complete
e047245de0ff: Pull complete
0cfb53380af7: Pull complete
c29612b1a095: Pull complete
cd7d4bd47719: Pull complete
21cd3d960a1f: Pull complete
f3962370d584: Pull complete
bd6f35a1ea17: Pull complete
bd0c271b250f: Pull complete
Digest: sha256:1c3d9a1ed55911f9b165dd122118bff5da57520effb180d36b5c19d2a0cfe645
Status: Downloaded newer image for jenkins/jenkins:alpine
 ---> e14be04b79e8
Step 2/6 : ARG docker_group_id=999
 ---> Running in f1922fa97177
Removing intermediate container f1922fa97177
 ---> 79460069fb98
Step 3/6 : RUN echo "Assuming docker group id: $docker_group_id"
 ---> Running in 11809f4ae767
Assuming docker group id: 999
Removing intermediate container 11809f4ae767
 ---> e89b345f6c74
Step 4/6 : USER root
 ---> Running in b2e311372bc9
Removing intermediate container b2e311372bc9
 ---> 9d4d8c3ad5b2
Step 5/6 : RUN old_group=$(getent group $docker_group_id | cut -d: -f1) &&     ([ -z "$old_group" ] || delgroup "$old_group") &&     addgroup -g $docker_group_id docker &&     addgroup jenkins docker
 ---> Running in 357046a8ac49
Removing intermediate container 357046a8ac49
 ---> 865b942324eb
Step 6/6 : USER jenkins
 ---> Running in dbc2976f62c0
Removing intermediate container dbc2976f62c0
 ---> c7e6fac0187c

Successfully built c7e6fac0187c
Successfully tagged jenkins_test_jenkins:latest
WARNING: Image for service jenkins was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating jenkins ... done

$ docker-compose exec jenkins docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                               NAMES
6c05ee1315e4   jenkins_test_jenkins   "/sbin/tini -- /usr/…"   47 seconds ago   Up 47 seconds   50000/tcp, 0.0.0.0:8081->8080/tcp   jenkins

Next Steps

Here is an excellent guide on how to setup Jenkins configuration as code. This will make this setup even better because nothing will need to be configured inside Jenkins manually - it will all be driven by code / files.

For a recent project, I wanted to add a custom tag to data coming in from a built-in input plugin for telegraf.

The input plugin was the procstat plugin, and the custom data was information from pacemaker (a clustering solution for linux). I wanted to add a tag indicating if the current host was the "active" host in my active/passive setup.

For this, the best solution I came up with was to use a recently released execd processor plugin for telegraf.

How it works

The execd processor plugin runs an external program as a separate process and pipes metrics in to the process's STDIN and reads processed metrics from its STDOUT.

Telegraf's filtering parameters can be used to select or limit data from which input plugins will go to this processor.

The external program

The external program I wrote does the following:

1. Get pacemaker status and cache it for 10 seconds
2. Read a line from stdin
3. Append the required information as a tag in the data
4. Write it to stdout

The caching is just an optimization - it was more to do with decreasing polluting the logs than actual speed improvements.

Also, I've done the Influxdb lineprotocol parsing in my code directly (because my usecase is simple), but this can be substituted by an actual library meant for handling lineprotocol.

#!/usr/bin/python

from __future__ import print_function
from sys import stderr
import fileinput
import subprocess
import time

cache_value = None
cache_time = 0
resource_name = "VIP"

def get_crm_status():
    global cache_value, cache_time, resource_name
    ctime = time.time()
    if ctime - cache_time > 10:
        # print("Cache busted", file=stderr)
        try:
            crm_node = subprocess.check_output(["sudo", "/usr/sbin/crm_node", "-n"]).rstrip()
            crm_resource = subprocess.check_output(["sudo", "/usr/sbin/crm_resource", "-r", resource_name, "-W"]).rstrip()
            active_node = crm_resource.split(" ")[-1]
            if active_node == crm_node:
                cache_value = "active"
            else:
                cache_value = "inactive"
        except (OSError, IOError) as e:
            print("Exception: %s" % e, file=stderr)
            # Don't report active/inactive if crm commands are not found
            cache_value = None
        except Exception as e:
            print("Exception: %s" % e, file=stderr)
            # Report as inactive in other cases by default
            cache_value = "inactive"
        cache_time = ctime
    return cache_value

def lineprotocol_add_tag(line, key, value):
    first_comma = line.find(",")
    first_space = line.find(" ")
    if first_comma >= 0 and first_comma <= first_space:
        split_str = ","
    else:
        split_str = " "
    parts = line.split(split_str)
    first, rest = parts[0], parts[1:]
    first_new = first + "," + key + "=" + value
    return split_str.join([first_new] + rest)

for line in fileinput.input():
    line = line.rstrip()
    crm_status = get_crm_status()
    if crm_status:
        try:
            new_line = lineprotocol_add_tag(line, "crm_status", crm_status)
        except Exception as e:
            print("Exception: %s, Input: %s" % (e, line), file=stderr)
            new_line = line
    else:
        new_line = line

    print(new_line)

Telegraf configuration

Here's a sample telegraf configuration that routes data from "system" plugin to execd processor plugin, and finally outputs to influxdb.

[agent]
  interval = "30s"

[[inputs.cpu]]

[[inputs.system]]

[[processors.execd]]
  command = ["/usr/bin/python", "/etc/telegraf/scripts/pacemaker_status.py"]
  namepass = ["system"]

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "telegraf"

Other types of dynamic tags

In this example, we wanted to get the value of the tag from an external program. If the tag can be calculated from the incoming data itself, then things are much simpler. There are a lot of processor plugins, and many things can be achieved using just those.

Updated for 2023: I've updated this post with the following changes:

1. Added a top-level sample playbook
2. Used ansible apt_module's cache_time parameter to prevent repeated apt-get updates
3. Install docker-compose-plugin using apt (provides docker compose v2)
4. Make installing docker compose v1 optional
5. Various fixes as suggested in comments
6. Tested against Debian 10,11,12 and Ubuntu 18.04 (bionic), 20.04 (focal), 22.04 (jammy) using Vagrant.

I've published a new post on how I've done this testing.

I wanted a simple, but optimal (and fast) way to install docker and docker-compose using Ansible. I found a few ways online, but I was not satisfied.

My requirements were:

• Support Debian and Ubuntu
• Install docker and docker compose v2 using apt repositories
• Prevent unnecessary apt-get update if it has been run recently (to make it fast)
• Optionally install docker compose v1 by downloading from github releases
  • But, don’t download if current version >= the minimum version required

I feel trying to achieve these requirements gave me a very good idea of how powerful ansible can be.

The final role and vars files can be seen in this gist. But, I’ll go through each section below to explain what makes this better / faster.

File structure

playbook.yml
roles/
├── docker/
│    ├── defaults/
│    │   ├── main.yml
│    ├── tasks/
│    │   ├── main.yml
│    │   ├── docker_setup.yml

Playbook

This is the top-level playbook. Any default vars mentioned below can be overridden here.

---
- hosts: all
  vars:
    - docker_compose_install_v1: true
    - docker_compose_version_v1: "1.29.2"
  tasks:
    - name: Docker setup
      block:
        - import_role: name=docker

Variables

First, we’ve defined some variables in defaults/main.yml. These will control which release channel of docker will be used and whether to install docker compose v1.

---
docker_apt_release_channel: stable
docker_apt_arch: amd64
docker_apt_repository: "deb [arch={{ docker_apt_arch }}] https://download.docker.com/linux/{{ ansible_distribution | lower }} {{ ansible_distribution_release }} {{ docker_apt_release_channel }}"
docker_apt_gpg_key: https://download.docker.com/linux/{{ ansible_distribution | lower }}/gpg
docker_compose_install_v1: false
docker_compose_version_v1: "1.29.2"

Role main.yml

The tasks/main.yml file imports tasks from tasks/docker_setup.yml and turns on become for the whole task.

---
- import_tasks: docker_setup.yml
  become: true

Docker Setup

This task is divided into the following sections:

Install dependencies

- name: Install packages using apt
  apt:
    name: 
        - apt-transport-https
        - ca-certificates
        - curl
        - gnupg2
        - software-properties-common
    state: present
    cache_valid_time: 86400

Here the state: present makes sure that these packages are only installed if not already installed. I've set cache_valid_time to 1 day so that apt-get update is not run if it has already run recently.

Add docker repository

- name: Add Docker GPG apt Key
  apt_key:
    url: "{{ docker_apt_gpg_key }}"
    state: present

- name: Add Docker Repository
  apt_repository:
    repo: "{{ docker_apt_repository }}"
    state: present
    update_cache: true

Here, the state: present and update_cache: true make sure that the cache is only updated if this state was changed. So, apt-get update is not run if the docker repo is already present.

Install and enable docker and docker compose v2

- name: Install docker-ce
  apt:
    name: docker-ce
    state: present
    cache_valid_time: 86400

- name: Run and enable docker
  service:
    name: docker
    state: started
    enabled: true

- name: Install docker compose
  apt:
    name: docker-compose-plugin
    state: present
    cache_valid_time: 86400

Again, due to state: present and cache_valid_time: 86400, there are no extra cache fetches if docker and docker-compose-plugin are already installed.

Docker Compose V1 Setup

WARNING: docker-compose v1 is end-of-life, please keep that in mind and only install/use it if absolutely required.

This task is wrapped in an ansible block that checks if docker_compose_install_v1 is true.

- name: Install docker-compose v1
  when:
    - docker_compose_install_v1 is defined
    - docker_compose_install_v1
  block:

Inside the block, there are two sections:

Check if docker-compose is installed and it’s version

- name: Check current docker-compose version
  command: docker-compose --version
  register: docker_compose_vsn
  changed_when: false
  failed_when: false
  check_mode: no

- set_fact:
    docker_compose_current_version: "{{ docker_compose_vsn.stdout | regex_search('(\\d+(\\.\\d+)+)') }}"
  when:
    - docker_compose_vsn.stdout is defined

The first block saves the output of docker-compose --version into a variable docker_compose_vsn. The failed_when: false ensures that this does not call a failure even if the command fails to execute. (See error handling in ansible).

Sample output when docker-compose is installed: docker-compose version 1.26.0, build d4451659

The second block parses this output and extracts the version number using a regex (see ansible filters). There is a when condition which causes the second block to skip execution if the first block failed (See playbook conditionals).

Install or upgrade docker-compose if required

- name: Install or upgrade docker-compose
  get_url: 
    url : "https://github.com/docker/compose/releases/download/{{ docker_compose_version }}/docker-compose-Linux-x86_64"
    dest: /usr/local/bin/docker-compose
    mode: 'a+x'
    force: yes
  when: >
    docker_compose_current_version == ""
    or docker_compose_current_version is version(docker_compose_version, '<')

This just downloads the required docker-compose binary and saves it to /usr/local/bin/docker-compose, but it has a conditional that this will only be done if either docker-compose is not already installed, or if the installed version is less than the required version. To do version comparison, it uses ansible’s built-in version comparison function.

So, we used a few ansible features to achieve what we wanted. I’m sure there are a lot of other things we can do to make this even better and more fool-proof. Maybe a post for another day.

There are many different ways to use replication in PostgreSQL, whether for high
availability (using a failover), or load balancing (for scaling), or just for
keeping a backup. Among the various tools I found online, I though bucardo is
the best for my use case - keeping a live backup of a few important tables.

I've assumed the following databases:

• Primary: Hostname = host_a, Database = btest
• Backup: Hostname = host_b, Database = btest

We will install bucardo in the primary database (it required it's own database
to keep track of things).

1. Install postgresql

    sudo apt-get install postgresql-9.4

2. Install dependencies on host_a

    sudo apt-get install libdbix-safe-perl libdbd-pg-perl libboolean-perl build-essential postgresql-plperl-9.4

3. On host_a, Download and extract bucardo source

    wget https://github.com/bucardo/bucardo/archive/5.4.0.tar.gz
    tar xvfz 5.4.0.tar.gz

4. On host_a, Build and Install

    perl Makefile.PL
    make
    sudo make install
    sudo mkdir /var/run/bucardo
    sudo mkdir /var/log/bucardo

5. Create bucardo user on all hosts

    CREATE USER bucardo SUPERUSER PASSWORD 'random_password';
    CREATE DATABASE bucardo;
    GRANT ALL ON DATABASE bucardo TO bucardo;

   Note: All commands from now on are to be run on host_a only.

6. On host_a, set a password for the postgres user:

    ALTER USER postgres PASSWORD 'random_password';

7. On host_a, add this to the installation user's ~/.pgpass file:

    host_a:5432:*:postgres:random_password
    host_a:5432:*:bucardo:random_password

   Also add entries for the other hosts for which users were created in step 5.

   Note: It is also a good idea to chmod the ~/.pgpass file to 0600.

8. Run the bucardo install command:

    bucardo -h host_a install

9. Copy schema from A to B:

    psql -h host_b -U bucardo template1 -c "drop database if exists btest;"
    psql -h host_b -U bucardo template1 -c "create database btest;"
    pg_dump -U bucardo --schema-only -h host_a btest | psql -U bucardo -h host_b btest

10. Add databases to bucardo config

     bucardo -h host_a -U bucardo add db main db=btest user=bucardo pass=host_a_pass host=host_a
     bucardo -h host_a -U bucardo add db bak1 db=btest user=bucardo pass=host_b_pass host=host_b

    This will save database details (host, port, user, password) to bucardo
    database.

11. Add tables to be synced

    To add all tables:

     bucardo -h host_a -U bucardo add all tables db=main relgroup=btest_relgroup

    To add one table:

     bucardo -h host_a -U bucardo add table table_name db=main relgroup=btest_relgroup

    Note: Only table which have a primary key can be added here. This is a
    limitation of bucardo.

12. Add db group

     bucardo -h host_a -U bucardo add dbgroup btest_dbgroup main:source bak1:target

13. Create sync

     bucardo -h host_a -U bucardo add sync btest_sync dbgroup=btest_dbgroup relgroup=btest_relgroup conflict_strategy=bucardo_source onetimecopy=2 autokick=0

14. Start the bucardo service

     sudo bucardo -h host_a -U bucardo -P random_password start

    Note that this command requires passing the password because it uses sudo,
    and root user's .pgpass file does not have the credentials saved for bucardo
    user.

15. Run sync once

     bucardo -h host_a -U bucardo kick btest_sync 0

16. Set auto-kick on any changes

     bucardo -h host_a -U bucardo update sync btest_sync autokick=1
     bucardo -h host_a -U bucardo reload config

That's it. Now, the tables specified in step 11 will be replicated from host_a
to host_b.

I also plan to write about other alternatives I've tried soon.