Recovery was pretty simple:

1. Asked the service provider to replace the disk
2. Find new disk ID etc using:

   lsblk -o NAME,SIZE,MODEL,SERIAL,LABEL,FSTYPE
   ls -ltrh /dev/disk/by-id/ata-*

3. Resilver using:

   sudo zpool replace lake <old_disk_id> <new_disk_id>

4. Watch status using:

   watch zpool status -v

Re-silvering is still ongoing, but hopefully completes without issues. Will run a manual zpool scrub at the end to make sure everything is okay.

I had a shell script to do this as well. Doing it from Emacs makes more sense because I can have a completion UI.

So, here's my modified version for Linux:

(defun sj/default-browser (&optional name)
  "Set the default browser based on the given NAME."
  (interactive
   (list
    (completing-read
     "Browser: "
     (split-string
      (shell-command-to-string
       "find /usr/share/applications ~/.local/share/applications -name \"*.desktop\" -exec grep -l \"Categories=.*WebBrowser\" {} \\;")
      "\n" t))))
  (let ((browser-desktop (file-name-nondirectory name)))
    (shell-command (format "xdg-mime default %s text/html" browser-desktop))
    (shell-command (format "xdg-mime default %s application/xhtml+xml" browser-desktop))
    (shell-command (format "xdg-mime default %s application/x-extension-html" browser-desktop))
    (shell-command (format "xdg-settings set default-web-browser %s" browser-desktop))))

As a plus, it automatically lists the installed browsers based on .desktop files on your system.

Issues I faced with X11:

1. Smooth scrolling broken
2. Apps work noticeably slower
3. Screen tearing
4. This bug in Emacs GTK build: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67654 (To be fair, this is a GTK-specific issue, not X11 specific)

I will go back to Wayland as soon as Zoom fixes this: https://community.zoom.com/t5/Zoom-Meetings/share-screen-linux-wayland-broken/m-p/203624/highlight/true#M112235

Syndicated to:

• https://bsky.app/profile/srijan4.bsky.social/post/3l5ycxn4gak27

Overall, I like the scrolling tiling pane paradigm. I realized I've been manually doing something like this using workspaces with 1-2 windows per workspace with two keybindings - one to change workspace and one to switch windows inside a workspace. So, this window management model really clicks for me.

I switched from GNOME to KDE several years ago due to getting burnt by extensions breaking too frequently, but hopefully things are better now.

Syndicated to:

• https://bsky.app/profile/srijan4.bsky.social/post/3l5icw34klr2e

My laptop has started freezing sometimes, not sure why. Usually, I can just force power off using the power button and start it again, but it has happened twice that I had to recover the system by booting via a USB drive, chrooting, and recovering the damaged files using fsck or pacman magic.

The linux kernel has:

a ‘magical’ key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up.

(More details on archwiki and kernel doc)

To enable, I did:

echo "kernel.sysrq = 244" | sudo tee /etc/sysctl.d/sysreq.conf
sudo sysctl --system

However, to trigger this on my laptop, I was not able to find the right key combination for SysRq. I was able to make it work using an external keyboard that has a PrintScreen binding on a layer, by using the following:

Press Alt and keep it pressed for the whole sequence: PrintScreen - R - E - I - S - U - B

Currently, PrintScreen on my external keyboard is bound to Caps lock long press + Up arrow.

Plymouth is a package that provides a themeable graphical boot process / splash screen all the way up to the login manager. This includes a graphical password prompt as well. Here are the steps I took to set this up:

1. First, I installed plymouth-git from the AUR. ArchWiki suggests plymouth-git instead of plymouth because it is actually less likely to cause problems for most users than the stable package.

2. Next, I updated the HOOKS section in my /etc/mkinitcpio.conf to include sd-plymouth:

HOOKS=(base systemd plymouth autodetect modconf kms keyboard sd-vconsole block sd-encrypt filesystems fsck)

3. And regenerated the initramfs:

# mkinitcpio -P

4. Next, I added the following kernel parameters:

quiet splash

ArchWiki also suggests adding vt.global_cursor_default=0, but my experience was better without it. With this option, the cursor in TTY terminals becomes hidden, not just for the boot sequence but even later.

With the above changes, after reboot, a nice password prompt is shown with a spinner image. But, this hid the beautiful OEM ROG logo that comes first at boot up. So, here are further tweaks I did to make it look as I wanted.

5. First, I tried using the built-in BGRT theme. This is a variation of the spinner theme that keeps the OEM logo if available (BGRT stands for Boot Graphics Resource Table).

# plymouth-set-default-theme -R bgrt

This did not show the spinner, but it still hid the OEM logo when asking for decryption password. Although it did show the logo again after password was entered. So, I guessed it just needed a little customization.

6. So, I made a copy of the bgrt theme to make my customizations.

# cd /usr/share/plymouth/themes
# cp -r bgrt bgrt-custom
# cd bgrt-custom
# mv bgrt.plymouth bgrt-custom.plymouth

7. These are the changes I had to make in bgrt-custom.plymouth to make it show the prompt like I wanted:

diff --git a/../bgrt/bgrt.plymouth b/bgrt-custom.plymouth
index e8e9713..ca7a293 100644
--- a/../bgrt/bgrt.plymouth
+++ b/bgrt-custom.plymouth
@@ -30,8 +30,8 @@ Name[he]=BGRT
 Name[fa]=BGRT
 Name[fi]=BGRT
 Name[ie]=BGRT
-Name=BGRT
-Description=Jimmac's spinner theme using the ACPI BGRT graphics as background
+Name=BGRT-Custom
+Description=Customized Jimmac's spinner theme using the ACPI BGRT graphics as background
 ModuleName=two-step

 [two-step]
@@ -39,9 +39,9 @@ Font=Cantarell 12
 TitleFont=Cantarell Light 30
 ImageDir=/usr/share/plymouth/themes//spinner
 DialogHorizontalAlignment=.5
-DialogVerticalAlignment=.382
+DialogVerticalAlignment=.75
 TitleHorizontalAlignment=.5
-TitleVerticalAlignment=.382
+TitleVerticalAlignment=.75
 HorizontalAlignment=.5
 VerticalAlignment=.7
 WatermarkHorizontalAlignment=.5
@@ -52,7 +52,7 @@ BackgroundStartColor=0x000000
 BackgroundEndColor=0x000000
 ProgressBarBackgroundColor=0x606060
 ProgressBarForegroundColor=0xffffff
-DialogClearsFirmwareBackground=true
+DialogClearsFirmwareBackground=false
 MessageBelowAnimation=true

 [boot-up]

Basically, I tweaked DialogClearsFirmwareBackground, DialogVerticalAlignment, and TitleVerticalAlignment to my liking. To set this custom theme, I ran:

# plymouth-set-default-theme -R bgrt-custom

8. This looked perfect. But, I noticed that this increased by boot up time considerably. Plymouth was taking a long time before displaying the password prompt. On further digging, I found a parameter called DeviceTimeout in /etc/plymouth/plymouthd.conf with default value of 8 seconds.

According to this merge request, this was needed to keep support for certain AMD GPUs. I don't have and AMD GPU, and anyway I think Plymouth is using the EFI framebuffer for this splash screen, not the GPU. So, I reduced it to 2 seconds to make things faster.

For photos or videos taken from my phone, I use google photos for backup - which works pretty well. Even if I delete the original files from my phone, the photos app still keeps them online.

Coming back to my laptop, I'm currently using duplicity (with the duply wrapper) to backup to multiple destinations. Why multiple locations? I wanted one local copy so that I can restore fast, and at least one at a remote location so that I can still restore if the local disk fails.

For off-site, I'm using the fantastic rsync.net service. For local backups, I'm using two destinations: a USB HDD at my home, and a NFS server at my work. Depending on where I am, the backup will be done to the correct destination.

This post will deal with the backups to my local USB disk.

Here's what I've been able to achieve: the backups will run every hour as long as the USB disk is connected. If it is not connected, the backup script will not even be triggered. I did not want to see backup failures in my logs if the HDD is not connected.

I've done this using a systemd timer and service. I've defined these units in the user-level part for systemd so that root privileges are not required.

Mounting the USB Disk

To automatically mount the USB disk, I've added the following line to my /etc/fstab:

UUID=27DFA4B43C8C0635 /mnt/Ext01 ntfs-3g nosuid,nodev,nofail,auto,x-gvfs-show,permissions 0 0

Duply config for running the backup

Here's my duply config file (kept at ~/.duply/ext01/conf) (mostly self-explanatory):

TARGET='file:///mnt/Ext01/Backups/'
SOURCE='/home/srijan'
MAX_AGE=1Y
MAX_FULL_BACKUPS=15
MAX_FULLS_WITH_INCRS=2
MAX_FULLBKP_AGE=1M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
VOLSIZE=4
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
DUPL_PARAMS="$DUPL_PARAMS --exclude-other-filesystems "

This can be run manually using:

$ duply ext01 backup

Exclusions can be specified in the file ~/.config/ext01/exclude in a glob-like format.

Systemd Service for running the backup

Next, here's the service file (kept at ~/.config/systemd/user/duply_ext01.service):

[Unit]
Description=Run backup using duply: ext01 profile
Requires=mnt-Ext01.mount
After=mnt-Ext01.mount

[Service]
Type=oneshot
ExecStart=/usr/bin/duply ext01 backup

The Requires option says that this unit has a dependency on the mounting of Ext01. The After option specifies the order in which these two should be started (run this service after mounting).

After this step, the service can be run manually (via systemd) using:

$ systemctl --user start duply_ext01.service

Systemd timer for triggering the backup service

Next step is triggering it automatically every hour. Here's the timer file (kept at ~/.config/systemd/user/duply_ext01.timer):

[Unit]
Description=Run backup using duply ext01 profile every hour
BindsTo=mnt-Ext01.mount
After=mnt-Ext01.mount

[Timer]
OnCalendar=hourly
AccuracySec=10m
Persistent=true

[Install]
WantedBy=mnt-Ext01.mount

Here, the BindsTo option defines a dependency similar to the Requires option above, but also declares that this unit is stopped when the mount point goes away due to any reason. This is because I don't want the trigger to fire if the HDD is not connected.

The Persistent=true option ensures that when the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. This is because I want to catch up on missed runs of the service when the disk was disconnected.

After creating this file, I ran the following to actually link this timer to mount / unmount events for the Ext01 disk:

$ systemctl --user enable duply_ext01.timer

That's it. Now, whenever I connect the USB disk to my laptop, the timer is started. This timer triggers the backup service to run every hour. Also, it takes care that if some run was missed when the disk was disconnected, then it would be triggered as soon as the disk is connected without waiting for the next hour mark. Pretty cool!

NOTES:

• Changing any systemd unit file requires a systemd --user daemon-reload before systemd can recognize the changes.
• The systemd documentation was very helpful.

Coming Soon

Although it would be similar, but I'll also document how to do the above with NFS or SSHFS filesystems (instead of local disks). The major difference would be handling loss of internet connectivity, timeouts, etc.

There are a few different ways in which this could be accomplished.

Mounting the root filesystem with read-only flags

Most parts of the linux root filesystem can be mounted read-only without much problems, but there are some parts which don't play well. This debian wiki page has some information about this approach. I thought this approach would not be very stable, so did not try it out completely.

Using aufs/overlayfs

aufs is a union file system for linux systems, which enables us to mount separate filesystems as layers to form a single merged filesystem. Using aufs, we can mount the root file system as read-only, create a writable tmpfs ramdisk, and combine these so that the system thinks that the root filesystem is writable, but changes are not actually saved, and don't survive a reboot.

I found this method to be most suitable and stable for my task, and have been using this for the last 6 months. This system mounts the real filesytem at mountpoint /ro with read-only flag, creates a writable ramdisk at mountpoint /rw, and makes a union filesystem using these two at mountpoint /.

The steps I followed for my implementation are detailed below. These are just a modified version of the steps in this ubuntu wiki page. I am using Debian in my implementation.

1. Install debian using live cd or your preferred method.

2. After first boot, upgrade and configure the system as needed.

3. Install aufs-tools.

4. Add aufs to initramfs and setup this script to start at init.

# echo aufs >> /etc/initramfs-tools/modules
# wget https://cdn.rawgit.com/srijan/383a8d7af6860de6f9de/raw/ -O /etc/initramfs-tools/scripts/init-bottom/__rootaufs
# chmod 0755 /etc/initramfs-tools/scripts/init-bottom/__rootaufs

1. Remake the initramfs.

# update-initramfs -u

1. Edit grub settings in /etc/default/grub and add aufs=tmpfs to GRUB_CMDLINE_LINUX_DEFAULT, and regenerate grub.

# update-grub

1. Reboot

Making changes

To change something trivial (like a file edit), just remount the /ro mountpoint as read-write, edit the file, and reboot.

# mount -o remount,rw /ro

To do something more complicated (like install os packages), press e in grub menu during bootup, remove aufs=tmpfs from the kernel line, and boot using F10. The system will boot up normally once.

Another method could be to use a configuration management tool (puppet, chef, ansible, etc.) to make the required changes whenever the system comes online. The changes would be lost on reboot, but it would become much easier to manage multiple such systems.

Also, if some part of the system is required to be writable (like /var/log), that directory could be mounted separately as a read-write mountpoint.