Synology DSM and Veeam 11

For a long time, Veeam has been telling its users to not use "low-end NAS boxes" (eg, Synology, QNAP, Thecus) as backup repositories for Backup & Replication (VBR), even though these Linux-based devices should be compatible if they have "x86" architecture (as opposed to ARM).

The reality is that none of these devices use "bog standard" Linux distributions, and due to their appliance-based nature, have some significant limitations on what can be done to their custom distributions.

However, there are many folks—both as home users or within small/budget-limited businesses—who are willing to "take their lumps" and give these things a shot as repositories.

I am one of them, particularly for my home "lab" environment. I've written about this use case (in particular, the headaches) a couple of times in this blog [1, 2], and this post joins them, addressing yet another fix/workaround that I've had to implement.

Background

I use a couple of different Synology boxes for backup purposes, but the one I'm dealing with today is the DS1817+. It has a 10GbE interface for connectivity to my network, a quad-core processor (the Intel Atom C2538) and 8GB RAM (upgradable to 16GB, but I haven't seen the demand that would require it). It is populated with 8x1TB SATA SSDs for ~6TB of backup capacity.

I upgraded DSM to 7.0 a while back, and had to make some adjustments to the NFS target service to continue to support ESXi datastores via NFS 4.1

Yesterday, I updated it to 7.1-42661 Update 2, and was greeted to a number of failed backup jobs this morning.

Symptoms

All the failed jobs have uniform symptoms: Timeout to start agent

With further investigation, I saw that my DS1817+ managed server was "not available", and when attempting to get VBR to re-establish control, kept getting the same error with the installation of transport services:

Installing Veeam Data Mover service Error: Failed to invoke command /opt/veeam/transport/veeamtransport --install 6162:  /opt/veeam/transport/veeamtransport: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory

Failed to invoke command /opt/veeam/transport/veeamtransport --install 6162:  opt/veeam/transport/veeamtransport: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory

Workaround

After failing to find a fix after some Linux-related searches, I discovered a thread on the Veeam Community Forum that addressed this exact issue [3]. 

This is apparently a known issue with VBR11 and Synology boxes, and as Veeam is moving further and further away from the "on the fly" deployment of the transport agent to a permanently-installed "Data Mover" daemon (which is necessary to provide the Immutable Backup feature), it becomes a bigger issue. Veeam has no control over the distribution—and would just as soon have clients use other architectures—and Synology would probably be happy with customers considering their own backup tool over competing options...

At any rate, some smart people posted workarounds to the issue after doing their own research, and I'm re-posting for my own reference because it worked for me.

1. Download the latest ACL library from Debian source mirrors. The one I used—and the one in the Forum thread—is http://ftp.debian.org/debian/pool/main/a/acl/libacl1_2.2.53-10_amd64.deb
2. Unpack the .deb file using 7zip
3. Upload the data.tar file to your Synology box. Feel free to rename the file to retain your sanity; I did.
4. Extract the tarball to the root directory using the "-C /" argument:
   tar xvf data.tar -C /
5. If you are using a non-root account to do this work, you'll need to use "sudo" to write to the root. You will also need to adjust owner/permissions on the extracted directories & files:
   sudo tar xvf data.tar -C /
   sudo chown -R root:root /usr/lib/x86_64-linux-gnu
   sudo chmod -R 755 /usr/lib/x86_64-linux-gnu
6. Create soft links for these files in the boxes filesystem:
   sudo ln -sf /usr/lib/x86_64-linux-gnu/libacl.so.1 /usr/lib/libacl.so.1
   sudo ln -sf /usr/lib/x86_64-linux-gnu/libacl.so.1.1.2253 /usr/lib/libacl.so.1.1.2253
7. Last, get rid of any previous "debris" from failed transport installations
   sudo rm -R /opt/veeam

Once the Synology is prepped, you must go back into VBR and re-synchronize with the Linux repository:

1. Select the "Backup Infrastructure" node in the VBR console
2. Select the Linux node under Managed Servers
3. Right-click on the Synology box being updated and select "Properties..." from the popup menu.
4. Click [Next >] until the only option is [Finish]. On the way, you should see that the Synology is correctly identified as a compatible Linux box, and the new Data Mover transport service is successfully installed.

Summary

I can't guarantee that this will work after a future update of DSM, and there may come a day when other libraries are "broken" by updates to VBR or DSM. But this workaround was successful for me.

Update

The workaround has persisted through a set of updates to DSM7. I have seen this come up with DSM6, but this workaround does not work on that; too many platform incompatibilities, I suspect. Need to do some more research & experimentation for DSM6...

Update: maintaining the pi-hole HA pair

In an earlier post, I shared how I got pi-hole working in my environment, thanks to a number of posts on a reddit thread. Since then, I've been living with the setup and tweaking my configuration a bit.

This post documents one of the tweaks that might be useful for others...

If you're using the method documented by Panja0, you know that there's a script in the pi-hole distribution (gravity.sh) that must be edited in order to synchronize files between the nodes of the HA pair. Well, he reminds you in the tutorial that it'll need to be re-edited every time you update pi-hole, or the synchronization won't occur.

As you might guess, I didn't remember when I updated a week ago, and couldn't understand why my settings weren't getting synchronized. So I went back to the post, reviewed my settings, and face-palmed myself when I discovered my oversight. I had failed to re-edit gravity.sh

After I did the necessary edits, I realized that, even if I'd remembered about it, I'd still need to refer to the original post to get the right command line, etc., for the edits.

I didn't want to spend the time to figure out how to trigger a script to make the update for me upon a pi-hole update, but I sure could figure out the script to do the correct updates!

I mean... come on: what better use of automation than to use a script to a) check to see if the update has already been performed, and b) if not, perform the update?

#!/bin/bash
# make sure the pihole-gemini script is being run by gravity.sh

GEMINI='su -c /usr/local/bin/pihole-gemini - <gemini user>'
GRAVITY=/opt/pihole/gravity.sh

TRIGGER=$(sed -e '$!{h;d;}' -e x $GRAVITY)
if [ "$TRIGGER" != "$GEMINI" ]
then
        # insert the gemini commandline before the last line of the script
        sed -i "$ i$GEMINI" $GRAVITY
fi

If you decide to use the script, just make sure that you make any necessary modifications for the first two script variables to match your installation. You also need it on both nodes of your HA pair!

In my setup, I'm saving this script in the /etc/scripts directory, which I'm using for other "keepalived" scripts. I'll remember to run it next time I update pi-hole, and that's all I'll need to recall!

Putting Pi-hole to work

I've been reading about my friends' use of Pi-hole on their home networks, and I've been curious about trying it to see how well it does. I've resisted doing so, primarily because of the single point of failure a pi-hole system represents: if it's unavailable, you get no DNS.

And we all know, it's never DNS...except when it is.

An alternative, naturally, it to run a pair of systems. Why not? Raspberry Pi devices are relatively cheap, and the software is all no-charge.

For most home users, that might be fine, but I run a lab in my home that also provides services to the household, so I had more permutations to worry about: what happens if my Pi dies? what happens if my domain controllers are unavailable? Etc.

The solution I've settled on is to run a primary Pi-hole server as a VM in my lab environment—which gives me more than enough performance and responsiveness, even under the most demanding of situations—and a secondary with a Raspberry Pi, so that even if the VM environment goes "pear shaped," I still get DNS resolution.

In order to accommodate several types of outages, yet avoiding the need to both double-up the configuration work (with the potential of missing an update and having weird results to troubleshoot) while providing pre-configured support for a couple of likely failure and maintenance scenarios, I've mated the two systems together in a failover cluster by configuring the "keepalive" daemon along with some scripting to keep the two systems in sync for the blocking function, while leaving some configuration elements (upstream DNS servers for one) independent of each other.

I didn't do the "heavy lifting" on the sync and keepalive aspects; those were provided by reddit user Panja0 in this post: https://www.reddit.com/r/pihole/comments/d5056q/tutorial_v2_how_to_run_2_pihole_servers_in_ha/

I'm running ubuntu server 19.10 (Eoan Ermine... whatever) instead of Raspbian Stretch/Buster, so there have been a number of changes I've had to make to the systems to adapt:

• To get keepalived installed, I needed libipset11, not libipset3 (mentioned in the comments of the HA tutorial)
• I had to modify the rsync command arguments in the synchronization script due to changes between Debian versions that I'm running versus the original post (mentioned in the comments of the HA tutorial)
• I had to permit my rsync user to skip password re-auth by editing the sudoers file; I think this may also be a version-specific issue.
• I added an NTP client to utilize my GPS-based hardware time server; this is super important when using a Raspberry Pi without a real-time clock HAT add-on.
• The primary system uses the lab's DNS (domain controllers) for its upstream DNS servers. In addition to avoiding the need to configure additional conditional forwarding rules for dnsmasq, this gives the Pi-hole server the identity of the clients via DNS
• The secondary uses OpenDNS servers—I have a household account with several filtering options enabled already—with a dnsmasq configuration for conditional forwarding on the domain.

Given my homelab, it was pretty trivial to set this up as a VM, but what really sold it to me was getting the Raspberry Pi running in concert. I originally started with a Pi 3 Model B that I had lying around after an old project that I'd quit, but the performance difference between the two platforms was so noticeable that going with a true primary/secondary setup made the most sense. I considered upgrading to the Pi 4, but decided that my desire to avoid purchasing micro-HDMI adapters outweighed the value in the more-robust, newer model. I did decide to go ahead and upgrade from the 3 to the 3+, however, when I discovered that my local MicroCenter had them for $34US; I also paired the new unit with a passive heatsink case, which has allowed the Pi to run significantly cooler (30°F) than the original setup, which utilized aluminium heatsinks and a non-vented plastic case.

Aside from this "vanilla" setup, I also took note of the additional block lists that my friend Tim Smith wrote about in a blog post. I need to let this "bake" for a while before considering it finished, but I'm liking what I'm seeing so far.