Debian Mirror on Libre Computer ROC-RK3328-CC

ROC-RK3328-CC

In the age of 100Mbps+ fiber internet connections it's hard to imagine why you'd need to mirror your OS packages locally. There are a couple of reasons, most of them go to being a good neighbor to the larger package mirrors and in my opinion it never hurts to have a locally acessible backup of the entire package tree in case things get weird and you lose internet connectivity for a while. Mostly I do it because between physical hosts and virtual machines I have quite a few Debian boxes floating around on different releases. Pointing them all at a local machine and just having the mirror grab the files saves the host on the other end a lot of bandwidth.

Small single board computers are great for this in theory. They take up little space and power but until recently that has come at the cost of performance and storage. I've never been terribly impressed with the Raspberry Pi and it's limited connectivity has made it less than ideal for this sort of task. Enter the Libre Computer ROC-RK3328-CC board. It shares the same form factor as the RPi B type boards (so it works with those cases) but has more RAM, a faster CPU, actual gigabit Ethernet, eMMC slot and a USB 3.0 port. It also uses less power than the RPi3B+. I went with the 4GB edition because why not but this could easily be done on the 2GB or 1GB version of the board if you want to save a little money. The killer features for this task is the gigabit and USB 3.0 port, you don't need 4GB of RAM to run a couple of cron scripts and nginx.

First you'll need an OS for you SBC. I'm currently running a self made Armbian but Libre Computer has Debian-based images as well. My Renegade board off a 32GB eMMC but it can be a bit tricky to flash one of those. The MicroSD slot should provide good enough performance to run the OS if you want to go that route for simplicity's sake. For my mirror I'm grabbing Jessie (oldstable), Stretch (stable), Buster (testing) and Sid (unstable) in AMD64 and i386 flavors using debmirror. This takes up around 600GB as of November 2018 so a 1TB hard drive or SSD should do the job. Personally I just went with a dual bay 3.5" USB 3 enclosure and a couple of 2TB drives with BTRFS RAID1.

The Debian distribution that is packaged for the ROC-RK3328-CC board has an empty fstab by default. After plugging in your USB drive, formatting it (if needed) and finding the UUID (ls -lah /dev/disks/by-UUID) add an entry like so:

UUID=your UUID here /mnt/usb   btrfs    nofail  0       2

The nofail option allows the board to continue booting if the drive is not present for some reason. Btrfs is optional, ext4 or xfs will work fine.

Best practices dictate setting up a separate user for the mirror:

groupadd mirror 
useradd -d /var/mirror -g mirror
mkdir /mnt/usb/mirror
chown -R mirror:mirror /mnt/usb/mirror
ln -s /mnt/usb/mirror /var/mirror 

Next add some packages:

apt-get install nginx ed screen xz-utils debmirror debian-keyring

Debmirror needs access to the GPG keys to verify the source of the packages, so you'll need to import them into the mirror account:

su - mirror
gpg --no-default-keyring --keyring trustedkeys.gpg --import /usr/share/keyrings/debian-archive-keyring.gpg

Every once in a while the keys will need updating (particularly when a new version of Debian is released), updating the keys is the same as the initial installation:

gpg --no-default-keyring --keyring trustedkeys.gpg --import /usr/share/keyrings/debian-archive-keyring.gpg 

Next up we need to script out the mirror update process. After some tinkering and searching mine ended up looking like this:

#!/bin/sh
FTP=ftp.us.debian.org
DEST=/mnt/usb/mirror/debian
VERSIONS=jessie,stretch,testing,sid
ARCH=amd64,i386
debmirror ${DEST} --host=${FTP} --root=/debian --dist=${VERSIONS} -section=main,contrib,non-free,main/debian-installer --i18 --arch=${ARCH} --passive --cleanup $VERBOSE

and I saved it to /usr/local/bin/mirror.sh

It's now time to do the initial sync, this can take a while so run it in screen session:

screen su mirror -c "/usr/local/bin/mirror.sh"

After that's finished (probably several hours later), we'll make the mirror accessible via nginx. In my case I had to uncomment the following line in /etc/nginx/nginx.conf:

server_names_hash_bucket_size  64;

as for some reason it was having a cow over the length of my hostname.

I created a vhost file at /etc/nginx/sites-available/000-yourhost.yourdomain.com:

server {
    listen       80;

    server_name wren.buttonhost.net www.wren.buttonhost.net;

    access_log /var/log/nginx/wren.buttonhost.net-access.log;
    error_log /var/log/nginx/wren.buttonhost.net-error.log;

        location / {
                root /var/mirror/;
                autoindex on;
                }
        }

I have the DNS assigned on my DNS server/gateway but you'll need to figure out how to deal with that. Just make sure yourhost.yourdomain.com points at this machine's IP.

Then link this file in the /etc/nginx/sites-enabled directory:

cd /etc/nginx/sites-enabled
ln -s ../sites/available/000-yourhost.yourdomain.com 000-yourhost.yourdomain.com.cfg

My machine is called wren.buttonhost.net, please change the name in the configuration file to your own machine's vhost name!

and restart nginx:

systemctl nginx restart

Afterwards you should be able to navigate to that host in a web browser and see your mirror.

Debian Mirror in Firefox

The last step is to automate the update process via cron, here in /etc/cron.d/debmirror:

# sync Debian mirrors three times a week
30 5 * * 1,3,5 mirror /usr/local/bin/mirror.sh

You now have a Debian mirror that you can fit in your pocket and run off a lithium battery pack if needed. Why would you need that? I don't know, why wouldn't you? Outside of the parts about the fstab, what packages to install and symlinking the /var/mirror directory to the USB drive the rest of this should work on any Debian machine regardless of architecture.