A Tour of My Overcomplicated Fileserver

A friend of mine is in the process of building their first NAS. It’s a beautiful thing, really. In heated discussions about system requirements and configurable options, I recalled my foray into my first (and current) hosted file server from many years ago. I made some unique design decisions that I think are worth sharing, if for nothing more than your amusement.

My goals for this fileserver were simple (at least I thought so). I wanted to be able to manage all files hosted on the NAS, administer access for users, and allow those users limited access to the file server, while still being able to upload and download files, according to their group membership. Additionally, I wanted the process to be “user-friendly” - to some degree, at least.

Well, SFTP exists. It’d be great if my users could use keyfiles, but I will sadly acknowledge that as not “user friendly” for most people. I don’t like the idea of opening up password auth for all users, so I can specify PasswordAuthentication specifically for the sftp_users group, which doesn’t even need to have SSH access. Since these users are only using SFTP, we can disable a bunch of forwarding/TTY related options. The users shouldn’t be able to access anything outside of the NAS, so we can set ChrootDirectory. Here’s what we have so far in /etc/ssh/sshd_config:

...

Subsystem       sftp    internal-sftp

# SFTP accounts for nas
Match Group sftp_user
    ChrootDirectory /mnt/nas
    ForceCommand internal-sftp -d /
    X11Forwarding no
    AllowAgentForwarding no
    AllowTcpForwarding no
    PermitTTY no
    PermitTunnel no
    GatewayPorts no
    PasswordAuthentication yes

I don’t want these users deleting files, so I specified the exact SFTP requests that members of sftp_users are able to execute.

ForceCommand internal-sftp -d / -p open,close,read,write,realpath,opendir,readdir,mkdir,readlink

However, a “destructive write” is essentially the same thing as deletion. How can we allow users to upload files, but not overwrite them? Don’t worry, we’ll get there.

Right after I made this change, I started having issues using the command-line program sftp. FileZilla continued to work just fine. From the man page of sftp-server, here’s what the -p flag does.

-p allowed_requests
             Specifies a comma-separated list of SFTP protocol requests that
             are permitted by the server.  All request types that are not on
             the allowed list will be logged and replied to with a failure
             message.

             Care must be taken when using this feature to ensure that re‐
             quests made implicitly by SFTP clients are permitted.

Well, I didn’t read that last part when I started out. sftp had issues accessing my server until I added stat to the list of permitted requests.

At first, I loved the idea of allowing my users to add data to my NAS in a categorized fashion, but I quickly learned that many of my users did not care about organization as much as myself. A sysadmin friend of mine recommended an incoming directory, for which all users would have write access. From there, an admin would have to triage the uploaded data. When a user logs in, they should be able to see the NAS and the incoming directories. The former should be read-only, and the latter writable, but somehow disallow for “destructive” writes. Both of these can be achieved with bindfs.

# read-only NAS
bindfs -u root -g root -p u=rwD,g=rD,dg=rD,o=rD /mnt/nas/share/ /nas_chroot/share

# writable but root-owned incoming directory
bindfs u root -g root -p u=rwxD,do=rwxD,fo=r /mnt/nas/incoming/ /nas_chroot/incoming

Of course, we’ll need to adjust the ChrootDirectory for sftp_users to /nas_chroot.

What’s neat about this setup is that when a user goes to write to the incoming directory, their write will succeed. However, when they list the contents of the directory, suddenly the uploaded file is no longer owned by the uploader! Permissions are retained on the actual filesystem, but are obfuscated in the bindfs mount. After uploading a file, the uploader can no longer modify the file they uploaded. This method has one issue, and that’s if an uploader’s connection fails while uploading a file, it’ll be stuck in an botched and immutable state. So the uploader would need to change the file name or upload path.

Now we have a somewhat-usable fileserver that allows peasants users to upload to an effectively immutable limited filesystem, and browse a read-only section of the NAS. However, I never mentioned how I manage users. I won’t get too into detail, but suffice it to say that I create LDAP accounts in an automated fashion by allowing users to register their own credentials at a non-published URL. The user must provide a TOTP code from a secret I control in order to register. Once successful, an account is created for them on the LDAP server. Thus, when I want to onboard someone, I communicate the registration link and provide them a TOTP token when they’re ready to join.

And that’s about it. It’s an SFTP server using bindfs to limit what users can access, with an onboarding flow through LDAP. It’s nothing impressive, but I had an interesting time building it and coming across the aforementioned issues.