a few weeks after provisioning a VPS from Hetzner, which had a nightmarish verification process to even use their platform mind you, i started getting pings from automated abuse detection.

i ignored this at first as "maybe my service was just doing some wonky networking", it was a mastadon instance so that didn't seem too far fetched to me.

i don't think mastadon port scans random IP ranges, so i shutdown my VPS from the cloud console ASAP and tried to figure out what went wrong.

it was a personal instance with no data i cared about leaking / losing, but it could've been much worse.

the provisioning process for a Hetzner box is pretty straight forward, if you've used a cloud console it has what you'd expect.

you can select an OS, choose from some preconfigured software, and then you can add your SSH keys!

i personally prefer to add my SSH keys via the provided serial console that most cloud providers give you, it also sends you a root password in the mail that you have to change.

there's also this box to specifically add a cloud-init configuration, i'm going to leave this blank since i don't want a cloud-init configuration.

that sums up our configuration, a basic Ubuntu machine, no ssh keys selected, and no cloud-init config.

and this is why i use(d) Hetzner, dirt cheap!

with that done we can SSH into the machine with the emailed admin credentials.

> ssh root@157.180.76.96

The authenticity of host '157.180.76.96 (157.180.76.96)' can't be established.
ED25519 key fingerprint is: SHA256:WxwzGh584qBajyNC2vFcQF2ToKzlcpK0tSPEZbS78iM
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?y

root@157.180.76.96's password: *********

You are required to change your password immediately (administrator enforced).
Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-71-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

 System information as of Wed Nov 19 10:09:04 PM UTC 2025

  System load:  0.08              Processes:             120
  Usage of /:   3.0% of 37.23GB   Users logged in:       0
  Memory usage: 5%                IPv4 address for eth0: 157.180.76.96
  Swap usage:   0%                IPv6 address for eth0: 2a01:4f9:c012:d8db::1

alright great, we're in the machine. we've changed the password, and now we want to secure our VPS so that only we can access it. (there are other things we should do, but this is the most important one in terms of urgency.)

we just need to only allow pub/priv key authentication for sshd. to do this we just need to edit the /etc/ssh/sshd_config. https://stackoverflow.com/questions/20898384/disable-password-authentication-for-ssh

in that file we'll find this snippet.

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes

alright cool, let's just uncomment that (back of mind: huh, weird that it's commented out, does it allow passwords by default? whatever.), and set it to no like it says.

# To disable tunneled clear text passwords, change to no here!
- #PasswordAuthentication yes
+ PasswordAuthentication no

and to enable our changes, just restart the ssh service.

sudo systemctl restart ssh

now let's exit and ssh, we should be "locked out".

logout
Connection to 157.180.76.96 closed.
➜  moon ssh root@157.180.76.96
root@157.180.76.96's password:

huh, that's weird, i swear we just disabled password authentication, maybe it won't work if we submit our new password?

root@157.180.76.96's password: ****
Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-71-generic x86_64)

wtf, what's going on here? let's look at that sshd config one more time.

root@hetzner-example:~# cat /etc/ssh/sshd_config

# this line is important
Include /etc/ssh/sshd_config.d/*.conf

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

huh, i wonder what that Include sshd_config.d thingy is doing? what's in there?

root@hetzner-example:~# ls /etc/ssh/sshd_config.d
50-cloud-init.conf

oh it's just a cloud-init confi-, wait i didn't use cloud-init did i? wtf is that?

root@hetzner-example:~# cat /etc/ssh/sshd_config.d/50-cloud-init.conf
PasswordAuthentication yes

WTF? is it a SINGLE file, whose sole purpose is to set PasswordAuthentication to yes, which appears to override me setting it to no? surely not.

to test i guess, lets delete it and reload sshd to see what happens. maybe i did something wrong?

root@hetzner-example:~# rm /etc/ssh/sshd_config.d/50-cloud-init.conf
root@hetzner-example:~#
root@hetzner-example:~# sudo systemctl restart ssh
root@hetzner-example:~# exit
logout
Connection to 157.180.76.96 closed.

great, now lets' try logging back in.

➜ moon ssh root@157.180.76.96
root@157.180.76.96: Permission denied (publickey).

oh, cool, so that's how my server got pwned.

tl;dr + key takeways

hetzner includes a weird cloud-init that enables SSH password authentication. hetzner does not tell you they do this, and even allow you to have an empty cloud-init configuration, which is the default.

sshd_configs apply the configuration "bottom up", the first thing set is the value it'll be evaluated to at runtime, meaning that us disabling the password authentication gets overriden by the Include. whatever password i set was obtained and thus my machine pwned.

when i did this for my mastodon server, the first thing i did was add my ssh keys to the machine. when i was able to login after disabling ssh i wasn't surprised, now i'll be checking on every machine i provision in the future.

lessons learned:

provide your ssh pubkeys up front to your cloud provider so they don't apply weirdo configs and send your password over email.

absolutely make sure all account passwords are secure and securely stored.

disable ssh root access entirely, leave it to the serial console if you can help it.

sandbox things into users.

something something fail2ban.

make sure you actually disable password authentication for SSHD.

remaining questions?

why does Hetzner do it this way? is this an Ubuntu thing? this behavior doesn't happen notably if you provide your SSH keys up front (my new preference after this fiasco).

hetzner already edits the sshd_config as the ubuntu ssh package has #PermitRootLogin prohibit-password as the default config (source, my friend octy https://octyl.net/) so this rules out not wanting to modify the default config.

in fact sshd by DEFAULT allows for password based ssh access without specifying PasswordAuthentication yes, so there's no apparent reason for them to even add that cloud-init configuration in the first place!

suggested fix?

add some kind of warning or comment via

sed -i -E 's/#?PasswordAuthentication yes/# disable this in the cloud-init thingy/ /etc/ssh/sshd_config

into the sshd_config to let others know they can't just disable it where they expect.