# Introduction – The problem

The problem is the following: You run some service where you need to have regular backups. The data to be backed up can be represented as something you can put in a tarred archive. The common way to do this is to pipe the tar output to OpenSSL and encrypt with AES-256, which is fine, except that it requires inputting the password constantly or putting a static password on the server. If you’ve guessed that we’re going to use public key cryptography, you’re right, but there is a problem there.

# The typical way of doing this with passwords

If you research this topic online, you’ll find the internet flooded with solutions that look like this:

$tar -czpf - /path/to/directory-to-backup | openssl aes-256-cbc -salt -pbkdf2 -iter 100000 -pass | dd of=output.aes There are a few problems with this… let me state them and let’s see whether we can do better: • You have to input the password manually for this to work, so this cannot go into a cron/periodic job unless you expose the password • Since you have to put the password manually, it has to be some text, which lacks entropy, given that pbkdf2 (as far as I know) uses SHA256; where if you know anything about bitcoin mining, you’ll know that SHA256 calculations have become very cheap… meaning that your password that you write with your hand, most likely, lacks entropy for security • It uses AES-256-CBC, which lacks authentication… but let’s not get into that since OpenSSL has support issues for the better alternative, GCM, but I’ll provide a simple solution to this at the end Let’s see if we can do better. # How can public key cryptography solve these problems? ### Introduction to public key cryptography Feel free to skip this little section if you know what public key crypto is. We call symmetric key cryptography, the kind of cryptography where the encryption and decryption, or signing and verifying, uses the same key. In public key cryptography, or asymmetric key cryptography, we use the public key to encrypt data, and private key to decrypt data. Or we use the private key to sign data, and public key to verify signatures. ### But public key encryption doesn’t really help as a replacement It may be a surprise, but there’s no way to encrypt large data with public keys. The reason for this is that encryption algorithms usually are either done in blocks (like in AES), or in streams (like Chacha20). Public key encryption that we have is either considered a block cipher (like RSA encryption), or is used to generate symmetric keys using some agreement protocol, like Diffie-Hellman, which then uses symmetric encryption. And given that RSA encryption is very expensive, because it’s an exponentiation of a large number, it’s not really practical to use it for large data, so no standard algorithms are found in software like OpenSSL to encrypt your backups. You’ll never see RSA-4096-CBC or RSA-4096-GCM for that reason. ### How then do we encrypt without a password? The trick is simple. We generate a truly random password, encrypt the password with RSA and store it in a file, use the password for encryption, then we forget the password and move on. The password then can only be recovered with a private key (which will be encrypted). Let’s see how we do this. First, in your local machine where you’re OK with unpacking the data, generate your RSA keys. Generate your decryption private key (encrypted with a passphrase): $ openssl genrsa -aes256 -out priv_key.pem 4096

and from that, generate your public key

$openssl rsa -in priv_key.pem -outform PEM -pubout -out pub_key.pem Now this password that you use to encrypt this key will be the only password you’ll ever type. Your backups don’t need it. Now copy the pub_key.pem file to the remote machine where backups have to happen. If you have a backup bash script, you can do the following now: #!/bin/bash # generate a random password with 256-bit entropy, so PBKDF2 won't be a problem password=$(openssl rand -hex 64)

tar -czpf - /path/to/directory-to-backup | openssl aes-256-cbc -salt -pbkdf2 -iter 100000 -pass pass:${password} | dd of=output.aes # encrypt the password using your RSA key, and write the output to a file echo${password} | openssl rsautl -encrypt -inkey pub_key.pem -pubin -in - -out output.password.enc

To view the password in that file, you can view it in stdout using this command:

$openssl rsautl -decrypt -inkey priv_key.pem -in output.password.enc -out - This will print the password to your terminal, which you can copy and use to decrypt your encrypted archive. Finally, to decrypt your data, run the command: $ dd if=output.aes | openssl aes-256-cbc -d -salt -pbkdf2 -iter 100000 | tar -xpzf -

and when asked, input the password you copied from the private key decryption of the command before

WARNING: It’s WRONG to use the same password and encrypt it again and again and again with RSA (with this scheme, at least). Be aware that encrypting the same password with the same public key will yield the same output. This means that someone with access to the encrypted files can recognize that you’re using the same password everywhere.

# Conclusion

No need, anymore, to choose between encrypting your backups and not being there. You can have both with this simple trick. This is also more secure because the 256-bit password generated randomly can never be brute-forced through the cheap pbkdf2, which basically means that brute-forcing the encryption itself is cheaper than brute-forcing the password that derived the key, which will guarantee the 256-bit encryption strength.

Now about using CBC instead of GCM, you can create your own HMAC to authenticate your archives. Or, even easier, you can call sha256sum on your archive and encrypt the output with your public key or even sign it. That’s good enough and can be streamlined into anything you automate for your backups.

I just wanna mention also that this isn’t an innovation of mine by any means… this is called in OpenSSL the “envelope encryption”, but for some reason it’s only supported when using the C library and not when using the OpenSSL executable. This is also a common way for encryption in many applications because asymmetric encryption for data is impractical with the available tooling. I wrote this article because it seems that no one talks about this and everyone is busy typing possibly insecure passwords in their encryption archives. Hopefully this will become more common after this article.

# Abandoning the server provider from 10-years: Stay away from myLoc/ServDiscount/Webtropia (review)

### Introduction

It’s a hot question what server providing service to use nowadays. The industry has boomed. The only thing that matters, though, is the reliability of a provider. With reliable I mean, the continuity of the service with no interruptions, the mutual understanding that problems can happen, and the urgency to correct problems to continue operation with minimal down-time. Ten years ago, myLoc was a good company. Nowadays, they’re so cheap and bad I’m leaving them. I guess inflation didn’t just inflate the currency.

### Why write this in a blog post?

Because people on the internet have become so un-empathetic, robotic, and exhausting to talk to or discuss with, and my blog will minimize that because no one cares about my blog. Maybe it’s a diary at this point. I can’t remember in the last few years when I posted something on social media, and didn’t get people licking the boot of corporations under the guise of “violating their terms” or whatever other dumb reason. As if common sense doesn’t have any value anymore. Trust isn’t of any value. I’m quite sure some genius out there will find a way to twist this story and make it entirely my fault, because humanity now evaluates moral decisions based on legality… what a joke we’ve become. Nowadays, you’re not allowed to be angry. You can’t be upset. You shouldn’t be frustrated. You can’t do mistakes. If you express any of your feelings towards an experience; your expression, regardless of how reasonable it’s, will be called a rant and dismissed. So, I guess my blog is the best place to put this “rant”, if you will. Who cares after all?!

I guess this will have a better chance to reach people than any robotic social media out there.

### Story starts with bad internet connection

A few months ago I contacted ServDiscount (a subsidiary of myLoc, where my server is, IIUC) about constant internet connection problems (ticket 854255-700134-52). The internet connection problems started like a year ago, but recently got so bad (while my home’s internet is around 1 Gbps). The problem was illustrated in many ways

• File copying through sftp/ssh, was almost impossible. The speed throttles continuously and the download stops.
• If I use any ssh tunnel, the terminal will not be usable anymore while the tunnel is doing any transfers, even small data was enough to block it
• Games with server software running on my server there could never be continued due to connection loss
• ftps downloads constantly show “connection reset by peer”, which I believe is the problem with everything else. The server-side connection was just resetting all the time.

As a patient and busy person, I didn’t care. This went on for MONTHS. I’m not exaggerating. It was when this problem went out of control. Then, under the assumption that “it’s my fault”, I rented a specific VPS from ServDiscount, and guess what? Same problems. The VPS had NOTHING but a game server in it (plus iptables firewall with blocked input policy + open some ports), yet it was impossible to finish a game or two. People from all around the world would get disconnected after a while.

### Bad support, all the way (starting October)

When I contacted support about this connection problem, the first blind response was “it’s my fault”. Because for this (bad) company, it can never be their fault. All their configuration is perfect. Everything they do is perfect. They can never do mistakes. That’s what pissed me off. They couldn’t explain the disconnects in the gaming server OR my original server. But of course, they claimed it might be a “botnet”… because why not? Since it’s virtually impossible to deny the presence of botnet except with plausible deniability, they can never lose this argument. Yet with the presence of the second server, it was easy to prove that something is wrong there. It was easy to reproduce the problem, since an http download would fail. That was ready to view, test and debug. So, after that pressure, some guy in their support said “I’ll take a look”, and asked for access. I took the risk and created an account for them with sudoers power and added their public key, but they never logged in. It was easily observable in the logs, for weeks. Just empty promises with zero accountability.

After nagging them, they said “they’re busy”… great support. As if I’m asking for a favor or something.

Then I got exhausted, and to appease them, I decided to wipe my server. I wish it were that easy.

### Get your backups, if you can

Because of these internet issues, my backups were trapped on that server. I’ve tried to download them in many ways, but my backup’s server software were constantly resetting the connection. Keep in mind that the software was used for years in the past. It’s not like I invented these solutions recently or something. In fact, my only “invention”, because of these problems, was to make these scripts run multiple times a day… just in an attempt to get the downloads to happen. For over a month, I couldn’t download any backups. My backup logs were flooded with “connection reset by peer” errors. What the hell am I supposed to do now?

Well, I remained patient. It was a mistake. I should’ve probably escalated and gotten angry? But I don’t think they would’ve done anything. When it’s a bad company, it’s a bad company.

### Docker, as a tool to easily move away from bad situations

I didn’t know too much about docker other than it’s a containerization service that I barely used. In fact I used LXD more because it’s modeled around preserving OSs, while docker is modeled around stateless preservation of data, where the software is renewed constantly, but the data is preserved.

The idea was simple: Instead of having to install all services on bare-metal and suffer when incompetent support treats me like this in migrations, I could just tar the containers with their data, move them somewhere else, and continue operation as if nothing happened. This way, I’ll save time I don’t really have, since I work now as a lead engineer and I can’t really actively solve the same problems like I did back then when I was in academia. Sad fact of life: The older you grow, the less time you have.

So, I started learning docker. Watched tutorials, read articles, how-to, and many other things. I built images, did many things, and things were going fine. I containerized almost all my services. But one service was the cause of all evil…

### Containerizing email services… big mistake and lots learned

The problem with containerization is that often you’ll need a proxy to deliver your connections. You can setup your docker container to share the host’s network, but I find that ugly and defeats the isolation picture of docker. My email server was proxied through haproxy.

Unfortunately, because of proxying, and because of my lack of understanding of how that worked (because I was learning), I misconfigured my email proxy and all the incoming connections to the email server were seen as “internal” connections because the source was the proxy software from within the server’s network, which basically meant that all incoming email relay requests were accepted.

### Our Nigerian friends and their prince were ready

Once that mistake was made, within a day, the server was being used as a relay for spam.

### ServDiscount and their reaction

On 16.12.2022, because one email ended in a spam trap, the “UCEProtect” service flagged my server, and let’s try to guess what ServDiscount/myLoc did:

1. Contacted me, informing me of the issue and asking to correct it
2. Switching the server to rescue mode, so that I can investigate the issue through the logs
3. Completely block access to my server with no way for me to access the logs to understand the problem
4. Refuse to unblock the server (even in rescue mode) unless UCEProtect gives the stamp of approval

Yep, it’s 3 and 4.

It’s completely understandable to block my server. After all, it was sending spam. No question there. But the annoying part is that I only knew about this whole thing by coincidence, when I attempted access to my server and couldn’t. They only sent an email, which I didn’t receive because my email server was blocked. They didn’t attempt a phone-call, SMS or anything. They didn’t even open a ticket IN MY ACCOUNT WITH THEM!

You might wonder whether it can get any more careless, passive and negligent from their end. Yes, it can!

Remember that I couldn’t even download my backups because of their crappy internet connection, which they didn’t diagnose or help fix. So I’m literally now trapped with no email access, my data is locked (for who knows how long), and… it gets worse.

I contacted their customer support. What do you think the answer was?

1. We understand. Let us switch your server to rescue mode so that you can understand what happened
2. Remove your server from UCEProtect’s list (by paying them money) then come back

You guessed it. It’s no. 2.

And again, remember, I can’t even gain rescue access to my server. So until this point, I don’t even have an idea what happened.

I go to UCEProtect, pay them around \$100 (it was a Friday). Then went back to customer support. Guess their response:

1. Thank you for removing your server from the list. Here’s access back so that you can investigate the issue.
2. We have zero communication with the “abuse department”, so wait for them to react… fill the form they sent you to the email that you can’t access.

You also guessed it. No. 2.

No amount of tickets, phone-calls, or begging helped. A robot would’ve understood better that I couldn’t access my email so I have no way of responding.

Whether it’s with bad policy or otherwise, they never help when you need help. They couldn’t care less. So what if your server was stopped for days or even weeks based on UCEProtect listing? We don’t care, and we won’t help you fix the problem.

### DNS change

While support was completely ignoring me, it occurred to me that I could switch my email MX DNS record to another service and receive my emails. Thanks to how MTAs (mail transfer agents) work, they don’t give up so easily. On 17.12.2022, I got the email on another email provider, who I paid just to solve this problem, filled the form (without knowing why the issue happened, because again, how the hell will I know without access to my logs, even in rescue mode?!).

### The holy abuse department is on the other side of the universe

So while I did everything they asked for. I filled the form, I removed my IP address from UCECrap, I didn’t get my server until days later. Because the abuse department is simply unreachable. There’s basically zero urgency to solve such a problem for a customer (who’s been with them for 10 years). Who gives a flying f**k if your server and work halted for days?! Not myLoc, apparently.

### Let’s summarize

• Bad internet connection with no help, so I couldn’t even get my backups
• No urgency to solve a problem I obviously didn’t do deliberately, with zero value for being with them for 10 years
• No access to server in rescue mode or VPN, yet I have to explain how the outbreak happened without logs, who knows how
• Only contact through email, which was blocked, which they never addressed
• Even after fulfilling their conditions, no one cares. No one helped, because the “abuse department” is on the other side of the universe.

What’s the point of having a server with a company that treats its customers like that?

What’s the point when accountability is too much to ask?

I don’t know.

### Conclusion

I’m not gonna sit here and claim that I’m perfect and none of this is my responsibility. But at least I expect common sense treatment for customers, urgency to help and reasonable way to access the server to see the logs, whether in rescue mode or VPN or whatever, and a reachable “abuse department” that doesn’t hold all the keys with no accountability to customers. All this, and add that I’ve been with them for a decade.

But basically, the lesson here is, if anything goes wrong with your server in myLoc, you can go f**k yourself. No one will help you. You don’t really matter to them. Unlike other companies that will simply put your server in rescue mode and ask you to fix the problem. MyLoc will simply block you. Maybe they’re taking a cut out of UCEProtect? I don’t know. But you can only wonder why a company will be that stupid. I’m out.

Have a great one.