Tag Archives: ssh

Abandoning the server provider from 10-years: Stay away from myLoc/ServDiscount/Webtropia (review)

Leave a comment

Introduction

It’s a hot question what server providing service to use nowadays. The industry has boomed. The only thing that matters, though, is the reliability of a provider. With reliable I mean, the continuity of the service with no interruptions, the mutual understanding that problems can happen, and the urgency to correct problems to continue operation with minimal down-time. Ten years ago, myLoc was a good company. Nowadays, they’re so cheap and bad I’m leaving them. I guess inflation didn’t just inflate the currency.

Why write this in a blog post?

Because people on the internet have become so un-empathetic, robotic, and exhausting to talk to or discuss with, and my blog will minimize that because no one cares about my blog. Maybe it’s a diary at this point. I can’t remember in the last few years when I posted something on social media, and didn’t get people licking the boot of corporations under the guise of “violating their terms” or whatever other dumb reason. As if common sense doesn’t have any value anymore. Trust isn’t of any value. I’m quite sure some genius out there will find a way to twist this story and make it entirely my fault, because humanity now evaluates moral decisions based on legality… what a joke we’ve become. Nowadays, you’re not allowed to be angry. You can’t be upset. You shouldn’t be frustrated. You can’t do mistakes. If you express any of your feelings towards an experience; your expression, regardless of how reasonable it’s, will be called a rant and dismissed. So, I guess my blog is the best place to put this “rant”, if you will. Who cares after all?!

I guess this will have a better chance to reach people than any robotic social media out there.

Story starts with bad internet connection

A few months ago I contacted ServDiscount (a subsidiary of myLoc, where my server is, IIUC) about constant internet connection problems (ticket 854255-700134-52). The internet connection problems started like a year ago, but recently got so bad (while my home’s internet is around 1 Gbps). The problem was illustrated in many ways

  • File copying through sftp/ssh, was almost impossible. The speed throttles continuously and the download stops.
  • If I use any ssh tunnel, the terminal will not be usable anymore while the tunnel is doing any transfers, even small data was enough to block it
  • http(s) downloads of large files have become impossible; a simple attempt to download a file of 400 MB can’t be accomplished.
  • Games with server software running on my server there could never be continued due to connection loss
  • ftps downloads constantly show “connection reset by peer”, which I believe is the problem with everything else. The server-side connection was just resetting all the time.

As a patient and busy person, I didn’t care. This went on for MONTHS. I’m not exaggerating. It was when this problem went out of control. Then, under the assumption that “it’s my fault”, I rented a specific VPS from ServDiscount, and guess what? Same problems. The VPS had NOTHING but a game server in it (plus iptables firewall with blocked input policy + open some ports), yet it was impossible to finish a game or two. People from all around the world would get disconnected after a while.

Bad support, all the way (starting October)

When I contacted support about this connection problem, the first blind response was “it’s my fault”. Because for this (bad) company, it can never be their fault. All their configuration is perfect. Everything they do is perfect. They can never do mistakes. That’s what pissed me off. They couldn’t explain the disconnects in the gaming server OR my original server. But of course, they claimed it might be a “botnet”… because why not? Since it’s virtually impossible to deny the presence of botnet except with plausible deniability, they can never lose this argument. Yet with the presence of the second server, it was easy to prove that something is wrong there. It was easy to reproduce the problem, since an http download would fail. That was ready to view, test and debug. So, after that pressure, some guy in their support said “I’ll take a look”, and asked for access. I took the risk and created an account for them with sudoers power and added their public key, but they never logged in. It was easily observable in the logs, for weeks. Just empty promises with zero accountability.

After nagging them, they said “they’re busy”… great support. As if I’m asking for a favor or something.

Then I got exhausted, and to appease them, I decided to wipe my server. I wish it were that easy.

Get your backups, if you can

Because of these internet issues, my backups were trapped on that server. I’ve tried to download them in many ways, but my backup’s server software were constantly resetting the connection. Keep in mind that the software was used for years in the past. It’s not like I invented these solutions recently or something. In fact, my only “invention”, because of these problems, was to make these scripts run multiple times a day… just in an attempt to get the downloads to happen. For over a month, I couldn’t download any backups. My backup logs were flooded with “connection reset by peer” errors. What the hell am I supposed to do now?

Well, I remained patient. It was a mistake. I should’ve probably escalated and gotten angry? But I don’t think they would’ve done anything. When it’s a bad company, it’s a bad company.

Docker, as a tool to easily move away from bad situations

I didn’t know too much about docker other than it’s a containerization service that I barely used. In fact I used LXD more because it’s modeled around preserving OSs, while docker is modeled around stateless preservation of data, where the software is renewed constantly, but the data is preserved.

The idea was simple: Instead of having to install all services on bare-metal and suffer when incompetent support treats me like this in migrations, I could just tar the containers with their data, move them somewhere else, and continue operation as if nothing happened. This way, I’ll save time I don’t really have, since I work now as a lead engineer and I can’t really actively solve the same problems like I did back then when I was in academia. Sad fact of life: The older you grow, the less time you have.

So, I started learning docker. Watched tutorials, read articles, how-to, and many other things. I built images, did many things, and things were going fine. I containerized almost all my services. But one service was the cause of all evil…

Containerizing email services… big mistake and lots learned

The problem with containerization is that often you’ll need a proxy to deliver your connections. You can setup your docker container to share the host’s network, but I find that ugly and defeats the isolation picture of docker. My email server was proxied through haproxy.

Unfortunately, because of proxying, and because of my lack of understanding of how that worked (because I was learning), I misconfigured my email proxy and all the incoming connections to the email server were seen as “internal” connections because the source was the proxy software from within the server’s network, which basically meant that all incoming email relay requests were accepted.

Our Nigerian friends and their prince were ready

Once that mistake was made, within a day, the server was being used as a relay for spam.

ServDiscount and their reaction

On 16.12.2022, because one email ended in a spam trap, the “UCEProtect” service flagged my server, and let’s try to guess what ServDiscount/myLoc did:

  1. Contacted me, informing me of the issue and asking to correct it
  2. Switching the server to rescue mode, so that I can investigate the issue through the logs
  3. Completely block access to my server with no way for me to access the logs to understand the problem
  4. Refuse to unblock the server (even in rescue mode) unless UCEProtect gives the stamp of approval

Yep, it’s 3 and 4.

It’s completely understandable to block my server. After all, it was sending spam. No question there. But the annoying part is that I only knew about this whole thing by coincidence, when I attempted access to my server and couldn’t. They only sent an email, which I didn’t receive because my email server was blocked. They didn’t attempt a phone-call, SMS or anything. They didn’t even open a ticket IN MY ACCOUNT WITH THEM!

You might wonder whether it can get any more careless, passive and negligent from their end. Yes, it can!

Remember that I couldn’t even download my backups because of their crappy internet connection, which they didn’t diagnose or help fix. So I’m literally now trapped with no email access, my data is locked (for who knows how long), and… it gets worse.

I contacted their customer support. What do you think the answer was?

  1. We understand. Let us switch your server to rescue mode so that you can understand what happened
  2. Remove your server from UCEProtect’s list (by paying them money) then come back

You guessed it. It’s no. 2.

And again, remember, I can’t even gain rescue access to my server. So until this point, I don’t even have an idea what happened.

I go to UCEProtect, pay them around $100 (it was a Friday). Then went back to customer support. Guess their response:

  1. Thank you for removing your server from the list. Here’s access back so that you can investigate the issue.
  2. We have zero communication with the “abuse department”, so wait for them to react… fill the form they sent you to the email that you can’t access.

You also guessed it. No. 2.

No amount of tickets, phone-calls, or begging helped. A robot would’ve understood better that I couldn’t access my email so I have no way of responding.

Whether it’s with bad policy or otherwise, they never help when you need help. They couldn’t care less. So what if your server was stopped for days or even weeks based on UCEProtect listing? We don’t care, and we won’t help you fix the problem.

DNS change

While support was completely ignoring me, it occurred to me that I could switch my email MX DNS record to another service and receive my emails. Thanks to how MTAs (mail transfer agents) work, they don’t give up so easily. On 17.12.2022, I got the email on another email provider, who I paid just to solve this problem, filled the form (without knowing why the issue happened, because again, how the hell will I know without access to my logs, even in rescue mode?!).

The holy abuse department is on the other side of the universe

So while I did everything they asked for. I filled the form, I removed my IP address from UCECrap, I didn’t get my server until days later. Because the abuse department is simply unreachable. There’s basically zero urgency to solve such a problem for a customer (who’s been with them for 10 years). Who gives a flying f**k if your server and work halted for days?! Not myLoc, apparently.

Let’s summarize

  • Bad internet connection with no help, so I couldn’t even get my backups
  • No urgency to solve a problem I obviously didn’t do deliberately, with zero value for being with them for 10 years
  • No access to server in rescue mode or VPN, yet I have to explain how the outbreak happened without logs, who knows how
  • Only contact through email, which was blocked, which they never addressed
  • Even after fulfilling their conditions, no one cares. No one helped, because the “abuse department” is on the other side of the universe.

What’s the point of having a server with a company that treats its customers like that?

What’s the point when accountability is too much to ask?

I don’t know.

Conclusion

I’m not gonna sit here and claim that I’m perfect and none of this is my responsibility. But at least I expect common sense treatment for customers, urgency to help and reasonable way to access the server to see the logs, whether in rescue mode or VPN or whatever, and a reachable “abuse department” that doesn’t hold all the keys with no accountability to customers. All this, and add that I’ve been with them for a decade.

But basically, the lesson here is, if anything goes wrong with your server in myLoc, you can go f**k yourself. No one will help you. You don’t really matter to them. Unlike other companies that will simply put your server in rescue mode and ask you to fix the problem. MyLoc will simply block you. Maybe they’re taking a cut out of UCEProtect? I don’t know. But you can only wonder why a company will be that stupid. I’m out.

Have a great one.

Tunnel through https to your ssh server, and bypass all firewalls – The perfect tunnel! (HAProxy + socat)

5 Comments

Disclaimer

Perhaps there’s no way to emphasize this more, but I don’t encourage violation of corporate policy. I do this stuff for fun, as I love programming and I love automating my life and gaining more convenience and control with technology. I’m not responsible for any problem you might get with your boss in your job for using this against your company’s firewall, or any similar problem for that matter.

Introduction

I was in a hotel in Hannover, when I tried to access my server’s ssh. My ssh client, Putty, gave this disappointing message

PuttyConnectionRefused

At first I got scared as I thought my server is down, but then I visited the websites of that server, and they were fine. After some investigation, I found that my hotel blocks any access to many ports, including port 22, i.e., ssh. Did this mean that I won’t have access to my server during my trip? Not really!

I assume you’re using a Windows client, but in case you’re using linux, the changes you have to do are minimal, and I provide side-by-side how to do the same on a linux client. Let me know if you have a problem with any of this.

Tunneling mechanism, and problems with other methods that are already available

There are software that does something similar for you automatically, like sslh, but there’s a problem there.

What does sslh do?

When you install sslh on your server, you choose, for example, port 443 for it. Port 443 is normally for http-ssl (https), that’s normally taken by your webserver. So you change also your webserver’s port to some arbitrary port, say 22443. Then, say you want to connect to that server: sslh analyzes and detects whether the incoming network packets are ssh or http. If the packets are ssh, it forwards them to port 22. If the packets looks like https, it forwards them to the dummy port you chose, which is 22443 as we assumed.

What’s the problem with sslh, and similar programs?

It all depends on how sophisticated the firewall you’re fighting is. Some firewalls are mediocre, and they just blindly open port 443, and you can do your sslh trick there and everything will work fine. But smart firewalls are not that dull; they analyze your packets and then judge whether you’re allowed to be connected. Hence, a smart firewall will detect that you’re trying to tunnel ssh, and will stop you!

How do we solve this problem?

The solution is: Masquerade the ssh packets inside an https connection, hence, the firewall will have to do a man-in-the-middle attack in order to know what you’re trying to do. This will never happen! Hence, I call this solution: “The perfect solution“.

How to create the tunnel?

I use HAProxy for this purpose. You need that on your server. It’s available in standard linux systems. In Debian and Ubuntu, you can install it using

sudo apt-get install haproxy

You will need “socat” on your client to connect to this tunnel. This comes later after setting up HAProxy.

How does HAProxy work?

I don’t have a PhD in HAProxy, it’s fairly a complicated program that can be used for many purposes, including load balancing and simple internal proxying between different ports, and I use it only for this purpose. Let me give a brief explanation on how it works. HAProxy uses the model of frontends and backends. A frontend is what a client sees. You set a port there, and a communication mode (tcp, for example). You tell also a frontend “where these packets should go”, based on some conditions (called ACL, Access Control Lists). You choose to which backend the packets have to go. The backend contains information about the target local port. So in short words, you tell HAProxy where to forward these packets from the frontend to the backend based on some conditions.

A little complication if you use https websites on the same server

If you use https webserver on the same machine, you’ll have a problem. The problem is that you’ll need to check whether the packets are ssh before decrypting them, because once you decrypt them, you can’t use them as non-encrypted again (hence haproxy doesn’t support forking encrypted and decrypted packets side-by-side). This is because you choose to decrypt in your frontend. That’s why we use SNI (Server Name Indication) and do a trick:

  • If there’s no SNI (no server name, just IP address), then forward to ssh
  • If server name used is ssh.example.com (some subdomain you choose), then forward to ssh (optional)
  • If anything else is the case, forward to the https web server port

We also use 2-frontends. The first one is the main one, and the second is a dummy frontend, and is only used to decrypt the ssh connection’s https masquerade. HAProxy decrypts only in frontends.

Let’s do it!

The configuration file of HAProxy in Debian/Ubuntu is

/etc/haproxy/haproxy.cfg

You could use nano, vi or vim to edit it (you definitely have to be root or use sudo). For example:

sudo nano /etc/haproxy/haproxy.cfg
Assumptions
  1. Your main https port is 443
  2. Your main ssh port is 22
  3. Your https webserver is now on 22443
  4. The dummy ssh port is 22222 (used just for decryption, it doesn’t matter what you put it)
Main frontend

This is the frontend that will take care of the main port (supposedly 443). Everything after a sharp sign (#) on a line is a comment.

#here's a definition of a frontend. You always give frontends and backends a name
frontend TheMainSSLPort
 mode tcp
 option tcplog
 bind 0.0.0.0:443 #listen to port 443 under all ip-addresses

 timeout client 5h #timeout is quite important, so that you don't get disconnected on idle
 option clitcpka

 tcp-request inspect-delay 5s
 tcp-request content accept if { req_ssl_hello_type 1 }

 #here you define the backend you wanna use. The second parameter is the backend name
 use_backend sshDecrypt if !{ req_ssl_sni -m found } #if no SNI is given, then go to SSH
 use_backend sshDecrypt if { req_ssl_sni -i ssh.exmple.com } #if SNI is ssh.example.com, also go to ssh

default_backend sslWebServerPort #if none of the above apply, then this is https

In the previous configuration, we have two paths for the packets, i.e., two backends:

  1. If the connection is ssh, the backend named “sshDecrypt” will be used.
  2. If the connection is https, the backend named “sslWebServerPort” will be used.
The https backend

I put this here first because it’s easier. All you have to do here is forward the packets to your webserver’s port, which we assumed to be port 22433. The following is the relevant configuration:

backend sslWebServerPort
 mode tcp
 option tcplog
 server local_https_server 127.0.0.1:22443 #forward to this server, port 22443

Now the https part is done. Let’s work on the ssh part.

The ssh front- and backends

We’ll have to use a trick, as mentioned before, to get this to work. Once a judgment is done for packets to go to ssh (using SNI), the packets have to be decrypted. This is not possible in a backend, thus we use a backend to forward the packets to a dummy frontend that decrypts the packets, and then send these to another backend to send the packets to the ssh server.

backend sshToDecryptor
 mode tcp
 option tcplog
 server sshDecFrontend 127.0.0.1:22222
 timeout server 5h

This forwards the packets to port 22222. Now we build a frontend at that port that decrypts the packets.

frontend sshDecyprtionPort
 timeout client 5h
 option clitcpka

 bind 0.0.0.0:22222 ssl crt /path/to/combined/certs.pem no-sslv3
 mode tcp
 option tcplog

 tcp-request inspect-delay 5s
 tcp-request content accept if HTTP

default_backend sshServ #forward to the ssh server backend

The file /path/to/combined/certs.pem has to contain your private key, certificate and certificate chain in one file of your SSL. Concatenate them all in one file.

Finally, the back end to the ssh server:

backend sshServ
 mode tcp
 option tcplog
 server sshServer1 127.0.0.1:22
 timeout server 5h

That’s all you need to create the tunnel.

Test your haproxy configuration on the server

To test your configuration, stop HAProxy using

sudo service haproxy stop

and run the following command to start HAProxy in debug mode:

sudo haproxy -d -f /etc/haproxy/haproxy.cfg

The “-d” flag is debug mode, and the “-f” flag is used to choose the config file. The typical output looks like:

Available polling systems :
 epoll : pref=300, test result OK
 poll : pref=200, test result OK
 select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Using epoll() as the polling mechanism.

If you don’t get any errors, then your configuration is OK. Press ctrl+c to close this foreground version of HAProxy, and start the HAProxy service:

sudo service haproxy start
Test your tunnel from your client

To test your client, you can use OpenSSL. The following command will connect to the server.

openssl s_client -connect ssh.example.com:443

You can also use your IP address. This will connect to HAProxy, and will be interpreted as ssh, if your configuration is correct. Once it reaches the ssh server, you’re good! You’ll see lots of stuff from OpenSSL, and finally a few seconds later the following message will appear if you’re using a Debian server:

SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u2

The message will change depending on your server’s linux distribution and OpenSSH server version. Once you see this message, this shows that you reached your ssh server successfully. You now have to setup a connection to your server’s tunnel.

Connecting to the ssh server using the tunnel

You need socat to connect to the https tunnel, and then you ssh to that tunnel. The program, socat, can be downloaded either as a zip package (please google it and try it, if it works, great. I had problem with OpenSSL dlls back then when I first tried this). Or you can use Cygwin to get it. Cygwin is a set of linux programs compiled for Windows. Don’t get too confident in the installer and just download all its components or you’ll easily consume 30 GB of diskspace and consume 10 hours installing these components. Just download what you need.

In case you’re using a linux client, socat is a standard program in linux. Just install it with your default package manager, e.g. in Debian/Ubuntu

sudo apt-get install socat
Running the socat tunnel

Open your Windows command prompt as administrator (or linux terminal), and use the following command to connect to your server using socat

socat -d TCP-LISTEN:8888,fork,range=127.0.0.1/32 OPENSSL-CONNECT:ssh.example.com:443,verify=0

Here we use port 8888 as an intermediate local port on your client. Once this works with no errors, you’re good to use an ssh client.

Warning: A “-v” flag is verbose for socat. Don’t do this for serious connections, but only for tests, as it writes everything on the terminal where socat is running, and since Windows Command Prompt prints messages synchronously, it’ll slow down everything for you.

Connect with your ssh client

Assuming you’re using putty, this is how your client should look like

PuttySSHTunnel

Or if you’re using a linux client, simply use this in your terminal

ssh 127.0.0.1 -p 443

And you should connect, and you’re done! Congratulations! You’re connected to your ssh server through https.

What about the other ports, other than ssh?

Once you got ssh working, everything else is easy. You can use an ssh SOCKS proxy tunnel. Putty does this easily for you. All you have to do is configure your connection as in the picture:

PuttySOCKS

This creates a SOCKS proxy. To use it, I provide the following example that I do on Thunderbird to secure my e-mail connections. You can do the same on any program you like, even on your web browser:

SOCKSProxySettingsThunderbird

You can do the same on linux. Please google how to create an ssh tunnel on linux for SOCKS proxy.

Conclusion

You’re connected! You can bypass any firewall you want just by being given access port 443. The only way to stop you is by cutting off your internet completely 🙂

Share this if you like it! I welcome your questions and any other thoughts in the comments.