Page 1 of 3

We're back online!

Posted: Sun Oct 28, 2018 3:15 am
by emk
Yay! We're back online. And my apologies for the outage.

What went wrong: This site uses HTTPS, which means that your friendly neighborhood script kiddie (or intelligence agency) can't read your password, private messages, etc., by snooping on network traffic. Well, probably the NSA can, but they won't admit it. But in order to secure your connection, this site needs to have a "certificate", which says that we're who we say we are.

Previously, we were getting our certificates from Let's Encrypt, a wonderful non-profit which has done more than anybody else to secure web traffic. They provide free certificates which renew automatically. We used a piece of software called docker-letsencrypt-nginx-proxy-companion to get our certificates from Let's Encrypt. Usually, this worked well—once or twice a year, either rdearman or I needed to reboot the server, but that was all.

But sometime early yesterday (US time), the certificate software fell over hard. I rebooted the server, and it didn't come back up. So I rebuilt the server from scratch (which takes like 10 minutes). But this meant that we upgraded to new versions of the OS and docker-letsencrypt-nginx-proxy-companion, and everything broke. I fixed three serious problems, but it still wasn't working, and I couldn't see why not. So I dumped docker-letsencrypt-nginx-proxy-companion and installed caddy-docker-proxy, which has like 2,000 fewer moving parts. Unfortunately, this also failed in several different mysterious ways, with no errors.

So I said "Arggghh! No more stupid Docker proxies that talk to Let's Encrypt! I'm going to cough up another $21/month (in addition to the $40/month it costs me) and pay for an Amazon Application Load Balancer with a certificate from AWS Certificate Manager!" Now, the downside of this is that it has like a dozen moving parts and it takes hours to set up correctly. The upside is that it's likely to keep working from now until the heat death of the universe, because once Amazon builds something like this, it just keeps running (and gets very slowly cheaper).

So, that finally worked.

The site is mostly back online. However, for now, you must connect using "https:" and not "http:". I'll work on setting up a redirect. I'll talk more about this soon, but for now I need sleep! Once again, my apologies for the unplanned outage.

(Also, while this was going on, I had to get rid of about a zillion leaves before an ice storm hit, and deal with a minor pluming leak.)

Anyway, I'll talk more about this later. But I just wanted to let people know what had been happening.

Re: We're back online!

Posted: Sun Oct 28, 2018 3:26 am
by emk
For the geeks in the audience, you can find all the configuration changes on GitHub. It's a giant pile of devops carnage, basically. I promise to add some more comments on the certificate stuff. But at least it's done, and certs should give us no more problems.

Oh, and the admins & moderators are working on longer-term plans to keep the forum online in the future, and to accept donations. But it's a long process with paperwork. We'll let everybody know when there's something to talk about!

Re: We're back online!

Posted: Sun Oct 28, 2018 3:33 am
by Ani
Thank you, thank you, thank you!!!

I'm so glad we're back. I missed you all!

Re: We're back online!

Posted: Sun Oct 28, 2018 3:52 am
by dgc1970
Thanks for all your hard work!

Re: We're back online!

Posted: Sun Oct 28, 2018 4:49 am
by arthaey
emk wrote:the admins & moderators are working on longer-term plans to…accept donations.

Thank you for volunteering both your time and money to the forum! I eagerly await a way to donate. :)

Re: We're back online!

Posted: Sun Oct 28, 2018 4:51 am
by PeterMollenburg
Thank you emk.... for the dishonarable mention of the NSA. It's okay, they're protecting national interests, so it's all very honest work they are doing, trust me ;) I sincerely do appreciate your hard work in keeping this forum working, seriously. Now, let me get back to my drug addition, porn addiction, sugar addiction, carb addiction, screen addiction, prescription drug addiction, alcohol addiction...NO NO NO, let me get back to MY LANGUAGE ADDICTION, dammit (this also includes collecting far too many language learning materials and renaming of files to get ready for more language learning) ! This is all in good humour :) (do I need to write a disclaimer to ensure I don't offend anyone? :? ) Many thanks emk, love your work! 8-) and the work of the mods and admins too!

Re: We're back online!

Posted: Sun Oct 28, 2018 5:45 am
by Teango
Thanks for all the time and effort you invest into this forum, emk, not only on this particular occasion, but on so many other occasions in the past. Merci mille fois! :)

Re: We're back online!

Posted: Sun Oct 28, 2018 6:58 am
by Skynet
EMK, thanks for the time, effort and money that you spend to ensure that we have somewhere FREE to congregate. I would be delighted to donate towards the continued existence of this noble cause.

Re: We're back online!

Posted: Sun Oct 28, 2018 11:06 am
by rdearman
On a side note, if you tried to post to the Super Challenge bot while the server was inoperative it didn't get it. The last message it got before the server came back online was at 10:21 GMT. So if you did try to post to the bot, check the website to see if it went through, and if it didn't you'll need to tweet it again for it to get picked up.

Re: We're back online!

Posted: Sun Oct 28, 2018 11:32 am
by rfnsoares
Thank you for the explanation, but you are speaking Greek to me :lol: