What's going on with this forum?

[lichtrausch]

Is it time for a new forum?

No.

[iguanamon]

Yes, this forum started because of HTLAL going down for weeks at a time because of outages- with no response- at all. rdearman is active. emk hasn't been actively posting in a long while but is involved with the tech. I remember, but can't find the post, someone saying that we had customized our version of the forum software and this may have something to do with it.

Unlike HTLAL, no one pays to be a member here, there are no ads. Both rdearman and emk are volunteers who give of their time and money. Let's give them a chance to fix things... and please don't expect it to be overnight.

Yes, the 502/503 error codes are getting old. Copy your post before hitting "submit".

[rdearman]

We're investigating the issue. The error seem to be related to the Load Balancer which we use. This system is hosted on the Amazon Web Services and because of the amount of people using the services we have had to split the load across multiple virtual services.

How did we get here?
I used to host the system on a shared service, but it kept getting database errors (anyone remember those?) (oh and we got hacked too) so EMK and I migrated it over to Amazon web services and used the AWS database backend, which wasn't shared. So, problem solved for a little while.

Next we had even more users, and we started getting loading errors where people were waiting forever for the page to load, if it ever did. So we needed to add more virtual machines and then start using the AWS load balancer to spread the load across different virtual machines. Which seemed ok, but then we had issues with cookies across various servers not being carried. So we installed a proxy server as well.

Then the proxy server lead to all users getting deleted when an admin removed a banned user, and if you were in Bratislava with me, you will remember me trying to fix this on my mobile phone in a restaurant late at night. But we restored the DB and got it fixed and removed the glitch that deleted all users.

We've been running very successfully without a hitch for a while now, except the occasional problem with people getting logged out. (Which we still haven't figured out, but suspect it is the version of PhpBB we are on).

Now it seems that AWS have updated the load balancer system and we are on the "Classic Load Balancer" and it doesn't seem to be playing nicely with the rest of the Amazon services.

So what are we doing about this? Well the issue is that both EMK and I need to work on this together, we both work full time, and he is in the USA, and I'm in the UK. So we're trying to coordinate a time when both of us are available. (I am on-call and working this weekend so couldn't meet up, and he is on holiday with his family so isn't near his computer and the AWS account details)

But the plan is: We're going to schedule a few hours together, we need to do some account maintenance and change some AWS permissions and possibly swap the account. We then need to migrate to the latest load balancer configuration and update and upgrade all the server instances. We need to upgrade the proxy server software to the latest version. Then we need to put in the latest PhpBB software and upgrade it. Then we need to apply all the patches / mods and plugins required for the LLORG site, like language packs, virtual keyboard, love hearts for posts, etc, etc.

Then, assuming we haven't messed anything up, the site should be back up and running.

So what's so holy about this one?

Nothing holy about this one. Feel free to establish your own server and system. You'll get no objection from me, and you'll save me and the rest of the moderation team a load of money and hassle.

[zenmonkey]

We're investigating the issue.

Thanks for all you are doing. And a big thank you for taking the time to explain the status and next steps.

[jeff_lindqvist]

There's nothing holy about this forum. However, consider this: RDearman is here every day, he can be contacted, and he even responds. Those are important factors.

[rdearman]

If this was a problem I could throw money at to solve, I would have gladly thrown it. But unfortunately it isn't, and it is one that needs both myself and EMK.

[outcast]

We're investigating the issue. The error seem to be related to the Load Balancer which we use. This system is hosted on the Amazon Web Services and because of the amount of people using the services we have had to split the load across multiple virtual services.

How did we get here?
I used to host the system on a shared service, but it kept getting database errors (anyone remember those?) (oh and we got hacked too) so EMK and I migrated it over to Amazon web services and used the AWS database backend, which wasn't shared. So, problem solved for a little while.

Next we had even more users, and we started getting loading errors where people were waiting forever for the page to load, if it ever did. So we needed to add more virtual machines and then start using the AWS load balancer to spread the load across different virtual machines. Which seemed ok, but then we had issues with cookies across various servers not being carried. So we installed a proxy server as well.

Then the proxy server lead to all users getting deleted when an admin removed a banned user, and if you were in Bratislava with me, you will remember me trying to fix this on my mobile phone in a restaurant late at night. But we restored the DB and got it fixed and removed the glitch that deleted all users.

We've been running very successfully without a hitch for a while now, except the occasional problem with people getting logged out. (Which we still haven't figured out, but suspect it is the version of PhpBB we are on).

Now it seems that AWS have updated the load balancer system and we are on the "Classic Load Balancer" and it doesn't seem to be playing nicely with the rest of the Amazon services.

I know nothing about the science, software, hardware, and economics of an internet forum. But reading this above, it seems to me that when it comes to forum technology, the world is still in the equivalent of the Vacuum Tube and crude transistors era, controlled with a wall of metal switches and miles of cables strewn across a two story building. Some smart girl or guy out there surely can usher us to the Intel 4004 era, no?

Aweome job, rdearman.

[IronMike]

Hey, wanted to thank Rick & Co., because I haven't had one of those 508 or whatever-the-number errors in a long time. Cheers brother and thanks for all the hard work!

[rdearman]

Hey, wanted to thank Rick & Co., because I haven't had one of those 508 or whatever-the-number errors in a long time. Cheers brother and thanks for all the hard work!

I think this is more about the spammers working out eventually that if you bombard a website so much that nobody can use it, they can't read your spam either. At the moment we are ticking along ok. Although I believe I will have to host the super challenge bot and reporting on a separate server.