What's going on with this forum?

Discuss the LLORG's and HTLAL forum's past and its future here.
lichtrausch
Blue Belt
Posts: 511
Joined: Thu Jul 23, 2015 3:21 pm
Languages: English (N), German, Japanese, Mandarin, Korean
x 1380

Re: What's going on with this forum?

Postby lichtrausch » Sat Feb 12, 2022 6:01 pm

TSS42 wrote:Is it time for a new forum?

No.
4 x

Beli Tsar
Green Belt
Posts: 384
Joined: Mon Oct 22, 2018 3:59 pm
Languages: English (N), Ancient Greek (intermediate reading), Latin (Beginner) Farsi (Beginner), Biblical Hebrew (Beginner)
Language Log: https://forum.language-learners.org/vie ... =15&t=9548
x 1294

Re: What's going on with this forum?

Postby Beli Tsar » Sat Feb 12, 2022 8:19 pm

lichtrausch wrote:
TSS42 wrote:Is it time for a new forum?

No.

I certainly hope not! But it has been unavailable most of the day for me too - the 503 error codes were back with a vengeance.
1 x
: 0 / 50 1/2 Super Challenge - Latin Reading
: 0 / 50 1/2 Super Challenge - Latin 'Films'

User avatar
iguanamon
Black Belt - 2nd Dan
Posts: 2352
Joined: Sat Jul 18, 2015 11:14 am
Location: Virgin Islands
Languages: Speaks: English (Native); Spanish (C2); Portuguese (C2); Haitian Creole (C1); Ladino/Djudeo-espanyol (C1); Lesser Antilles French Creole (B2)
Studies: Catalan
Language Log: viewtopic.php?t=797
x 14187

Re: What's going on with this forum?

Postby iguanamon » Sat Feb 12, 2022 9:36 pm

Yes, this forum started because of HTLAL going down for weeks at a time because of outages- with no response- at all. rdearman is active. emk hasn't been actively posting in a long while but is involved with the tech. I remember, but can't find the post, someone saying that we had customized our version of the forum software and this may have something to do with it.

Unlike HTLAL, no one pays to be a member here, there are no ads. Both rdearman and emk are volunteers who give of their time and money. Let's give them a chance to fix things... and please don't expect it to be overnight.

Yes, the 502/503 error codes are getting old. Copy your post before hitting "submit".
11 x

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23122
Contact:

Re: What's going on with this forum?

Postby rdearman » Sat Feb 12, 2022 11:44 pm

We're investigating the issue. The error seem to be related to the Load Balancer which we use. This system is hosted on the Amazon Web Services and because of the amount of people using the services we have had to split the load across multiple virtual services.

How did we get here?
I used to host the system on a shared service, but it kept getting database errors (anyone remember those?) (oh and we got hacked too) so EMK and I migrated it over to Amazon web services and used the AWS database backend, which wasn't shared. So, problem solved for a little while.

Next we had even more users, and we started getting loading errors where people were waiting forever for the page to load, if it ever did. So we needed to add more virtual machines and then start using the AWS load balancer to spread the load across different virtual machines. Which seemed ok, but then we had issues with cookies across various servers not being carried. So we installed a proxy server as well.

Then the proxy server lead to all users getting deleted when an admin removed a banned user, and if you were in Bratislava with me, you will remember me trying to fix this on my mobile phone in a restaurant late at night. But we restored the DB and got it fixed and removed the glitch that deleted all users.

We've been running very successfully without a hitch for a while now, except the occasional problem with people getting logged out. (Which we still haven't figured out, but suspect it is the version of PhpBB we are on).

Now it seems that AWS have updated the load balancer system and we are on the "Classic Load Balancer" and it doesn't seem to be playing nicely with the rest of the Amazon services.

So what are we doing about this? Well the issue is that both EMK and I need to work on this together, we both work full time, and he is in the USA, and I'm in the UK. So we're trying to coordinate a time when both of us are available. (I am on-call and working this weekend so couldn't meet up, and he is on holiday with his family so isn't near his computer and the AWS account details)

But the plan is: We're going to schedule a few hours together, we need to do some account maintenance and change some AWS permissions and possibly swap the account. We then need to migrate to the latest load balancer configuration and update and upgrade all the server instances. We need to upgrade the proxy server software to the latest version. Then we need to put in the latest PhpBB software and upgrade it. Then we need to apply all the patches / mods and plugins required for the LLORG site, like language packs, virtual keyboard, love hearts for posts, etc, etc.

Then, assuming we haven't messed anything up, the site should be back up and running.

TSS42 wrote:So what's so holy about this one?

Nothing holy about this one. Feel free to establish your own server and system. You'll get no objection from me, and you'll save me and the rest of the moderation team a load of money and hassle. :)
25 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

Beli Tsar
Green Belt
Posts: 384
Joined: Mon Oct 22, 2018 3:59 pm
Languages: English (N), Ancient Greek (intermediate reading), Latin (Beginner) Farsi (Beginner), Biblical Hebrew (Beginner)
Language Log: https://forum.language-learners.org/vie ... =15&t=9548
x 1294

Re: What's going on with this forum?

Postby Beli Tsar » Sun Feb 13, 2022 5:37 am

rdearman wrote:So what are we doing about this? Well the issue is that both EMK and I need to work on this together, we both work full time, and he is in the USA, and I'm in the UK. So we're trying to coordinate a time when both of us are available. (I am on-call and working this weekend so couldn't meet up, and he is on holiday with his family so isn't near his computer and the AWS account details)

But the plan is: We're going to schedule a few hours together, we need to do some account maintenance and change some AWS permissions and possibly swap the account. We then need to migrate to the latest load balancer configuration and update and upgrade all the server instances. We need to upgrade the proxy server software to the latest version. Then we need to put in the latest PhpBB software and upgrade it. Then we need to apply all the patches / mods and plugins required for the LLORG site, like language packs, virtual keyboard, love hearts for posts, etc, etc.

Then, assuming we haven't messed anything up, the site should be back up and running.

Thank you - both for that and for all the work you both do in general. You are very busy people, and that you keep this site up unpaid as well is amazing.

I - and doubtless many others - am very grateful for all that thankless work you do. I've learned so much on this site, and it's this kind of technical hard labour that makes that possible.
3 x
: 0 / 50 1/2 Super Challenge - Latin Reading
: 0 / 50 1/2 Super Challenge - Latin 'Films'

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: What's going on with this forum?

Postby zenmonkey » Sun Feb 13, 2022 6:00 am

rdearman wrote:We're investigating the issue.

Thanks for all you are doing. And a big thank you for taking the time to explain the status and next steps.
3 x
I am a leaf on the wind, watch how I soar

User avatar
jeff_lindqvist
Black Belt - 3rd Dan
Posts: 3135
Joined: Sun Aug 16, 2015 9:52 pm
Languages: sv, en
de, es
ga, eo
---
fi, yue, ro, tp, cy, kw, pt, sk
Language Log: viewtopic.php?f=15&t=2773
x 10462

Re: What's going on with this forum?

Postby jeff_lindqvist » Sun Feb 13, 2022 9:11 am

There's nothing holy about this forum. However, consider this: RDearman is here every day, he can be contacted, and he even responds. Those are important factors.
10 x
Leabhair/Greannáin léite as Gaeilge: 9 / 18
Ar an seastán oíche: Oileán an Órchiste
Duolingo - finished trees: sp/ga/de/fr/pt/it
Finnish with extra pain : 100 / 100

Llorg Blog - Wiki - Discord

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23122
Contact:

Re: What's going on with this forum?

Postby rdearman » Sun Feb 13, 2022 12:08 pm

If this was a problem I could throw money at to solve, I would have gladly thrown it. But unfortunately it isn't, and it is one that needs both myself and EMK. :(
5 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
outcast
Blue Belt
Posts: 585
Joined: Sat Dec 05, 2015 3:41 pm
Location: Florida, USA
Languages: ~
FLUENCY
Native: ENglish, ESpañol
Advanced: -
High Basic: DEutsch (rust), FRançais (rust), ZH中文
Basic: -
~
ACQUIRING
Formally: KO한국말, ITaliano, HI हिन्दी
Dabbling: HRvatski, GW粵語
Dormant: POrtuguês
~
Plan to learn: I BETTER NOT GO HERE FOR NOW
~
x 679

Re: What's going on with this forum?

Postby outcast » Mon Feb 14, 2022 2:19 am

rdearman wrote:We're investigating the issue. The error seem to be related to the Load Balancer which we use. This system is hosted on the Amazon Web Services and because of the amount of people using the services we have had to split the load across multiple virtual services.

How did we get here?
I used to host the system on a shared service, but it kept getting database errors (anyone remember those?) (oh and we got hacked too) so EMK and I migrated it over to Amazon web services and used the AWS database backend, which wasn't shared. So, problem solved for a little while.

Next we had even more users, and we started getting loading errors where people were waiting forever for the page to load, if it ever did. So we needed to add more virtual machines and then start using the AWS load balancer to spread the load across different virtual machines. Which seemed ok, but then we had issues with cookies across various servers not being carried. So we installed a proxy server as well.

Then the proxy server lead to all users getting deleted when an admin removed a banned user, and if you were in Bratislava with me, you will remember me trying to fix this on my mobile phone in a restaurant late at night. But we restored the DB and got it fixed and removed the glitch that deleted all users.

We've been running very successfully without a hitch for a while now, except the occasional problem with people getting logged out. (Which we still haven't figured out, but suspect it is the version of PhpBB we are on).

Now it seems that AWS have updated the load balancer system and we are on the "Classic Load Balancer" and it doesn't seem to be playing nicely with the rest of the Amazon services.



I know nothing about the science, software, hardware, and economics of an internet forum. But reading this above, it seems to me that when it comes to forum technology, the world is still in the equivalent of the Vacuum Tube and crude transistors era, controlled with a wall of metal switches and miles of cables strewn across a two story building. Some smart girl or guy out there surely can usher us to the Intel 4004 era, no? :D

Aweome job, rdearman.
2 x
"I can speak wonderfully and clearly in zero languages, and can also fluently embarrass myself in half a dozen others."

The End of Language learning: 10 / 10000

User avatar
IronMike
Black Belt - 2nd Dan
Posts: 2554
Joined: Thu May 12, 2016 6:13 am
Location: Northern Virginia
Languages: Studying: Esperanto
Maintaining: nada
Tested:
BCS, 1+L/1+R (DLPT5, 2022)
Russian, 3/3 (DLPT5, 2022) 2+ (OPI, 2022)
German, 2L/1+R (DLPT5, 2021)
Italian, 1L/2R (DLPT IV, 2019)
Esperanto, C1 (KER skriba ekzameno, 2017)
Slovene, 2+L/3R (DLPT II in, yes, 1999)
Language Log: viewtopic.php?f=15&t=5189
x 7265
Contact:

Re: What's going on with this forum?

Postby IronMike » Sun Apr 24, 2022 11:46 am

Hey, wanted to thank Rick & Co., because I haven't had one of those 508 or whatever-the-number errors in a long time. Cheers brother and thanks for all the hard work!
8 x
You're not a C1 (or B1 or whatever) if you haven't tested.
CEFR --> ILR/DLPT equivalencies
My swimming life.
My reading life.


Return to “LLORG & HTLAL discussion area”

Who is online

Users browsing this forum: No registered users and 2 guests