Recent difficulties

Discuss technical problems and features here
User avatar
Iversen
Black Belt - 4th Dan
Posts: 4768
Joined: Sun Jul 19, 2015 7:36 pm
Location: Denmark
Languages: Monolingual travels in Danish, English, German, Dutch, Swedish, French, Portuguese, Spanish, Catalan, Italian, Romanian and (part time) Esperanto
Ahem, not yet: Norwegian, Afrikaans, Platt, Scots, Russian, Serbian, Bulgarian, Albanian, Greek, Latin, Irish, Indonesian and a few more...
Language Log: viewtopic.php?f=15&t=1027
x 14962

Re: Recent difficulties

Postby Iversen » Sat Mar 12, 2022 8:38 pm

One reason that I suspect the number of culprits is limited is that I didn't see any errors at all Thursday (or was it Friday?) until late in the night. If the bottlenecks are the result of the cumulative efforts of many sources why should the error codes then be absent one whole day, but then suddenly return with full force in the evening?

And yes, I have read a few things about those error codes, and I also know that you can't even be sure that one culprit = one IP code. But the types of attacks could be relevant - like rdearman's observation that there had been a lot of bogus applications (8000). Bots don't do that. I am not sure which filters there are, but I find it surprising that there aren't more bogus posts (and as a moderator I should have seen more than other members).

Luke's idea about accumulated email attempts is interesting, but I don't know enough about the system to know whether it also is the case - but if it is then my idea about one evil invader would of course be off the table.
0 x

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Recent difficulties

Postby rdearman » Sat Mar 12, 2022 10:03 pm

I'll try to answer all the questions with the information I have. The main problem at this moment in time is that I cannot actually login to the server to fix anything. My ssh key got wiped out a little while ago and EMK hasn't had the time to get me back on. (Which is why we want to switch it back to my account).

Iversen wrote: If the bottlenecks are the result of the cumulative efforts of many sources why should the error codes then be absent one whole day, but then suddenly return with full force in the evening?

When there is a lull in requests the server starts to build up CPU credits on the AWS site, and then it will function normally until these get depleted again.

luke wrote:Does the site have a feature that sends emails when a thread is updated?
Are some accounts "subscribed" to hundreds or thousands of threads, meaning they would generate a lot of email if it were working?
Could that the email isn't working be the bottleneck? I.E., PHPbb tries to connect to the email server to deliver the aforementioned "subscriptions", and the connection eventually times out (perhaps to be re-tried later). During the "waiting for the email server" connection (timeout), there aren't enough threads to handle normal PHPbb operations?

I'm game to help figure out what's going on. Solving performance issues is something I've got some experience with. I'm very interested in getting this one solved. If the servers are Linux, that's my thing.

I understand my little theories may be annoying, but it's the sort of stuff I do in the real world and I like tracking stuff down and figuring out what's going on to "make the world a better place" (tm).

The email feature is currently turned off because this version of the software doesn't support the encrypted connection. So there are no timeouts there, since it never attempts it. I have a nice docker instance of a Postfix server ready to take emails as soon as I can upgrade the phpBB software. The server is a Linux machine, but the actual websites are docker instances, which are built from the GitHub software. Previously, we had a hacker inject crap into the php, so emk setup the system so the docker would pull the version off GitHub, so we'd overwrite any malicious code injections. But, over time the terraform software moved on, but we didn't keep it up to date. So when I attempted to upgrade the site via the terraform system a little while ago it fell over in a big smoking heap.

I spent 20+ years as a unix system admin, so I completely understand the desire to dig around and fix stuff. :)

zenmonkey wrote:It likely is not a DDOS but rather forumbots in volume. These try to sign on and post spam. And likely what is happening is that this loads up the cookie directory on the server and regular requests then time out. It’s a reported issue with this version of the software. Turning off the site won’t solve it.

Probably upgrading will help and it may be necessary to install CloudFlare CDN or the AWS equivalent to balance traffic. Or simply, as EMK wrote, the instance needs to be larger. Setting up CloudFlare or upgrading the forum software or the AWS instance isn’t rocket science but it is dedicated work and time. And right now only one person has the access to do this. And since this is all voluntary work, and a lot of it, it takes time to get done.

Yes, this.
What needs to happen is:
  • Switch the account.
  • I pay for the addition of an ACL (Access Control List) on the load balancer, so I can throttle or block IP addresses which go over the data limits I configure. (e.g. Nasty web-crawlers and spammers)
  • Manually upgrade the software to the latest version (The newest version is already forked on to the GitHub and tested)
  • Point the email services to my postfix docker and get the emails going again.
  • Monitor the situation to see if extending the load balancer (less cost than a server upgrade) allows us to continue with the server model we're currently using.
11 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
Le Baron
Black Belt - 3rd Dan
Posts: 3511
Joined: Mon Jan 18, 2021 5:14 pm
Location: Koude kikkerland
Languages: English (N), fr, nl, de, eo, Sranantongo,
Maintaining: es, swahili.
Language Log: https://forum.language-learners.org/vie ... 15&t=18796
x 9390

Re: Recent difficulties

Postby Le Baron » Sun Mar 13, 2022 1:34 am

Do you think he's just pissing about to annoy everyone? Calm yourself man.
3 x

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Recent difficulties

Postby rdearman » Sun Mar 13, 2022 10:37 am

Simple answer is that emk is the technical genius behind a startup company worth millions of dollars and for the last six or eight months has been doing 16/18 hour days driving his company forward towards an IPO. Which means he barely even sees his family and I don't like putting pressure on him for what is in reality a hobby website.
13 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
iguanamon
Black Belt - 2nd Dan
Posts: 2354
Joined: Sat Jul 18, 2015 11:14 am
Location: Virgin Islands
Languages: Speaks: English (Native); Spanish (C2); Portuguese (C2); Haitian Creole (C1); Ladino/Djudeo-espanyol (C1); Lesser Antilles French Creole (B2)
Studies: Catalan
Language Log: viewtopic.php?t=797
x 14194

Re: Recent difficulties

Postby iguanamon » Sun Mar 13, 2022 2:09 pm

With HTLAL, FX never addressed any concerns about the forum. Both emk and rdearman have been communicative with us. It seems emk has moved on. That's life. I understand. Hopefully, soon, he will get time to pass control over to rdearman. The situation was more serious at HTLAL when the site went down for weeks at a time with no access at all and no communication with FX. FX charged for elite "pro" memberships. I actually paid for a pro membership stupidly thinking I was helping the forum. FX sold ads. His site was a for profit site. The image and perception was of someone who was collecting money and didn't care about keeping the forum up and running or improving the site. Perception is reality.

The membership was at the point of rebellion. The main fear people had was "losing the archive". We have several technologically-minded people who are members on the forum. Rdearman created the first version of the forum a couple of months before the exodus happened. Emk was a popular member who also had the technological chops to do something about the situation and he joined with rdearman to make LLORG happen.

We all moved over to the new forum fairly quickly in and around July, 2015... almost seven years ago. This forum was started with good intentions. There was to be a foundation set up to make sure that it would not be dependent upon one individual. The administrators were/are technically capable. To be fair, the 502/503 errors, while extremely annoying and discouraging to new and old members alike, have not shut down the site... yet. It is still usable for those who are determined to use it... but , obviously, this is a situation that is not tenable and which will not be tolerated for long.

I do not have either the technology skills, nor the money, nor the time, to design and operate a new forum that would keep our non-profit status and ideals alive. If someone wants to come up with and run a similar forum with similar status and ideals without technological problems, well, go ahead and start one. Time will tell if it becomes a viable alternative. My vote is to give our administrators time to resolve the issues rather than re-invent the wheel.

I do not think we have reached this point. I know what it's like to be so busy as to not have time for much else. I think both emk and rdearman have earned a lot of leeway and goodwill in these matters. If not for them, where would we be right now? How long this tolerance will last is an open question. If emk can find the time to make the moves rdearman suggests, it sounds like we will be able to return to stability and grow again. If this doesn't happen within a couple of months, then goodwill and leeway will begin to erode in the same manner as before and a crisis point may be reached where the forum will either survive in its current form or blow abandoned in the wind.

I think what this incident underlines and highlights is that no matter how well-intentioned, we have left ourselves vulnerable to the problems with having the forum administration and management in the hands of a limited amount of people. What happens if rdearman himself loses interest in the forum or becomes incapacitated in some way? The original intent when this forum was started was to eliminate this weak-point. This is a wake-up call to once again design a mechanism to make sure the forum is as independently strong and self-maintaining as possible. If emk is indeed leaving the forum, I wish him well, but I would like to see someone else take his place.

I'd also like to remind people that HTLAL is still up and running for the time-being and can be used to communicate about outages, if not for actual active discussion- our numbers would soon overwhelm the site if we tried to make it an alternative site for language discussion en masse. The twitter account designed to communicate with the membership appears to be no longer active.
14 x

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Recent difficulties

Postby rdearman » Sun Mar 13, 2022 6:36 pm

Let's take a breath. It isn't all that bad, and we did learn some lessons from HTLAL issues of the past. For one thing the DB is the forum, it is the one thing you need to recreate it is on AWS, and we've given the DB information to more than just EMK and me. The DB is backed up, and the login details have been distributed more widely, and the backups are also archived elsewhere. Which means another member of the moderation team could fire up a new version of the opensource and available phpBB and point it to the DB, and Bob's your uncle new site.

I could start a new instance of the system, now, but the problem is really I would have to find a different domain name. None of that is a concern if it was a permanent change of domain, one of the moderators could get someone like Luke to do it for them in a day or two.

This is why the optimum solution would be a charity organisation, without a reliance on individuals. If the charity owned the account and the domain, etc. then the charity can just hire people to switch things around. So I'll speak with the moderation team, and our charity consultant and see what we can do about making that option active again.

In the meanwhile, I have no intention of becoming dis-interested or incapacitated, so we'll get things switched over and sorted out soon. Honestly, I use the site more than almost anyone, and these things annoy me as much as they do you. :oops:
23 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: Recent difficulties

Postby zenmonkey » Sun Mar 13, 2022 7:36 pm

Well, part of the problem is also that the forum has been over engineered. You’ve now got a professional system in the background that has allowed for rapid redeployment, multi developer testing, instancing, etc. And that’s great when we where developing, testing and deploying add ons. I appreciated participating in that.

It’s solid enough to support dozens of developers pulling different codes, etc… except no one is doing that. And therefore the admin overhead results in this one person bottleneck.

Does EMK want to pass the reigns if he doesn’t have the time to manage these issues?
No judgment either way.
6 x
I am a leaf on the wind, watch how I soar

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: Recent difficulties

Postby zenmonkey » Sun Mar 13, 2022 7:42 pm

rdearman wrote:This is why the optimum solution would be a charity organisation, without a reliance on individuals. If the charity owned the account and the domain, etc. then the charity can just hire people to switch things around. So I'll speak with the moderation team, and our charity consultant and see what we can do about making that option active again.


When you have that discussion, consider that a US nonprofit does not need to declare to the IRS if revenue is below $5000.
3 x
I am a leaf on the wind, watch how I soar

User avatar
iguanamon
Black Belt - 2nd Dan
Posts: 2354
Joined: Sat Jul 18, 2015 11:14 am
Location: Virgin Islands
Languages: Speaks: English (Native); Spanish (C2); Portuguese (C2); Haitian Creole (C1); Ladino/Djudeo-espanyol (C1); Lesser Antilles French Creole (B2)
Studies: Catalan
Language Log: viewtopic.php?t=797
x 14194

Re: Recent difficulties

Postby iguanamon » Sun Mar 13, 2022 7:56 pm

rdearman wrote:Let's take a breath. It isn't all that bad, and we did learn some lessons from HTLAL issues of the past. For one thing the DB is the forum, it is the one thing you need to recreate it is on AWS, and we've given the DB information to more than just EMK and me. The DB is backed up, and the login details have been distributed more widely, and the backups are also archived elsewhere. Which means another member of the moderation team could fire up a new version of the opensource and available phpBB and point it to the DB, and Bob's your uncle new site. ... In the meanwhile, I have no intention of becoming dis-interested or incapacitated, so we'll get things switched over and sorted out soon. Honestly, I use the site more than almost anyone, and these things annoy me as much as they do you. :oops:

This is what FX never did in the last years of HTLAL, come to the forum and address the issues. This is how it differs from the HTLAL situation. Anyone who was an active member at HTLAL from 2010-2015 knows how bad the situation got over there which forced our move. We are nowhere near that point. As I said, both emk and rdearman have earned leeway and goodwill over the past 7 years. If it weren't for rdearman and emk taking the initiative, we wouldn't be here. Thank you, rdearman!
23 x

User avatar
zenmonkey
Black Belt - 2nd Dan
Posts: 2528
Joined: Sun Jul 26, 2015 7:21 pm
Location: California, Germany and France
Languages: Spanish, English, French trilingual - German (B2/C1) on/off study: Persian, Hebrew, Tibetan, Setswana.
Some knowledge of Italian, Portuguese, Ladino, Yiddish ...
Want to tackle Tzotzil, Nahuatl
Language Log: viewtopic.php?f=15&t=859
x 7030
Contact:

Re: Recent difficulties

Postby zenmonkey » Sun Mar 13, 2022 8:31 pm

Absolutely.
3 x
I am a leaf on the wind, watch how I soar


Return to “Technical Support and Feature Requests”

Who is online

Users browsing this forum: No registered users and 2 guests