Diagnostics and Maintenance

Discuss technical problems and features here
User avatar
MorkTheFiddle
Black Belt - 2nd Dan
Posts: 2114
Joined: Sat Jul 18, 2015 8:59 pm
Location: North Texas USA
Languages: English (N). Read (only) French and Spanish. Studying Ancient Greek. Studying a bit of Latin. Once studied Old Norse. Dabbled in Catalan, Provençal and Italian.
Language Log: https://forum.language-learners.org/vie ... 11#p133911
x 4824

Re: Diagnostics and Maintenance

Postby MorkTheFiddle » Tue Feb 15, 2022 6:23 pm

I've bookmarked the HTLAL threads on language learning we've noted here so I can hop over there and read them--cause they are long enough--for the times when 503s bog us down here. I'm aware of the irony.
7 x
Many things which are false are transmitted from book to book, and gain credit in the world. -- attributed to Samuel Johnson

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Diagnostics and Maintenance

Postby rdearman » Sat Feb 19, 2022 10:33 pm

Good News! We think we have found the problem! It seems we were experience a DDOS (denial of service attack) from web-crawlers. EMK (super detective) discovered the issue and the crawlers were causing CPU burst failures. We believe the 502/503 errors should now reduce to nothing. We've also banned the offending web crawlers.

Next set is some upgrades, etc. We apologise for the time it took to find the problems.
11 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
IronMike
Black Belt - 2nd Dan
Posts: 2554
Joined: Thu May 12, 2016 6:13 am
Location: Northern Virginia
Languages: Studying: Esperanto
Maintaining: nada
Tested:
BCS, 1+L/1+R (DLPT5, 2022)
Russian, 3/3 (DLPT5, 2022) 2+ (OPI, 2022)
German, 2L/1+R (DLPT5, 2021)
Italian, 1L/2R (DLPT IV, 2019)
Esperanto, C1 (KER skriba ekzameno, 2017)
Slovene, 2+L/3R (DLPT II in, yes, 1999)
Language Log: viewtopic.php?f=15&t=5189
x 7265
Contact:

Re: Diagnostics and Maintenance

Postby IronMike » Sat Feb 19, 2022 11:46 pm

Thanks Rich. Still getting them, but am sure that they'll lessen in the coming days.
0 x
You're not a C1 (or B1 or whatever) if you haven't tested.
CEFR --> ILR/DLPT equivalencies
My swimming life.
My reading life.

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Diagnostics and Maintenance

Postby rdearman » Sun Feb 20, 2022 12:08 am

Bit of a technical explanation, but here it goes. In order to save money on the cost of running the service on AWS, we use a server called a T2 server, which is very small and has a very limited amount of "CPU credits". Each request on the service burns up a tiny bit of the CPU credits. This isn't normally a problem, since this website doesn't get enough traffic to really put a dent in the CPU credits which we accumulate.

On average, we'd expect to be running with about 125 CPU credits at any given time. However, if you run out of CPU credits, then AWS momentarily halts your CPU. This means you get a 503 out of service error. Normal usage would be 5-10 people on site would not even us 1 or 2 CPU credit, which is why normally we are at the top of the CPU credit range 125-150 credits.

It seems, however, Yandex web-crawler decided not to play nicely and swamped the site requesting page after page and burning through our credit. Each time we built up a little credit, the crawler came back and ate it all up again. I've banned Yandex and about 25 other web-crawlers from the site until we can build up a little credit. This will probably take about12-24 hours for us to get back up to a level where users will not see a problem. I'll probably not turn on any web-crawlers other than Google or Bing for another week or two in order to build up some credit.

In a previous thread, I said that this wasn't an issue that could be solved by throwing money at it. It seems I was incorrect, and it is an issue what can be solved by throwing money and purchasing a higher model, like a t3 or t2.unlimited. However, there is a pretty significant uplift in cost which personal budgets might not allow. Therefore, it might be in future that I may take you up on the generous offers to donate via a PayPal button, but for the moment we'll continue to foot the bill and see monitor the web-crawlers and the CPU credits.

I'm also hoping to do some software upgrades shortly.
15 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
luke
Brown Belt
Posts: 1243
Joined: Fri Aug 07, 2015 9:09 pm
Languages: English (N). Spanish (intermediate), Esperanto (B1), French (intermediate but rusting)
Language Log: https://forum.language-learners.org/vie ... 15&t=16948
x 3631

Re: Diagnostics and Maintenance

Postby luke » Sun Feb 20, 2022 12:27 am

TSS42 wrote:Why not permanently ban all web crawlers except Google and Bing? Why only for a week?

Or get them to be our pay pals. :)
2 x
: 124 / 124 Cien años de soledad 20x
: 5479 / 5500 5500 pages - Reading
: 51 / 55 FSI Basic Spanish 3x
: 309 / 506 Camino a Macondo

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Diagnostics and Maintenance

Postby rdearman » Sun Feb 20, 2022 8:14 am

TSS42 wrote:Why not permanently ban all web crawlers except Google and Bing? Why only for a week?

There are other web-crawlers which are used by a more international audience. For example, Baidu dominates the Chinese-speaking market. As well as DuckDuckGo which is a Google clone without the privacy concerns (it is the one I normally use).

Yandex dominates the Russian-speaking market, but it just doesn't play well with others. It ignores the robots.txt file (a file which tells crawlers how fast and how much they can take) and although this might be some glitch in their programming, they are going on to a permanent ban from crawling this site.

The reason we want various search engines to craw our site is that it helps people find us, and it lets us search the site without using the rather poorly implemented internal search engine on the site. :geek:
10 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.

User avatar
rdearman
Site Admin
Posts: 7231
Joined: Thu May 14, 2015 4:18 pm
Location: United Kingdom
Languages: English (N)
Language Log: viewtopic.php?f=15&t=1836
x 23127
Contact:

Re: Diagnostics and Maintenance

Postby rdearman » Sun Feb 20, 2022 6:11 pm

Well... That upgrade didn't go to plan. But the couple of hours outage did let us build up some CPU credits.
8 x
: 0 / 150 Read 150 books in 2024

My YouTube Channel
The Autodidactic Podcast
My Author's Newsletter

I post on this forum with mobile devices, so excuse short msgs and typos.


Return to “Technical Support and Feature Requests”

Who is online

Users browsing this forum: No registered users and 2 guests