Struggling to handle load. Are we getting heavy traffic from somewhere?

Discuss technical problems and features here
t123
White Belt
Posts: 49
Joined: Sat Jul 18, 2015 1:48 pm
x 67

Re: ANNOUNCEMENT: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby t123 » Thu Nov 08, 2018 10:25 pm

Could be some high traffic site linking with a no-referrer policy. Or a hack attempt if there are SQL injection commands appended to the query string. Do you have the IP addresses, if so a reverse DNS on a few could help.
0 x

User avatar
Tristano
Blue Belt
Posts: 640
Joined: Mon Jul 20, 2015 7:11 am
Location: The Netherlands
Languages: Native: Italian
Speaks: English, Dutch, French, Spanish
Understands but not yet speaks: Romanian
Studies: German
Can't wait to put his hands on: Scandinavian languages, Slavic languages, Turkish, Arabic and other stuff
Language Log: viewtopic.php?f=15&t=5141
x 1015

Re: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby Tristano » Fri Nov 09, 2018 8:55 pm

I don't think it would help. If the site has many international users, you will see many different countries. If it is a hack attempt, it most likely go through proxies.
0 x

Stefan
Green Belt
Posts: 379
Joined: Sun Dec 20, 2015 9:59 pm
Location: Sweden
Languages: -
x 920
Contact:

Re: ANNOUNCEMENT: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby Stefan » Fri Nov 09, 2018 9:18 pm

rdearman wrote:It might be that we are simply falling victim to "The slashdot effect".

Doubt it because then it would all be on the same page. Reddit traffic has something like 90% bounce rate.
0 x

Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1619
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6318
Contact:

Re: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby emk » Sat Nov 10, 2018 11:27 am

There was a brief moment of downtime a few minutes ago while I downgraded the server from a t2.medium (which exceeds my monthly budget for this) to a t2.small. But I've configured the t2.small to run 3 copies of the forum instead of just 1, which should help. Here are the changes I made.

The graph that rdearman and I need to keep an eye on today is "CPU Credit Balance":

llo-cpu-credit-balance-2.png


We're not paying for a dedicated server, or even for a dedicated CPU in another server. Instead, we're paying for CPU "credits", which we "spend" during heavy load, and recharge slowly when nothing is happening. This allows us to have a relatively powerful server for $20/month, as long as we don't use all that power all the time.

But if that line ever reaches 0, then our server will slow to a crawl, and it will start returning assorted "50x" errors. Right now, it's looking OK—the line turned slightly upwards after I took the screenshot. So let's cross our fingers, and hope that 3 copies of the forum on the same server will be enough.
You do not have the required permissions to view the files attached to this post.
4 x

Online
User avatar
emk
Black Belt - 1st Dan
Posts: 1619
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 6318
Contact:

Re: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby emk » Sat Nov 10, 2018 12:12 pm

CPU credit balance seems to be recharging nicely right now:

llo-cpu-credit-balance-3.png


So it looks like a t2.small with 3 copies of the forum will be enough.

Here's what we're running, and approximately what it costs (rounded to the near $10):

  • t2.small for the forum ($20/month). We know that the next smaller size (t2.micro) runs of out CPU Credits every time a web crawler hits it, and so far it looks like we can avoid using a t2.medium if we're careful. Here's tool for comparing AWS server types.
  • db.t2.micro for the forum ($20/month). This is a completely managed database, with automated backups and recovery. It's usually under 5% load, and I've never seen it above 20%. So we have some headroom here. (See the chart below.) Basically this is a luxury that we pay for because manually hosting our own database guarantees that within any 5 year period, we're likely lose the database server, and only then realize that our backups have been broken for the last 6 months. But with an RDS db.t2.micro, Amazon handles everything. If our database server crashes, they'll allocate a new one, copy over all of our data, and apply the most recent backup, all without me lifting a finger.
  • An application load balancer ($20/month). This is essentially a "switchboard". It allows us to run multiple domain names (forum.language-learners.org, super-challenge.language-learners.org), and map each domain name to 1 or more processes running on 1 or more servers. It also handles HTTPS/TLS security certificates (the green lock in the browser that keeps bystanders from snooping on your traffic and stealing your password), and it renews them automatically as long as we keep paying our AWS bill. We tried running without this for a long time—we were basically running a custom load balancer directly on our t2.small—but that had certificate renewal problems that took down the forum and required me to panic (more than once!). The good news about the new load balancer is that it allowed me to quickly scale up from 1 copy of the forum to 4 copies when we were getting heavy load.
By the way, here's what the database load looked like over the last two weeks:

llo-rds-db-cpu.png


The graph on the left is more important. It looks like we've had two big midweek spikes, but only last week's was big enough to tip over the forum. It also looks like things have slowed back down.

For the moment, I think we're OK. Technically, we could certainly scale up quite a bit: We could use bigger servers, more servers, a bigger database, etc., with no problem, though obviously it would cost more. And we could upgrade everything in an hour or so if it comes to that. If I wanted to spend another weekend tweaking, we could even add servers automatically when the load got too high, and remove them when things slowed down again. So we're in good shape.

Organizationally, the moderators are working on setting up an entity that can take donations, and rdearman is getting a FIDO2 security key which will allow us to share more of the backend work.
You do not have the required permissions to view the files attached to this post.
14 x

User avatar
Serpent
Black Belt - 3rd Dan
Posts: 3657
Joined: Sat Jul 18, 2015 10:54 am
Location: Moskova
Languages: heritage
Russian (native); Belarusian, Polish

fluent or close: Finnish (certified C1), English; Portuguese, Spanish, German, Italian
learning: Croatian+, Ukrainian; Romanian, Galician; Danish, Swedish; Estonian
exploring: Latin, Karelian, Catalan, Dutch, Czech, Latvian
x 5179
Contact:

Re: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby Serpent » Mon Nov 12, 2018 1:15 pm

Is anyone else getting logged out more often than before?
1 x
LyricsTraining now has Finnish and Polish :)
Corrections welcome

User avatar
jeff_lindqvist
Black Belt - 3rd Dan
Posts: 3135
Joined: Sun Aug 16, 2015 9:52 pm
Languages: sv, en
de, es
ga, eo
---
fi, yue, ro, tp, cy, kw, pt, sk
Language Log: viewtopic.php?f=15&t=2773
x 10462

Re: Struggling to handle load. Are we getting heavy traffic from somewhere?

Postby jeff_lindqvist » Mon Nov 12, 2018 3:34 pm

More often now (I was kicked out after five minutes).
0 x
Leabhair/Greannáin léite as Gaeilge: 9 / 18
Ar an seastán oíche: Oileán an Órchiste
Duolingo - finished trees: sp/ga/de/fr/pt/it
Finnish with extra pain : 100 / 100

Llorg Blog - Wiki - Discord


Return to “Technical Support and Feature Requests”

Who is online

Users browsing this forum: No registered users and 2 guests