Slow forum & Gateway time-outs

Discuss technical problems and features here
Bluepaint
Blue Belt
Posts: 813
Joined: Fri Jul 03, 2015 7:41 pm
x 583

Slow forum & Gateway time-outs

Postby Bluepaint » Sat Oct 21, 2017 10:55 am

Have made admins aware and am awaiting them fixing this. It takes time as people aren't always immediately available. We will be as quick as we can!
1 x

User avatar
emk
Brown Belt
Posts: 1282
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 4060
Contact:

Re: Slow forum & Gateway time-outs

Postby emk » Sat Oct 21, 2017 12:21 pm

Rhian wrote:Have made admins aware and am awaiting them fixing this. It takes time as people aren't always immediately available. We will be as quick as we can!

Sigh, somebody ate all our CPU credits again:

llo-credits-depleting-again.png
llo-credits-depleting-again.png (134.65 KiB) Viewed 300 times


Before fixing this, I'm going to dig through our logs and try to find out what crawler or other system is using up all our CPU credits. This forum is not currently spec'ed for heavy downloads, hour after hour. I mean, it's nice that the search engines want to index us, but somebody's not behaving well.
2 x

User avatar
emk
Brown Belt
Posts: 1282
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 4060
Contact:

Re: Slow forum & Gateway time-outs

Postby emk » Sat Oct 21, 2017 12:27 pm

Oh, look, it's Yandex again (among others), despite the fact that I followed all of Yandex's instructions to ban their crawlers completely until I had time to try to configure them for slow crawling only. Not happy. Not cool, Yandex.
1 x

Bluepaint
Blue Belt
Posts: 813
Joined: Fri Jul 03, 2015 7:41 pm
x 583

Re: Slow forum & Gateway time-outs

Postby Bluepaint » Sat Oct 21, 2017 2:10 pm

Seems to be working again. Thanks to Emk! Cheers for giving up your Saturday morning!!
1 x

User avatar
Elenia
Black Belt - 1st Dan
Posts: 1583
Joined: Sun Jul 19, 2015 1:22 am
Location: London
Languages: English (N), Swedish (???), French (Massively Atrophied) German (lowly beginner, somehow learnt to read)


Finnish?!
Language Log: viewtopic.php?t=708
x 2369
Contact:

Re: Slow forum & Gateway time-outs

Postby Elenia » Sat Oct 21, 2017 2:11 pm

Thanks for getting on this so quickly, emk! Sorry that your hard work has all been undone.
0 x

User avatar
emk
Brown Belt
Posts: 1282
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 4060
Contact:

Re: Slow forum & Gateway time-outs

Postby emk » Sat Oct 21, 2017 2:14 pm

Rhian wrote:Seems to be working again. Thanks to Emk! Cheers for giving up your Saturday morning!!

It's only temporary, sadly. I've upgraded to a t2.small, which refreshed the CPU quota, and I'm trying to tell Googlebot to slow down using Google's webmaster tools. And I'm updating robots.txt to keep the dumber bots from hitting the signup page thousands of times per day, though that might be spammers.
1 x

Bluepaint
Blue Belt
Posts: 813
Joined: Fri Jul 03, 2015 7:41 pm
x 583

Re: Slow forum & Gateway time-outs

Postby Bluepaint » Sat Oct 21, 2017 4:43 pm

emk wrote:
Rhian wrote:Seems to be working again. Thanks to Emk! Cheers for giving up your Saturday morning!!

It's only temporary, sadly. I've upgraded to a t2.small, which refreshed the CPU quota, and I'm trying to tell Googlebot to slow down using Google's webmaster tools. And I'm updating robots.txt to keep the dumber bots from hitting the signup page thousands of times per day, though that might be spammers.


Yandex bot is showing in active users atm
1 x

User avatar
emk
Brown Belt
Posts: 1282
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 4060
Contact:

Re: Slow forum & Gateway time-outs

Postby emk » Mon Oct 23, 2017 12:37 pm

I've upgraded to a t2.small, bringing our total costs to an estimated $40/month. And I've limited GoogleBot from downloading more than a page every 10 seconds. Once again, we see a similar pattern. Here's the new t2.small:

llo-t2-small.png
llo-t2-small.png (100.04 KiB) Viewed 100 times

For comparison, here's the old t2.micro again, with a total cost around $30/month, and more crawlers enabled:

Image

In both pictures, you can see that our CPU credits go up sharply after initial launch, then take a sudden hit. Once the "hit" is passed, then things drift for a while. The new instance has a much larger total "reserve", and it regains CPU credits twice as fast. And even when the credits are exhausted, the baseline performance is twice as high as before.

So far, we're holding steady: Our credit balance is remaining level, but it's not climbing back up towards its maximum. I'm crossing my fingers that this t2.small will be enough. We could avoid this whole "CPU credit" system by switching to an m3-class instance, where we get 100% power at all times.
1 x

Bluepaint
Blue Belt
Posts: 813
Joined: Fri Jul 03, 2015 7:41 pm
x 583

Re: Slow forum & Gateway time-outs

Postby Bluepaint » Mon Oct 23, 2017 12:39 pm

How much does an M3-class cost per month?

We might be fine now but I'd like to know, have you any idea?
0 x

User avatar
emk
Brown Belt
Posts: 1282
Joined: Sat Jul 18, 2015 12:07 pm
Location: Vermont, USA
Languages: English (N), French (B2+)
Badly neglected "just for fun" languages: Middle Egyptian, Spanish.
Language Log: viewtopic.php?f=15&t=723
x 4060
Contact:

Re: Slow forum & Gateway time-outs

Postby emk » Mon Oct 23, 2017 12:53 pm

Rhian wrote:How much does an M3-class cost per month?

We might be fine now but I'd like to know, have you any idea?

You can look at different server costs using this handy online tool. Here are the t2 and m3 type instances. m4 is also nice, but even the cheapest versions are fairly pricey. We don't need a lot. We probably want to avoid "reserved" instances unless we know that we won't need to upgrade.

For the t2-class instances, here are our options:

llo-cpu-credit-chart.png
llo-cpu-credit-chart.png (94.03 KiB) Viewed 90 times

Once you know what the server costs, you need to add in the other expenses. Amazon is reasonably priced, but they meter and bill for everything. This includes the database (and the automatic database backups), the bandwidth, the disk volume (and its backups), etc. Here's what the total costs looked like half way through the month, using a t2.micro:

llo-monthly-costs.png
llo-monthly-costs.png (235.5 KiB) Viewed 93 times


RDS is the managed database, which doesn't seem to need any upgrades right now. (We want a managed database, because that makes it far more difficult to screw up backups and accidentally lose the entire forum.) EC2 is the server and its disks. Route53 is the DNS server that translates "language-learners.org" into a numeric Internet address. Sometimes there's an extra dollar a month for CodePipeline (which does the builds), or an extra ~$12 once a year to renew the domain name. I did a whole bunch of extra work to get 100% of costs through Amazon and not spread them around.

The difference in costs between the t2.micro and the t2.small is about $8 or $9/month, which is how I estimated $40 after upgrading to the t2.small.
0 x


Return to “Technical Support and Feature Requests”

Who is online

Users browsing this forum: No registered users and 1 guest