Page 1 of 2

How's the new server performing?

Posted: Fri Apr 21, 2017 11:52 am
by emk
The forum has been moved from a shared PHP host to Amazon Web Services (AWS). The actual site is served up by two virtual machines:

  • A "t2.micro" serving the site. This is managed using ECS (the EC2 Container System) and a Docker container built from the source code on GitHub. We're probably going to automate building and deploying using AWS CodePipeline at some point, so we'll just be able to push to a GitHub branch and update the site's software.
  • A "db.t2.micro" hosting the database. This is a dedicated database VM managed and backed up by AWS.
The main "t2.micro" operates using a "burst" system. Like a cheetah, it can go very fast for a short while, and then it slows down. We get 6 CPU credits per hour, and a "reserve" of roughly 150 credits. If we use up all our hourly credits, we'll draw on the reserve. If the reserve goes to 0, then the site slows to 10% of 1 CPU. At the moment, it doesn't look like we're even close—we might want to downgrade to a t2.nano, which gets 3 credits per hour, and which only costs half as much. Anyway, here's what it looked like this morning:

llo-server-cpu-usage.png
llo-server-cpu-usage.png (37.64 KiB) Viewed 301 times

The database seems to be ticking along pretty nicely without too much load, at least for the moment:

llo-database-performance.png
llo-database-performance.png (48.16 KiB) Viewed 301 times

We're not using very much CPU—only about 5 to 10% of what we've been allocated. But I'm a little concerned by the IOPS. We currently have 100 IOPS, and it looks like we're already using a third of that. Again, there's a burst mode available, up to about 3000 IOPS, but once we use it up, we go back to 100.

In both cases, we can upgrade the underlying servers if we want to go faster (or downgrade to save money). It would take literally a couple of hours. But faster servers cost more money. You can play around with the various server sizes (look in the "Linux on Demand" column for the price) and database server sizes (look in the "MySQL on Demand" column for the price).

So, how's forum performance looking right now? Is everything reasonably fast and snappy? Do you ever see sudden, massive slowdowns, as if we exhausted our CPU credits or IOPS?

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 11:55 am
by Adrianslont
So far it's all fine for me.

Thanks for your efforts, guys.

Edit: better than before actually - this comment just posted without the usual lag I get when I make a post.

Edit: typo and to check the lightening speed again!

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 12:03 pm
by tommus
Based on a sample of two check-ins and reading several posts, I think the new Forum setup is working perfectly. Very fast. Thanks again to emk and rdearman for their stellar work on our behalf. I'm sure it will become a separate thread but we the membership need to step forward and financially bear the cost of running the Forum.

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 12:06 pm
by iguanamon
The speed is smoking fast here in the Caribbean right now.

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 2:42 pm
by emk
iguanamon wrote:The speed is smoking fast here in the Caribbean right now.

The server is currently running in one of Amazon's Virginia data centers, which have excellent connectivity, especially to the US and Europe. You might see a little more network latency in Asia, but not much—especially not compared to the lags introduced by the old database server.

We're now using only 3 IOPS, down from about 30 earlier. Basically the database is mostly asleep. We can handle 100 IOPS sustained, and 3000 for a brief burst. So performance should be great, except possibly during the daily backup. But backups are important!

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 3:45 pm
by rlnv
Thanks for taking the time to share the technical details of the new setup. Great choice in AWS. It certainly looks like you admins have taken performance seriously. Excellent speed where I sit. Thank you!

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 4:05 pm
by Arnaud
As fast as Speedy Gonzales :mrgreen:

Re: How's the new server performing?

Posted: Fri Apr 21, 2017 4:54 pm
by jeff_lindqvist
A lot faster than before. :)

Re: How's the new server performing?

Posted: Sat Apr 22, 2017 1:36 am
by smallwhite
Browsing noticeably faster then before. Australia. Thanks a lot.

Re: How's the new server performing?

Posted: Sat Apr 22, 2017 2:32 am
by emk
I'm seeing some bad performance at the moment, and I'm investigating. It doesn't seem to be the database or the server. Both have adequate CPU credits and IOPS in reserve. More when I figure out what's up, and my apologies!

I need to make sure that there isn't a database backup or something like that killing performance.

(Sigh. New servers are always a bit tricky and we set this one up in a hurry.)