OSG site slow to respond?
- HNB
- PAT. # 2.972.923
- Posts: 13565
- Joined: Tue May 10, 2011 7:55 am
- Location: Puyallup, Washington
- Contact:
Re: OSG site slow to respond?
What about a return of some official OSG t-shirts with profit going directly back into the site? That way we get something cool to show of the site and any money can go into helping with upkeep? Then it is optional and also gives users something cool to show their love? OSG hats? OSG buttons or pins?
I know that is pretty commercial, but you can be non-profit (free to use and free to enjoy) and still sell merch if the money goes into the site.
I know that is pretty commercial, but you can be non-profit (free to use and free to enjoy) and still sell merch if the money goes into the site.
Christopher
Lilith Guitars
Lilith Guitars
- GilmourD
- PAT. # 2.972.923
- Posts: 1861
- Joined: Fri Apr 24, 2020 8:11 pm
- Location: Rutherford, NJ
Re: OSG site slow to respond?
How much RAM is allocated to the VM? I imagine you're running a Linux distro without a GUI, so it's pretty lean, but concurrent SQL queries will eat up RAM, I imagine. I mean... These days a regular desktop gets minimum 16 gigs in my mind. My home and work desktops both have 32 gigs. A Linux VM probably doesn't need that much, though.luau wrote: ↑Wed Aug 09, 2023 5:40 amI made a change that might affect people receiving email notifications. Please let me know if you aren't getting emails you're expecting.
It's a single VM with MySQL and Apache.GilmourD wrote: ↑Tue Aug 08, 2023 1:00 pmSame, and it's so inconsistent.
A question for luau: Is this a private machine that the server is running on? A shared machine? A VM? Is the database and the forum instance running on the same box?
I ask because I'm starting to wonder if the hangup is network or server resource based.
ATM, I think we just need more RAM but am still convincing myself of it.
When I became the SysAdmin at my current job (after the previous director was fired for doing nothing and lying to the superintendent that his pet project was completed when it wasn't even started) I discovered that one of our DOMAIN CONTROLLERS was configured as a VM with only FOUR GIGS of RAM. It was configured with AD, DHCP, and DNS, so it was getting hammered constantly.
- luau
- Admin
- Posts: 10045
- Joined: Tue Dec 12, 2006 7:07 am
- Location: Ft. Lauderdale, FL, USA
Re: OSG site slow to respond?
2GB. I realize it doesn't seem like a lot, but keep in mind it was perfectly adequate up until a week ago or whatever. Requirements for Windows machines with GUIs and Linux machines without are pretty different. I can't imagine trying to do anything with Windows with 4GB, not even Minesweeper.GilmourD wrote: ↑Wed Aug 09, 2023 7:11 amHow much RAM is allocated to the VM? I imagine you're running a Linux distro without a GUI, so it's pretty lean, but concurrent SQL queries will eat up RAM, I imagine. I mean... These days a regular desktop gets minimum 16 gigs in my mind. My home and work desktops both have 32 gigs. A Linux VM probably doesn't need that much, though.luau wrote: ↑Wed Aug 09, 2023 5:40 amI made a change that might affect people receiving email notifications. Please let me know if you aren't getting emails you're expecting.
It's a single VM with MySQL and Apache.GilmourD wrote: ↑Tue Aug 08, 2023 1:00 pm
Same, and it's so inconsistent.
A question for luau: Is this a private machine that the server is running on? A shared machine? A VM? Is the database and the forum instance running on the same box?
I ask because I'm starting to wonder if the hangup is network or server resource based.
ATM, I think we just need more RAM but am still convincing myself of it.
When I became the SysAdmin at my current job (after the previous director was fired for doing nothing and lying to the superintendent that his pet project was completed when it wasn't even started) I discovered that one of our DOMAIN CONTROLLERS was configured as a VM with only FOUR GIGS of RAM. It was configured with AD, DHCP, and DNS, so it was getting hammered constantly.
That said, I'm sure we could throw hardware at it, virtualized or no, and make this go away. But are we really serving that much more content to that many more people all of a sudden? We really aren't.
- GilmourD
- PAT. # 2.972.923
- Posts: 1861
- Joined: Fri Apr 24, 2020 8:11 pm
- Location: Rutherford, NJ
Re: OSG site slow to respond?
Well, that's why I was wondering what the setup was. Is there another VM on the same cluster that's sucking up resource time? Or is there maybe even a runaway process on the forum VM?
Honestly, without having my hands in the machine, I'm kinda just spitballing.
A "ping-t" doesn't even show anything but the occasional non-returned ping after like 10 minutes and the ping times are fairly consistent otherwise, so the server's ICMP is functioning correctly and consistently, but that does bypass DNS by going straight to IP.
Every time I see any delay in the waterfall it's in the SSL section of the resource load time, which is why I was initially thinking it's networking.
Interestingly, in trying to pull up a fresh waterfall I've refreshed the home page a whole bunch of times and haven't had any delays.
- luau
- Admin
- Posts: 10045
- Joined: Tue Dec 12, 2006 7:07 am
- Location: Ft. Lauderdale, FL, USA
Re: OSG site slow to respond?
According to Digital Ocean the hypervisor is fine. There's no way I can verify that but there's no real reason not to believe them.
There is no runaway process. I'm so tired of looking at atop and its logs I could die. The only real problem I can turn up is some occasional excess page swapping with atop which Digital Ocean doesn't even cover in their guide. From the atop man page:
So while RAM is an issue, it's hard to see how it's a big enough issue to cause what we're seeing. I'll take it up with Digital Ocean when I share my findings with them.If the number of pages swapped out ('swout' in the PAG-line) is larger than 10 per second, the memory resource is considered 'critical'. A value of at least 1 per second is considered 'almost critical'.
Whatever the case, the site is behaving very well ATM, so we've got that going for us, which is nice. And it's nice to chat about it with you, too, so thanks for taking the time.
edit:
- Embenny
- PAT. # 2.972.923
- Posts: 10363
- Joined: Tue May 24, 2016 5:07 am
Re: OSG site slow to respond?
I've had a solid hour of full speed OSG and I'm so happy.
I realized I've been checking the site much less often with the loading issues, after one or two 30-second timeouts I often give up and wander away.
I realized I've been checking the site much less often with the loading issues, after one or two 30-second timeouts I often give up and wander away.
The artist formerly known as mbene085.
- luau
- Admin
- Posts: 10045
- Joined: Tue Dec 12, 2006 7:07 am
- Location: Ft. Lauderdale, FL, USA
Re: OSG site slow to respond?
I do the exact same thing, but with the added layer of not wanting my requests to make things worse for everyone else.
I apologize to all of you for this taking so long. It's supremely frustrating.
- GilmourD
- PAT. # 2.972.923
- Posts: 1861
- Joined: Fri Apr 24, 2020 8:11 pm
- Location: Rutherford, NJ
Re: OSG site slow to respond?
Yeah, it's like something snapped back into place and it's been working just fine.luau wrote: ↑Wed Aug 09, 2023 11:16 amAccording to Digital Ocean the hypervisor is fine. There's no way I can verify that but there's no real reason not to believe them.
There is no runaway process. I'm so tired of looking at atop and its logs I could die. The only real problem I can turn up is some occasional excess page swapping with atop which Digital Ocean doesn't even cover in their guide. From the atop man page:So while RAM is an issue, it's hard to see how it's a big enough issue to cause what we're seeing. I'll take it up with Digital Ocean when I share my findings with them.If the number of pages swapped out ('swout' in the PAG-line) is larger than 10 per second, the memory resource is considered 'critical'. A value of at least 1 per second is considered 'almost critical'.
Whatever the case, the site is behaving very well ATM, so we've got that going for us, which is nice. And it's nice to chat about it with you, too, so thanks for taking the time.
edit:
And same! I'm so used to having to translate from tech-to-English in a school system that sometimes I forget how to talk straight tech. LOL
- GilmourD
- PAT. # 2.972.923
- Posts: 1861
- Joined: Fri Apr 24, 2020 8:11 pm
- Location: Rutherford, NJ
Re: OSG site slow to respond?
Oh, man. I think we jinxed it. Just posting my last reply had delays. D'oh!
- Embenny
- PAT. # 2.972.923
- Posts: 10363
- Joined: Tue May 24, 2016 5:07 am
Re: OSG site slow to respond?
I can't speak for everyone, but speaking for myself, you have nothing to apologize for. I appreciate all the voluntary effort you put in to keeping things running!
Shit happens. All you can do is try your best to deal with it.
The artist formerly known as mbene085.
- daemon
- PAT. # 2.972.923
- Posts: 836
- Joined: Tue Jan 03, 2012 6:44 pm
- Location: Cincinnati, Ohia
Re: OSG site slow to respond?
I'm amazed this thing can even run as well as it has on 2GB, but if it's been that way forever then it seems like something else has changed.
* Is someone applying updates? (apt-get dist-upgrade or dnf/yum update)
* Is there any monitoring available that could show trends? (snmp, or running atop as a service)
* Are there any clues in the logs? (/var/log/messages, dmesg, etc)
* Is someone applying updates? (apt-get dist-upgrade or dnf/yum update)
* Is there any monitoring available that could show trends? (snmp, or running atop as a service)
* Are there any clues in the logs? (/var/log/messages, dmesg, etc)
- DeathJag
- PAT. # 2.972.923
- Posts: 2297
- Joined: Wed Aug 08, 2018 11:44 am
- luau
- Admin
- Posts: 10045
- Joined: Tue Dec 12, 2006 7:07 am
- Location: Ft. Lauderdale, FL, USA
Re: OSG site slow to respond?
There were no updates applied around the time the trouble started but the machine is being updated. I do have atop running as a service now.daemon wrote: ↑Wed Aug 09, 2023 12:02 pmI'm amazed this thing can even run as well as it has on 2GB, but if it's been that way forever then it seems like something else has changed.
* Is someone applying updates? (apt-get dist-upgrade or dnf/yum update)
* Is there any monitoring available that could show trends? (snmp, or running atop as a service)
* Are there any clues in the logs? (/var/log/messages, dmesg, etc)
There was no monitoring running at the time and I've found nothing of interest in the system logs, web server log, or mysql log.
- MayTheFuzzBeWithYou
- PAT. # 2.972.923
- Posts: 2455
- Joined: Mon Jul 03, 2017 12:28 am
- Location: Linz, Austria
Re: OSG site slow to respond?
It appears to be veeeeery fast now.
I‘m happy that way. Please don‘t change it back!
I‘m happy that way. Please don‘t change it back!
- GilmourD
- PAT. # 2.972.923
- Posts: 1861
- Joined: Fri Apr 24, 2020 8:11 pm
- Location: Rutherford, NJ
Re: OSG site slow to respond?
MayTheFuzzBeWithYou wrote: ↑Thu Aug 10, 2023 3:03 pmIt appears to be veeeeery fast now.
I‘m happy that way. Please don‘t change it back!