stevegrant

Whisper it quietly, but I've done nothing to the setup for the last few weeks - it's just a weird fluctuation that I can't really explain. There's nothing that seems to actually cause the bad spells, it's not a spike in traffic or anything like that, there's no specific user action that I can see that's taking place, etc. Summary: dunno.

I had a load of email notifications this morning from a monitoring system telling me the site was up and down fairly regularly in the space of an hour or so. In all honesty, I've not really done much to have made it more stable over the last week or so, other than starting the re-indexing process which takes at least a week - it's currently 86% of the way through

Brentford in a similar situation too, having had a disappointing defeat yesterday at Birmingham. They were also pretty underwhelming - albeit with a weakened XI due to international call-ups - last week against Wycombe in the first round. I don't see it being a thriller

The weirdest thing is that the load balancer is effectively just a firewall at the moment because we only need one web server right now, so I would have thought traffic just automatically passes straight through, there's no logic for the load balancer to consider. Auto-scaling is in place if it's needed, as that's just a prudent way of setting things up so I don't need to manually change things when traffic increases significantly, as it does at half-time and full-time during games.

We're using an AWS MariaDB instance - not one of the most powerful ones, but it should be plenty for what this place does. In fact, the day/night of the Man City game during the restart when we set our new record for most people online simultaneously, we were actually on a smaller DB instance than we have now and it held firm. There's no real correlation between traffic and performance. I don't think I have the option to do that, but will have a poke around to see if it's possible.

Wonder if that might be as a result of the huge threads in that forum. In theory it shouldn't make any difference as it should only be looking at the topic itself, not any of the posts inside, but the way some of this site works under the hood, I wouldn't be surprised if there's some horrendous query that's slowing it down

Must admit I'm not 100% sure where I'd isolate any of those issues, tbh.

If Harrogate get through, the game will be at Doncaster anyway, Harrogate's new pitch won't be ready.

They didn't pick Ollie Watkins or Said Benrahma for today's first round game against Wycombe on the basis that they're expecting them both to leave so it's fairly pointless trying to work on a system when they're not likely to be a part of the team in the coming weeks. Anyone else who missed out today was on international duty.

phpBB still exists, but it's pretty basic these days. Absolutely zero appetite to move to another new platform, to be honest. We'd identified this one a few years ago but the issue was always having the time to do the migration. Their conversion process is not quick (it took three days running it on my fairly powerful Macbook rather than on a web server), especially as the old database was already around 3GB with more than 2m posts, so I don't really want to have to go through that sort of pain again I'm currently running the forum's re-indexing process, to see if something happened at one stage a while ago that screwed up the internal indexing. Worth noting that this is a code-based index rather than the indexing you'd ordinarily apply to the database itself, so god knows whether that'll make any difference or not, but that process was started on Thursday and is currently around 35% of the way through I'd say the last few days have been "stable" but not express pace. Looking at the latency numbers on the load balancer, it's averaging around 1 second, which is still too high really. But it's at least usable for now...

Third Round Bradford or Lincoln v Liverpool Bristol City or Northampton v Burton or Aston Villa Morecambe or Oldham v Newcastle or Blackburn Reading or Luton v Manchester United Derby or Preston v Brighton or Portsmouth Ipswich or Fulham v Rochdale or Sheffield Wednesday Manchester City v Bournemouth or Crystal Palace Millwall or Cheltenham v Burnley or Sheffield United Chelsea v Middlesbrough or Barnsley Wolves or Stoke v Gillingham or Coventry Leicester v Arsenal West Brom or Harrogate v SOUTHAMPTON or Brentford Newport or Cambridge v Oxford or Watford Fleetwood or Port Vale v Everton or Salford Leyton Orient or Plymouth v Tottenham West Ham or Charlton v Leeds or Hull

Second Round Burnley v Sheffield United Burton v Aston Villa Bradford v Lincoln Leeds v Hull Everton v Salford Wolves v Stoke Rochdale v Sheffield Wednesday West Brom v Harrogate Middlesbrough v Barnsley Derby v Preston Newcastle v Blackburn Morecambe v Oldham Fleetwood v Port Vale West Ham v Charlton Ipswich v Fulham Oxford v Watford Bournemouth v Crystal Palace Leyton Orient v Plymouth SOUTHAMPTON v Brentford Bristol City v Northampton Reading v Luton Millwall v Cheltenham Brighton v Portsmouth Newport v Cambridge Gillingham v Coventry

The reason for the lack of an update was basically because I was so fucking fed up of just reading a constant stream of whinging from people. One user who I attempted to help many years ago when he wanted a "fresh start" with a new username (but managed to out himself inside about 6 hours) and has been one of the more constant presences in this thread won't be troubling us again. The issue appears to be a particular inefficiency within the database (for those that understand this sort of thing, some of the key tables are denormalised and the indexing is a mess) - looking in depth at the structure of it, I'm amazed they've managed to make a viable business out of something that is basically a creaking pile of crap with a shiny skin that makes it look modern. There is a new version due out soon, so I'm hoping they've fixed some of these issues in that release, but it seems at the moment that we basically get three or four days of acceptable performance followed by a week of carnage. I've got some DB experts looking at it at the moment, but the concern is that if we make changes to the database itself (adding indexes, partitions on large tables, etc), that might have negative knock-on effects to other tables and big queries that are run often.

Kayne McLaggon scored a consolation for Barry Town United in their 5-1 Europa League qualifier defeat this evening against the mighty Faroese side FK Runavik.

To clarify, the 15% figure for non-league is for the first two weeks, then it goes up to 30%. There's also the slight caveat that the 15%/30% is based on the minimum required capacity for ground grading at that level of football. Taking Sholing as an example, they are in Step 4 of the National League System (i.e. the non-league pyramid, where the National League is Step 1), and the minimum capacity for grounds at that level is 1,350. 15% of that is 200, so that'll be their capacity for Stage 1 of the re-opening. Stage 2, from 31st August onwards, that capacity raises to 30%, so they'd then be allowed 400 in.

It can be today for you if you like? Getting pretty fucked off with the snide comments from people, to be honest. Absolutely nothing, that's what's so confusing. I've got an AWS specialist company looking at the database as I'm pretty sure the bottleneck is all there (the server CPU is constantly very low, so it's not a load issue), but ultimately the problem is actually likely to be in the software itself as there are a couple of queries that are probably run on every page load that are causing the issue. It'll just be a case of changing some settings in the database that will mitigate the issues until they're (hopefully) fixed in the next version of the forum software - that's supposedly due fairly soon.

They can take promotion and cup-related bonuses out of the equation, but the usual win and appearance fees would still be included, I think.

Pretty sure there is an instalment plan - it's just not being run by an external finance company where there could be potential legal issues with pro-rata refunds.

Playing around with some of the caching options - there are some really weird and seemingly inconsequential settings that, having looked at the database logs, might be doing a bit more than expected...

Couple of points: Firstly, as I've said before, it doesn't seem to actually be a case of resources - the CPU usage on the one single server we're currently using is barely getting above 30% at peak periods (we have auto-scaling in place to build extra servers to take the load if it goes above a 5-minute average of 40%). There seems to be occasional bottlenecks that don't correspond with traffic spikes; for example the evening after the Man City game we had 1160 people online at the same time and it coped with it without too many issues. I, and a few others I've spoken to, believe there are a few flaws in the way this software works, but the developers obviously aren't willing to accept that and are pretty unhelpful when it comes to trying to debug things, so we're largely left to our own devices. I've made another slight tweak this evening to the way emails (registrations, daily digests, etc) are sent out, because that was something I saw a huge problem with on another site, and I suspect that might be an issue here. Secondly, while your sentiments are noted, I must admit I don't particularly share them. The previous iteration of the site was largely left to run itself for a number of years which has seen a tailing-off (which was a mistake on my part but I simply didn't have the time to work on a full migration which the terrible world situation has afforded me this summer), but I've still not seen any other Saints site that has as many users as this one. It's not perfect, but to be honest I'm not losing any sleep at the prospect of a few people getting pissed off at some technical difficulties after a big migration.