How come Vivillion gets variants, but Gastrodon is stuck at 2, while Arbok's dex entry is ignored?
Aside from possible hardware restrictions:
Arbok: On the meta level, it could be a case of the dex writers having a different idea from the design team. Remember, the designers do not come up with a Pokedex description nor stats/Abilities (they can suggest some ideas but ultimately it's up to the teams that are in charge of those aspects of a Pokemon). Now the sprites have slightly altered the hood pattern every so often as a nod to this, however I would not be surprised if this practice has now ended with them making a definite 3D model that'll last them years.
For an in-lore reason, well it's not wrong. There are variations, it's just not as many or extreme one may have thought.
Shellos & Gastrodon: Meta level, it was originally just one Pokemon and meant for Ruby & Sapphire. But then it got shelved, brought back for Diamond & Pearl, and I guess one designer heard about
allopatric speciation and thought would be neat to give this Gen 3 reject a little makeover via that concept. They were just satisfied with an east and west version. Thinking about it, they might have been a test to see if the concept could work and thus paved the way for more Pokemon with different forms (and eventually getting us regional variants). Now, there are plenty more they could do with them, just look at all the other wild looking seaslugs there are, but that's probably no where near GF's mind at the moment. Maybe for potential DP remakes...
In-lore, well the idea of allopatric speciation is that a group from a species if forced to adapt differently after having been separated from the whole. It's not the case of want but need, a species doesn't want to split into a different evolution chain if it can prevent it. In Sinnoh it just so happens that a group of Shellos & Gatrodon got separated between Mt. Coronet, were unable to rejoin, and so went on to live & reproduce and evolve differently. Other region's Shellos & Gastrodon are from one of these two origin groups and have not ever faced their own major separation from others of its species, thus only the two variations. If there are any more it's likely going to be a species that in an extreme environment like maybe deep in the ocean or alternate dimension (hey, this is Pokemon, that's more of a common occurrence than you think).
Vivillon: Meta level, XY was going to be the first they did a same-date global release on a system which recorded where you were in the world and so thought it would be neat if there was a Pokemon which looked different depending on where you live in the world. Butterflies are a wide spread species with many different wing patterns so they were by far the easiest to do this concept with. In addition the Global Link was, at the time, an ambitious idea of creating a central "interaction hub" and so had some place they could show all the trading going on, even making a special counter just for Vivillon to encourage the trade of it and its different wing patterns.
In-lore, it's just explained the pattern of their wings is based on climate and habitat. They're just a sensitive species with how their wings develop. Minerals, how much sun they get, temperature, humidity, plenty of factors you can come up with and go "well that's the reason why the wing pattern looks different from that other region". As for why their wing patterns change the way they do who knows, that's just a "genetics" handwave most likely.
I guess I’m overly attached to the idea of Pokémon being unique even when you first obtain them. EVs on their own with customisable natures make each individual Pokémon of a given species completely interchangeable at the point of capture/hatching (barring gender, ability, and shininess I guess). Having said that, it’s weird of me to worry so much about the numerical individuality of randomly generated fictional animals.
Honestly I think what LGPE did would be a better way of making Pokemon feel more unique from one another: different sizes & weights. Sadly I don't think we'd ever go deep into making Pokemon unique from one another (especially with the concept of Shiny sort of limiting the idea of giving Pokemon different shades of color like they did in the first Stadium game, unless they redid all the Shinys which looked only a slight different from their normal counterparts to allow this).
As for IVs making a Pokemon feel unique, only in a sense where you either don't notice it because the IVs are good or wondering why your Pokemon isn't hitting as hard/taking a hit that well/isn't fast enough because their IVs in those stats are bad. And if you only notice something when it's having a negative effect on gameplay, is it really something you'd want to keep? At least with EVs you can change them.
But it also shouldn't be denied that the amount of innovation seen in the series over the past 25 has been almost negligible compared to how the games industry has evolved in the meantime. To put it in perspective, Red and Green were released in Japan more than half a year before Super Mario 64. I think Abilities is the most significant gameplay change that has happened since then (and that happened 18 years ago, next week). Otherwise, it sort of feels like the "design freeze" for the Pokémon games occurred two and a half decades ago. Game Freak has showed a remarkable unwillingness to revise any of its core concepts since then, even as the capabilities of its hardware have risen exponentially, the games industry has changed dramatically, and players are beginning to have higher expectations too. I really wonder if it will keep going or if a break is coming soon.
Sounds like just the standard GF's "Winner's curse" or "If it ain't broke don't fix it" case that they've been suffering from in last decade.
Until one of the games actually fails to sell properly, doubt it. They've always been extremely adverse to innovation (hell it's a miracle gen 8 got free camera in the wild area lol).
The cynic in me also wants to point out something: we all say we want changes, but do we really?
I once suggested ways they could update the battle UI to make it at least a bit more flexible in actions you'd expect to be able to command your Pokemon plus some other features while keeping it a turn-based battle system. Everyone bit my head off about it and did not want to discuss it any further. And I can't help but think this is maybe WHY we haven't seen a major change to the UI and battle mechanics: we may ask for it but when we get it there's major backlash from the changes. Some people say they want the battles to be just like the anime, well they sort of tried having battles like this in the PokePark games which GF gave two whacks at. No one cared. Meanwhile they introduce a new "gimmick" like Mega Evolution, Z-Moves, and Dynamax and they see much more positive reception though they touched little else about the battle mechanics aside some new ideas that work within it.
While I would like to see changes (and went out of my way to think of ways which would add depth but not be too intrusive), I also don't blame GF for not having done it cause of the potential blowback.