You'd need some pretty strong arguments when arguing for restricting other player's options - otherwise you're essentially saying that your preferences are more important than theirs.
I don't think "balance" is a valid argument - DM isn't supposed to be balanced in any way; the chaos is part of it's charm. For players that take this game a bit more seriously and want a fair fight, there is duel - but for a casual it would be pretty depressing to join and find it's filled with Swadian clones spazzing out greatswords.
"they're ruining it for everyone" would be a valid argument but there is no data for that - by definition, casuals won't be frequenting the forums and voting on polls. Not sure how many different players we have on arena per month, but it would be a lot more than the ones that voted in the poll. And if we're talking about the interest of the DM player base as a whole, casuals matter because they generally outnumber regulars by quite a bit. You'd also have to consider that even among regulars, a significant number prefer cav/archer - even if just for "speed travel" or other misc reasons like a class-specific item. Of course, such limits would also be an excellent opportunity for trolling as players will take these limited classes just to deny them to others.
Cav and archers are annoying af, but spears, shields and thrown items are pretty good defences. Not perfect; great lances and "tank" horses are pretty much unbeatable by an inf, regardless of what kind of spear you have - but most of the times, when you just equip the spear, cavs turn away - even apex predators prefer easy prey (ppl with no spear or not looking) When there are too many cavs, most maps have at least some locations where you can be safe from them or that can be defended. You can also build walls - and defend them; this can be barricades, or "living" walls - ad-hoc gangs of spearmen which make life hard for the cursed cavs - hopefully. It's like a fun meta-game. Not saying this as advice, of course - just another (casual) inf perspective.
Another argument against, mentioned by Anny (cav), and previously by BlackDeath (archer) - cav/archer is like an equalizer for players with bad ping; it evens the odds a bit.