Do I misunderstand? You said you wanted to keep everything below 10-12 stories, which would mean undoing the city's modern era and rolling everything back to 1960 or thereabouts.
Consider this: since ~1960 Victoria's built form has changed the least by far of any Canadian city west of Ontario. Roll Vancouver or Calgary or Edmonton back to before the modern highrise era and they would be unrecognizable places. Meanwhile, a city like Kelowna is just now embarking on ambitious elaboration of its modern form. And yet people want to say Victoria is the place that should be rolled back? It doesn't make any sense because Victoria is the place that has exhibited the least dramatic change. Ongoing development and redevelopment that fits and that works has been Victoria' bread and butter. The proof is in the pudding: the city has built all sorts of good new stuff without trading off its heart and soul.
Seriously, almost 60 years of ongoing development and redevelopment, and yet it's still very much the same place it was when the process started (the only really drastic changes being the erasure of several prominent historic buildings that were unfortunately replaced with bland and rather irrelevant stuff -- although you need to go back all the way to the late 1980s for the most recent misstep of this nature). If you don't believe it, just look at aerials of the city core today and compare them with aerials from decades ago. You have to scrutinize the pics to identify the new construction, even though the new construction has supposedly overwhelmed and ruined the place. If you can hardly see it then how the heck can it have overwhelmed and ruined the place? In fact, the impact has been very subtle.
This is what I mean when I say people need to relax. Literally every day since the first highrise building went up all those decades ago we've been hearing people say it's all over, the city is being ruined. So here we are 60 years later and we're still hearing people say it's all over, the city is being ruined.
"I moved to Victoria in 2015 and I'm upset because the city is being ruined."
"I moved to Victoria in 2006 and I'm upset because the city is being ruined."
"I moved to Victoria in 1998 and I'm upset because the city is being ruined."
"I moved to Victoria in 1989 and I'm upset because the city is being ruined."
etc.
I have no doubt whatsoever that people will still be saying it 25 years from now. But a place simply cannot be on the precipice of ruination for such an extended period. It's impossible. Victoria has so, so many examples of things that were supposed to be the straw that would break the city's back, only to become just one more piece of the fabric.
That building was controversial? Everybody loves that building now.
Thus, we can be 100% certain that no extreme change of course is necessary. The long-established process is working. Victoria slowly but surely maintains and/or enhances its uniqueness with every passing day. Lest we forget, in terms of what you're allowed to do re: development and redevelopment, Victoria is without doubt the least extreme place in western Canada, if not in the entire country. You still have to prepare for a multi-year battle if you want to build a 4-story apartment building or some townhouses or even a simple house in some instances.
Worrying about Victoria in this regard is like worrying about the health of the only person in the room who exercises and eats right and doesn't drink or smoke. It's silly. Victoria made its big mistakes a long time ago and ultimately developed the formula that has been working very well for many years, and particularly since the late 1990s. Stay the course. Let the city be what it is. There's no need to make hasty changes.