The Twisted Fork is good, especially given your downtown location.
In my opinion, most good brunch places aren't downtown. Red Wagon in East Vancouver is really good (
http://www.redwagoncafe.com) as is Jethro's in West Van (
http://www.jethrosfinegrub.com). I've heard nothing but great things about The Farmer's Apprentice's brunch. Never tried it myself.
Vancouver has always had a ton of all day breakfast/brunch spots, so it's hard for me to know what's new and which ones are still around.
Here's a link that'll get you started:
http://itstodiefor.ca/best-food-in-v...-in-vancouver/