Also, remember that Vancouver is a starting port for Alaska Cruises. I can totally see tourists doing a trip to Victoria before or after an Alaskan Cruise.
A 2PM return seems weird. I know tourists who would go to Victoria for the day if it returned at 5pm. Heck, I know people who do day trips on BC Ferries, using public transit or a scheduled charter bus... but perhaps they will partner with another operator, who will offer either a flight back... or a bus back.
If you see how much money is being spent on the Rocky Mountaineer trains... $$$ you'll see people will pay for luxury. I AM disappointed that the Whistler Mountaineer was discontinued though. You can only take that train as part of a longer (and much more expensive) package.
Look at the great reviews for this 13-hour tour to Victoria:
https://www.viator.com/tours/Vancouv...s/d616-3914_3B
I'm guessing that's 7am to 8pm, with actual time on island being from 10:30ish to 5pm.