Vayne works in top lane because of the age of the buisers, lol. Her poke and her passive (on w) stacking with botrk is basically a death sentence for anyone building tank. Same goes for Quin. But neither of them can handle a GP because of the range.
Basically Vayne top lane is something that can work well against a tank like a Garen or a Nasus. The reason is that her W - Silver Bolts deals up to 12% of an opponent's maximum health as true damage. If you combine that with a Blade of the Ruined King and a Lord Dominik's Regards, she becomes a very good anti-tank option. Generally the build that I like best in that situation looks something like this:
OR
But there is also a bruiser build I've seen that looks like this:
OR
OR
And here's the rune page you want to run right now, regardless of the build:
OR
But there is also a bruiser build I've seen that looks like this:
OR
OR
And here's the rune page you want to run right now, regardless of the build: