Playing safe and getting trades in early is how you want to play. Proc your passive if you can but don't die for it. If you'll receive more damage than you can do just focus the minion wave.

Be careful when roll-engaging as Vayne because you won't have it to dodge cc. DO NOT ult to hide yourself if you're low if they have aoe's (area of effect) abilities i.e. Karma's Q because they can still hit you with it revealing your position.

Avoid getting cc and play for picks, the main reason Vayne can play top lane is she's good in soloing as well. Her max health damage makes it easy for her to take on 1 on 1's unless they aren't super tanky on top of damaging. These solo picks give her each unique champion stack for her hunter secondary rune. Keep in mind to use your active items that you should have late game because they're EXTREMELY useful in team fights if you're getting focused (which you should be because your the main damage source).