This is currently what I'm using for Maokai. With this build I am able to dominate my lane with great harass and initiate (but not sustain as full tank) team fights.

I use flat magic pen, mp5 and ability power.

Why do I use these? because it allows me harass my lane early and the extra ability power carries through for when you make it to 18 and are building more tank style items.

9/9/12

I used these for a Ryze build I saw on this site a long time ago, these masteries work very well because the extra mr and armor plus the mana regen and magic penetration all compliment for lane sustainability. The more harass you get off, the more times you can proc your passive.

Now most items are interchangeable but the most important thing you start with is a mana crystal and 2 health pots, or if your afraid of a jungler you can get a ward. I usually go with the pots. The rest of the build is strait forward, Tanky burst is what you get with rod of ages and rylai's. I sometimes get the sorcerers boots instead of the mercury treads if their team doesn't have much cc. I Get the glacial shroud because buy this time you need more mana and the armor is great, the cd is also nice. Also you can change out the abyssal if you need to be more tanky, I just like it because when your team engage in a team fight and you pop that ultimate enemy's either focusing you or team mates with in it are all in the perfect range to get that negative to their magic resist. Then again if you don't have a lot of burst then getting it for your own damage isn't too wise.

I start off with sapling toss, its your main harass and also your main farming tool. Don't use it to farm until about lvl 6 because around there you can toss it at the caster creeps in the back, the impact damage plus the explosion will kill those creeps. Now for early harass, if your laning against say a Mundo who sits in your bushes throwing cleavers, one sapling in their will make him think twice about trying to harass heavy, now if say you have a ranged dps or a caster laning with you this is where Twisted Advance shines. If someone is kind of low and not porting back your opportunity has arrived. Sitting in the bushes and making sure Twisted Advance is in range, I start with the sapling toss at my target and immediately pop Twisted Advance snaring them in place for when your sapling blows. I was laning with a Ryze and we became quite the team, sapling/snare plus Ryze snare/nuke. Also at all costs DON'T tower dive, even if you think you can get the kill. You can sustain pretty well will your passive but your not a full on tank. Not that I'm advocating tower diving, I'm just saying being a hybrid you have to know your place.
Now on to team fights, Maokai is pretty damn good with his cc, but he can't sustain 5 enemy's focusing him for too long. Most people forget that team mates within your ult take significantly reduced damage. this is where you shine, you can start it off with your sapping toss and then quickly snaring one of their squishies with Twisted Advance, slamming them with Arcane Smash for the little extra stun and then popping your ult when the enemy team returns fire.
Using your cc and your burst to quell enemy damage out put, keep an eye on your health. Your no good to your team dead. Witch is why I have flash to get out of enemy range but still remain near my ult. As for sequence of leveling my skills I start with sapling toss then twisted advance and then arcane smash, leveling sapling and arcane smash equally and leveling my ult when Its possible at 6, 11 and 16.

I use flash and teleport, you can interchange these with spells your more comfortable with especially with flash having such a terribad cool down. But these are what suit my style of play.

Remember your Tanky burst, Don't think your unkillable and be super ballsy. Also don't try to use your ult for damaging purposes in team fights, its mainly meant for absorbing damage not dealing it. Also for early game don't harass so much that you run your mana super low so you can't use abilities when it counts.