I started playing Mal recently and he has quickly become my favorite mid/ap character. I am not a great player, but every time I play in PvP matches or vs AI (intermediate, even though that is still not saying much) I have always gone positive with near 0 deaths. Malzehar has large cool downs early on for his q, w, e skills and large mana costs.

One key thing to remember about magic penetration is that:

If 40% MPen: and the opponent has 100 magic resist
Your attacks will be affected by 100 - 100x.40 = 100-40 = 60 MResist

While if it gives +40 MPenn: and the opponent has 100 magic resist
Your attacks will be affected by 100 - 40 = 60MResist

MResist works in:
if someone has 50 MResist and 1000 health, it will take 1500 magic damage to kill the champion. 1 MResist = 1% more health

From the wikia:

25 magic resistance → ×0.8 incoming magic damage (20% reduction).
100 magic resistance → ×0.5 incoming magic damage (50% reduction).
−25 magic resistance → ×1.2 incoming magic damage (20% increase).

I have purchased very few runes, so am not an expert on it. But my guess for this build would involve Ability Power/Cooldown Reduction/ and Mana Regen runes.

I play a 21/0/9 split for masteries. I expect to use a lot of Ability Power and really enjoy Magic Penetration throughout my build (as you might have noticed already).

This is where my bread (ap) & butter (mana regen) appear.

Although I essentially only showed the major purchases, every time I return, I like to usually purchase something.

1) I start off with an early "Meki Pendant":
Malzehar has large mana costs early on and it is easy for a user to quickly run out of mana.

2) This is quickly followed by a "Tear of the Goddess":
Usually by this time, I have forced my opponent to at least recall from mid. This upgrade allows for a steady increase in mana throughout his levels and by mid game, Mal's early mana troubles are quickly cleared up.

3) Sorcerer's shoes + Archangel's Staff:
If your lucky, you have enough money for boots when you return for the Tear. You should at this point upgrade your shoes and, if you have enough money upgrade your Tear to Archangel's Staff. The staff grants great AP and is great combined with the MPen from the Sorcerer's shoes.

4) Void Staff:
This should be able to be purchased around mid game and grants a lot of AP as well as add 40% MPen. This is a deadly combo, especially mid game.

5) Needlessly Large Wand/ Blasting Wand:
Depending on how much gold you have when you are forced to return, you want to increase your Ability Power and start building toward your Deathcap.

6) Sheen or Deathcap?
This purchase also depends on how much gold you have. I personally spring for the cap and hopefully have enough to at least purchase a piece of Sheen.

7) Finishing sheen:
By this time, the match is usually close to ending, but in case the match extends, you want to finish the sheen and head towards the Lich Bane.

8) I have never been faced in a match where I would start building toward my last item. Depending on how their team is building, I am assuming you are doing enough damage to be considered a problem (so they should build some magic resist). I would build toward a Haunting Guise because of the magic penetration.

I personally focus his "e" early on.

The "Call of the Void" ability is nice due to the silence portion of the ability. It can be helpful if used properly. And later on can decimate minion waves. I feel it is too easy to be avoided, by enemy champions, to upgrade it multiple times early on. It can help with ganking (the silence part as well as if you lead them correctly), but I prefer the combination of his two abilities.
aka: if you notice if Fiddles, Nunu, Galio are charging their ult.

The "Null Zone" ability is a nice AOE effect. It's cooldown is slightly problematic though. Early game, it is great for harming creeper waves, especially in combination with the "Malefic Visions" ability. This combo is used for early harassment/farming.

"Malefic Visions" is my go to spell:
In the laning phase of the game, it can be used in combo with Null Zone to harass enemy champions. You cast this on the enemy wave (the fighters) and cast Null Zone on them, while attacking them. When you kill the minion, it moves on to another as well as giving you some mana. This repeats, unless you cannot kill the minion in time. If it can, it will jump to the magic minions and finally to the opposing champion for some harassment damage.

In late game, it is useful to pick up some kills while focusing on the team battle. If you cast it on champions low on health, they should die as they flee for health.

"Nether Grasp" is great for ganking.
It pretty much stuns all champions in one spot. It is useful for initiating or possibly catching an opponent Squishy while your allies converge on them. It is also helpful when you want to kill/target an enemy before they can do heavy burst damage. AKA a fed carry such as Cass or Ashe. Basically whoever you can get in range and out of position that is not tanky. Try not to use it on tanks because it leaves you vulnerable to the opposing DPS.

These are basic, the three I could see using are the two I mentioned (flash, ghost) as well as potentially teleport. Some might argue for ignite, but due to its smaller range, I feel it should only really be viable when you use your ult. I feel your too squishy to actually be close enough to use ignite, it still could be viable though.

Essentially I build him basically with MPenn and AP. I could see some other items being purchased, but I do not feel his attack speed is high enough for items like Malady that decreases magic resist based on stacks caused by basic attacks because his attack speed is slow. I only know what works for me and has worked for me, so you might disagree with this build or even provide suggestions for the runes sections or different parts of the build.

Please feel free to provide suggestions and I am only currently level 22, so I have not had much chance to participate in higher level matches. It has worked out really well for me in the past. The only part I struggle with Mal is his "Q" ability, so I personally use a lot of his "e" and "w". GLHF with my favorite character in the game.