If I wish to explore defensive items more deeply, I'd think: general-purpose, armour or magic resistance?
Generally, Guardian Angel and Moonflair Spellblade are what I think of as the most effective defensive items on Jax when used alone as defensive items. I think Guardian Angel's passive is better than the 50 ability power and plus 10 magic resistance and reduced cc passive combined, but now that I've listed them I think I might be wrong. Anyway, it hardly matters because Moonfliar is only available at Twisted Treeline and Crystal Scar. So: buy Guardian on Summoner's Rift and Howling Abyss.
If it's armour, I'd pick Randuin's Omen. Maybe. If not, I probably wouldn't. Maybe I would if I were facing a team reliant on attack speed or something. My impression is it's overated on Jax very much. The main reason I'd pick it if against high physical threats is because of lack of other items, I think.
Magic resistance? Maw of Malmortius. It's a good item, me thinks, against wizards (mages) as a carry because I think it captures the balance between offense and defense pretty well. And it adds to the scaling-the-more-I-attack properties of Rageblade and Relentless Assault.

If I decide to build very heavy defense on Jax against a team with varied damage types, I'd probably think like this: Health, armour, magic resistance. Guardian Angel (primary defensive item on him) and Frozen Mallet are a nice combination, sort of, because they sort of fill all the defensive requirements and add a slow to go with Trinity Force's and an interesting twist (Gaurdian Angel) - interesting twists are almost always nice.
If, for some reason, I wanted three defensive items I'd be less sure about what to pick, but I've prepared two building scenario's: Frozen Mallet, Maw of Malmortius, Randuin's Omen. Health, armour, resistance. As some of you may know, the amount of armour and resistance one has increases protection against physical and magical damage respectively by adding a percentage of one's maximum health equal to that amount when experiencing damage by physical and magical damage respectively. That's why health is important. On the other hand, Jax isn't a tank, so he should focus on this only casually. Mallet: health, Maw: resistance, Omen: armour. The more effective scenario would likely be Guardian Angel, Frozen Mallet and Maw of Malmortius or Randuin's Omen depending on the threat. I prefer it to the other scanrio because it just seems more practical; Omen is good but it has a sort of specific boost what with 70 armour and an attack speed slow which makes it more appropriate for specific threats, Maw of Malmortius is very good but has the same problem. Mallet and Guardian are the only stable items on Jax because they're versatile and have a passive which helps him in one way or another. Moonfliar Spellblade replaces Mallet in Crystal Scar and Treeline because the cc reduction is one of the things my normal build lacks and this section is all about what the builds above lack. Anyway, heavy defense on Jax is stupid so it could hurt your head to think about it too much. My usual builds have a standard form of defense for all carries: lifesteal and spellvamp.

As a side note, I've noticed Rageblade has become unpopular on Jax. One reason I've been able to salvage about this mysterious statistic is that it's because Jax won't be able to engage in long-term trades because people will be wary of this. My opinion is that if you don't manage to play to your strengths, you might not deserve to play the champion. Also, two things: Rageblade's first passive and Relentless Assault. These two combined should make the speed of the scaling in your damage output pretty quick, in addition to your stun which allows a few extra hits, give or take a few.