Kog'Maw is certainly one of my favorite champions. I 'mained' him back when he was OP (Patch 6.13, I think?) but I still love pulling him out now and then: that delicious machine-gun attack speed, that item synergy with on-hits really not present on many other champions. And, of course, his late-game damage is absolutely insane: Level 18 Kog'maw with 6 items has without a doubt the highest DPS in the game, able to 100-0 3 champions simultaneously in seconds with his Bio-Arcane Barrage and Runaan's Hurricane; of course, he abuses the stacking mechanics on Fervor of Battle , Wit's End and Guinsoo's Rageblade by attacking so quickly as to stack them nearly instantly. So how could we build this champion to deal even more damage?

(DISCLAIMER: Kog'maw doesn't need to deal the maximum possible amount of damage. Similar to Vayne, he needs to itemize somewhat defensively because he has no inherent safety in his kit, and because ADCs can't deal damage while they're dead. My preferred build contains a Frozen Mallet, which applies a slow on his Runaan's bolts, but there exist other options.)

Well, we know that Nashor's Tooth is the only on-hit effect in the game that scales. What I mean by this is that unlike other on-hit effects which have a set value ( Blade of the Ruined King ALWAYS deals 6% of the target's max health, Wit's End ALWAYS deals 40 magic damage; these numbers can only be improved with penetration). If we could get infinite AP, we would also deal infinite on-hit damage. Sadly we cannot get infinite AP, but there are a number of AP-increasing buffs in the game, such as Crest of Insight, Zeke's Harbinger, Infernal Drake etc. Normally Kog'maw doesn't benefit from these at all, because they scale multiplicatively with existing AP and he does not usually build any. Furthermore, Kog doesn't benefit from AD, because his Bio-Arcane Barrage reduces his AD to 55%, making AD and crit useless for him.

So yes, in THEORY Kog'maw could stand to gain unlimited damage from a Nashor's Tooth and lots of AP, but this holds true for every other champion as well! Some champions, like Teemo and Kayle, use Nashor's to build AP on-hit builds, while most other champions do not. What makes these champions special is that they both have AP on-hit scalings in their kit (see Toxic Shot for an example): Nashor's Tooth is just a way of 'increasing' that scaling. So champions like Tristana with NO scaling have a total of 15% with Nashor's, while champions WITH scaling have a total of 40-45% with Nashor's. That's interesting... if Kog'maw had an AP scaling on his auto-attacks, this could be a good potential synergy!

But Kog'maw's Bio-Arcane Barrage DOES have an AP scaling: "While Bio-Arcane Barrage is active, Kog'maw deals a bonus 4/8/12/16/20 + [2 + (0.75 per 100 AP)]% of target's max health as bonus magic damage on-hit." That math can be a little bit confusing, especially when comparing it to a FLAT AP scaling like 30%. What it means, though, is:
-against a target with 100 max HP, 100 AP = 0.75 bonus damage;
-against a target with 2000 max HP, 100 AP = 15 bonus damage;
-against a target with 3500 max HP, 100 AP = 26.25 bonus damage.
So, one could say it has a 15% AP scaling against low-health targets and a 25+% AP scaling on high-health targets (at level 18, that is). That's somewhat inferior to a flat 30% on another champion, but since Kog'maw has double the attack speed during this duration, the 'real' value of this scaling is arguably higher.

It's quickly looking like AP Kog'maw has a higher damage potential than your standard Kog, and this is actually true if you work out all the math. Even without any buffs to our AP like Crest of Insight, the full 6 item build does considerably more damage. Here's the build:

Why build the Tooth so late? Because %max health damage is very poor early-game: as you saw with the math above, the scalings were just barely good enough against 2000 health targets. In the early game where champions have less health, the scaling is worse, and we benefit more from attack speed and flat on-hit damage. This makes Wit's End the best 1st item possible, with Runaan's following soon after. Guinsoo's Rageblade and Hextech Gunblade increase DPS and keep us alive respectively, but both also grant valuable AP that we'll need before we can consider building Nashor's Tooth. Once that's done, we can finish with Rabadon's Deathcap for the greatest possible AP value.

The build has only Wit's End and Gunblade for defensive items, so we're vulnerable to burst from AD assassins. The only source of movement speed is Runaan's; we've skipped boots, so we'll be slow. Unlike standard Kog, our damage will be somewhat low when Bio-Arcane is on cooldown. The entire build is ~2600 gold more expensive than the standard one, or nearly a full item's worth. On the upside, all of Kog'Maw's other abilities scale with AP, and will hit extremely hard late-game. Finally, this really is the highest damage build on Kog'maw, I didn't make that up. Is it worth it?

....No, not really. But I recommend trying this out in ranked!