I'm making the numbers up, but cost to ESPN to broadcast a half hour of Scrabble is $25,000; the cost to do a half hour of track is $250,000. If you're a bean counter armed with this fact (and you also know/suspect that more people care about Scrabble than track) it's a no-brainer.
Same applies to poker, pool, spelling bees; anything that requires merely one fixed camera and 50 feet of cable (again, I exaggerate).
If you've ever been involved in setting up a facility that's going to host a televised meet, you'll know that they roll in an 18-wheeler chock full of equipment 2-3 days early and have a crew of a dozen or more people to get the place set up, let alone provide the talent to do the telecast. So think about paying all those people for 2-3 days and also covering housing (and at least some of the food) and you'll understand why so little of track gets on TV.