After reading the “Post-American trials performance in Europe” thread, it dawns on me that the tried-and-true methodology of the T&FN Annual Rankings may no longer be viable. First of all, the T&FN Rankings are the de facto Gold Standard. All attempts to quantify rankings through some BCSian algorithm are always doomed to failure, because they can’t take into account the vicissitudes of an athlete’s season. They rarely pass the common sense filter. T&FN takes the normally abhorred ‘selection committee’ approach and it works! But . . . the methodology, over 60 years old, may no longer reflect the realities of 21st century pro T&F.
The sacrosanct criteria are, in order of importance: 1. honors won 2. head-to-head record 3. sequence of marks
So let’s examine each. 1. Honors won – obviously still the best way to separate the best of the best. It starts with the OG/WC results, but also looks at ‘lesser’ meets and considers order-of-finish there. We need to come back to this category in a bit.
2. Head-to-head – Back in the day, this category made a lot of sense, because the Peaking Phenomenon was not as pronounced, and athletes tended to try all-out at every major meet they went to. But now, athletes tend to ‘train through’ meets, even Diamond League meets, with an eye on the bigger prize, the OG or WC. The Diamond League is an excellent concept, but wins and losses there are not as important, over the course of an entire season, as being ready when your training is designed to prepare you to be (‘rest’ being the key component there). Plus, not all meets are created equal. Athlete A may have a losing, 4-7, head-to-head record, but in the key matchups that the athlete was pointing towards, s/he may be 3-0. An early-season or late-season meet can’t compare with a mid-season one.
3. Sequence of marks. While we Marks Snobs love this category, think about Ricky Bruch sitting in Malmo, Sweden, throwing 225’ every other day and compiling an unbeatable string of marks. Obviously big marks are only important when they are made at big (int’l) meets, not the National Championships in your backyard.
So where does that leave us? With a need to rank ‘events-in-meets’ and look at records there. By this I mean that not all events are created equal at all meets. The Zurich GL has been generally regarded as the best one-off meet every year, but not every event is as ‘significant’ as the next. It depends of who shows up and how good the marks are. If Bolt, Powell, Gatlin, Gay, etc. show up and run in the 9.6-9.8 range, THAT is a ranking data-point worth noting. If people are ducking other, and it’s won in 9.9 (in a +wind), not so much.
What this means is that the Ranking Criteria need to reflect the realities of each event in each meet and that meet’s significance in the season – in a much more nuanced way than just the a, b, c way as nominally given. I’m going to guess that they already do much of what I’m saying here, but one casualty of a new-think needs to be dropping the head-to-head category altogether, at least in the most literal sense.
Too complicated? Perhaps . . . but we’re worth it!!
1. I'm a "marks snob" too, so for me, records, whether WR, CR, HSR, MR.....or even a significant PR....is worth more to me than whatever medal someone wins. A medal in, say, the men's 5K, won in 13:35....is worth MUCH less than a medal won in 12:57!! (Weather and other factors taken into account.)
2. Agree totally about head-to-heads. Except in the high hurdles, especially the women's, WHERE are the head-to-heads?? Bolt and Blake in the same meet.....but in different races! Actually, recently the women's HJ has been good in this area, with Vlasic, Barrett, and Chicherova meeting often.
3. Honors won. Agree!! Don't look at the meet, look at the field IN the meet!! An OG or WC event won against a "weak" field should be worth less than, say, Stanford or Pre or Oxy won against a solid field!!
VERY interesting thread, Marlow!! Can't wait for GH to come in on it!!
Though I want to say that the methodology is outmoded and should be changed, what exactly would it be changed to? I'm a marks snob, too; and I believe the the European meets get a pass for excellence that they shouldn't but we also have to remember that May/June in Europe is like March/April in the US. You can't count their races when they're slow and then NOT count the US races - especially when there is such a big climate difference between the primary track countries of North America and Europe. If Americans are in Europe running slow than it MUST be counted. And, if NACAC or EAA athletes are in AUS or RSA in February than the marks MUST be counted. You can't have it both ways.
Also, head to head matters. There is just too much differentiation between the meets that marks alone would even bother a marks snob like me. Example? Stadium races versus field races. Euro DL races versus Clermont, Rieti, Port of Spain, Eugene, Des Moines, Indianapolis, Anything run in Texas, 6000 ft of altitude, loughborough, Hawai'i discus...the list goes on.
Honors won? Please tell me you're exaggerating that anything run at Stanford or Oyx or Pre would count for anything. It only tells us how fast someone runs in a non-race setting. Unless and until pacing/rabbits are outlawed no one should ever base a distance race purely on time.
Marlow wrote:but one casualty of a new-think needs to be dropping the head-to-head category altogether
I think you're correct that many of the nuances you describe already come into play in that smoke filled room. To the extent that I wonder if the head to head is really a problem. Is it always the case that the athlete with the better head to head is above the athlete they have beaten? I have not poured over the rankings in as much detail as you, but I wonder if the 'lesser' athlete is ranked higher more often than we think?
Last edited by Daisy on Thu Jul 11, 2013 11:52 am, edited 1 time in total.
Marlow wrote:(cont'd) What this means is that the Ranking Criteria need to reflect the realities of each event in each meet and that meet’s significance in the season – in a much more nuanced way than just the a, b, c way as nominally given. I’m going to guess that they already do much of what I’m saying here, but one casualty of a new-think needs to be dropping the head-to-head category altogether, at least in the most literal sense.
Too complicated? Perhaps . . . but we’re worth it!!
I cannot glean what you mean here... Either they think the value of the meet through green ... or they don't red ... or am I missing a connection?
Are we talking about World rankings......event-by-event......or USA rankings, event-by-event? Or is this about AOY, either World or USA??
Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!! This contradicts the "honors won" criteria, as well as head-to-head!! 1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!
I'm sure there are other, better, examples of anomalies in the rankings!!
Things must be a little slow where Marlow is at these days. A lot of words about the subject, but nothing much of substance. I believe just about everything that he mentioned is wrong or needs to change in the rankings is already accounted for by the rankers.
So I challenge Marlow to take a look at last year's world rankings, and tell me where they went terribly wrong. Be as specific as possible. But not too wordy.
Many of the events reflect the results of the biggest meet of the year, with some changes here and there. Most notable are the events where one country had many more than 3 of the top 10 in the world. A few events don't follow the Olympic results very well, but those are like the men's 1500, where the medalists had few other top results. There is some switching of positions from the Olympic results, but usually because one person dominated the season, but had a narrow loss in the Olympics.
The adage is: If it ain't broke, don't fix it. Thus, you think it is broken. Let us start by getting one thing straight. Track and Field is a sport, and the essence of sport is competition. To think that a fast mark coming in fifth or wherever in an event run for time is superior to winning a race of significance is fundamentally flawed, and so those that do the ranking will not be moved by the LetsRun metric of performance. Time is a very good indicator for the run-of-the-mill people, but is inadequate for determining those that are the best.
mcgato wrote:So I challenge Marlow to take a look at last year's world rankings, and tell me where they went terribly wrong. Be as specific as possible. But not too wordy.
Done already (sorta). I have being doing the 'Marks Snob Rankings' in some events for the past couple of years and my outcomes have differed from T&FN's. I do not pretend to be 'better' (!), but just a different way to look at things, more aligned with what I've said (so verbosely) here.
I did a search and here's one of them (there's others):
aaronk said: "Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!! This contradicts the "honors won" criteria, as well as head-to-head!! 1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!"
I'm sure there are other, better, examples of anomalies in the rankings!![/quote]
Actually, aaronk, your memory is slightly faulty. For 1974, Dick Taylor of New Zealand was ranked #1 by virtue of winning the Commonwealth Games. Pre actually only ranked 10th(1!) probably because his run was basically a solo effort with no real competition. And it was actually in Eugene, although the previous year, he ran an AR 6mile in Bakersfield (27:09.4) Rankings courtesy of T&FN!
There is some legitimacy to the argument, though. Is a win in the 100m at European Championships in 2012 greater than a win in the 100m at SEC's or NCAA's? Is a 3rd place at Finnish championships for Javelin better than a win at NCAA's? Is a win at the Melbourne Classic or Moscow Challenge or Golden Spike Ostrava better than the w100m at USATF? Is a World Challenge distance race in Kingston better than an Area Permit race in Padova?
huntinwr wrote:aaronk said: "Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!! This contradicts the "honors won" criteria, as well as head-to-head!! 1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!"
I'm sure there are other, better, examples of anomalies in the rankings!!
Actually, aaronk, your memory is slightly faulty. For 1974, Dick Taylor of New Zealand was ranked #1 by virtue of winning the Commonwealth Games. Pre actually only ranked 10th(1!) probably because his run was basically a solo effort with no real competition. And it was actually in Eugene, although the previous year, he ran an AR 6mile in Bakersfield (27:09.4) Rankings courtesy of T&FN! [/quote]
Thanks for the correction! I think I remember why Pre's race stood out for me. I remember being totally PISSED OFF at T&FN for ranking him 10th!! Being a "marks snob", I've always believed marks should be rated # 1. (Especially back then, when pacers/rabbits weren't common. I don't think Pre was paced in that run!)
Soooooo......how did Pre rank off that 6 mile in 1973??
10,000m finishing times: 1. Pre, OTC, 27:43.6 (bests Frank Shorter's (Florida TC) 1972 27:51.4 time Also bests Hayward Field record of 28:35.6 set by Shorter in 1972) 2. Terry Williams,regon, 29:00.8 3. Pat Tyson, OTC, 29:03.0 4. Mike Long, OTC, 29:05.4 5. Jon Anderson, OTC, 29:11.8 6. Eric Sigmont, OTC 29:42.6 7. Damien Koch, OTC, no time
Please disregard that WC final. As least as a predictor of how the Rankings would shake-out. The medalists ended up ranking 3-5-7, while the first two in our compilation finished 10-12 in Daegu. What can happen when a "tactical" race is compounded by significant contact......
Heretofore known as a good racer, Uceny actually put up the fastest time of the year in winning at the Brussels finale. That combined with an early pair of DL wins (plus a 2nd and a 3rd) more than made up for her Korean tumble....Selsouli came on as a big factor in September, but it was too late.
Does everybody agree?? What happened to "honors won"? And if marks are criteria # 3, then why make such prominent mention of Uceny having run the year's fastest!!
Last edited by aaronk on Thu Jul 11, 2013 1:17 pm, edited 1 time in total.
aaronk wrote:Can't remember all the details----don't own the issue anymore.....
Could be thinking of Bruce Bickford who ranked #1 in 1985 for 10,000. I think that it was noted that he only ran one that year. But the 10 is hard to rank without a championship meet, as there are only a few top quality races over the year.
batonless relay wrote:Is a win in the 100m at European Championships in 2012 greater than a win in the 100m at SEC's or NCAA's?
That is why the IAAF scrapped their point based ranking system. They treated the EC as superior to the NCAA in all events, which is stupid. TFN also thought that was a stupid system and wrote disparaging things about it. They held their rankings as superior, because they knew the difference between the quality of meets by event.
So far people keep bringing up hypothetical issues that TFN understands and factors into their rankings. Nobody has pointed out a single ranking from TFN that is out of line. Thus I have to conclude that they do as good of a job as can be done.
sjm1368 wrote:My only issue with the rankings is that they matter so much. Athlete contracts are tied to TFN rankings.
It's hard to realize that a committee looking at results can decide whether an athlete makes $40k the next year or $25k. Or something similar.
That just sucks that a committee decides that
T&FN's Rankings existed for decades before there were such things as shoe contracts.
The shoe companies came late to the party and apparently were looking for an impartial method of assessing value, so they (without asking us, I might add) (and at no compensation to us, I might also add) hopped a ride on our bandwagon.
While we're flattered that the shoe people have elected to use our musings (which were created solely for the enjoyment of our subscribers, nothing else) as gospel, it has certainly done much to poison our relationship with some athletes and their agents through the years.
The only thing we can say is that not everybody can be No. 1, and if we did a ranking the reverse way to please somebody, there's always somebody on the other side who feels equally aggrieved. A true lose-lose for us in many ways.
Well, someone needs to say that the Annual Edition is one of the favorite issues for all of us each and every year. Always has been, always will be. Especially coming during one of those cold, dark, dreary winter months when very little real track is happening. Thanks to all those involved and don't stop doing it.
gh wrote:a. which were created solely for the enjoyment of our subscribers, nothing else b. a true lose-lose for us in many ways.
a. the Rankings are certainly 'enjoyable', but they have become much more than that. They are, arguably, the most important service that T&FN provides to the world-wide athletics community, among many, many other things. b. That is most unfortunate, but human nature being what it is [echo], it's understandable.
We would all be much the poorer without them, so make no mistake, this thread is not meant as denigration.
1. For the winners (and even the nominees), winning this award is the epitome of your career. It means you've arrived.
2. The winners (and the finalists) are always heavily criticized.....and seen as not worthy, sometimes even of being nominated in the 1st place.
3. The Oscars are thought to be political. T&FN Rankings......not!!
4. Lesser known movies (independents) are rarely, if ever given top prize. (Although that seems to be changing somewhat.....a little bit more each year.) With T&FN, winning the Rankings is based on achievement....period!! Doesn't matter if you have a famous coach, are tied to a famous corporation, etc. Perform well, win the Biggies, and you Rank!!
5. Both are endlessly open for energetic, strident, lively, and fun debate!
6. Both have survived for many decades.....despite all the changes in their respective fields (Oscars....from silents to sound to color to whatever! T&FN Rankings from AAU to TAC to USATF.....from amateur to pro......from Harrison Dillard to Aries Merritt!!)
Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!!
aaronk wrote:Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!!
Ummm, Chariots of Fire????
Wasn't Forrest Gump a runner!
Ouch! Nothing like going for a run and having someone yell at you "Run Forrest Run!" And that damned Vangelis song...Certainly not the kind of music to make one get pumped up, although I place more blame on that slow motion beach scene than the music. Back in the day, someone would be playing that song ad nauseum at every friggin' race. I remember doing a marathon where it seemed like the song was playing the entire time. Then we passed someone's house who was playing Black Sabbath. That certainly got the juices flowing!
Great topic, and one that I have contemplated initiating from time to time. Over the years I have felt that the stated rankings criteria were deficient in various ways, but that the acutal rankings tended to be pristine. I might quibble with a place here and there, but I always come away feeling that the ranking committee did a fantastic job.
I think one of the reasons that the rankings are so good in spite of deficiencies in the ranking criteria is that the committee members are free to deviate from the criteria as they see fit. There are so many special circumstances that can and do arise that a lot of flexibility in the application of the criteria is needed to get the rankings right. The members of the ranking committee have, in my opinion, exercised this flexibility admirably to come up with outstanding results in the final published rankings.
But I do have one concern about the rankings future that I don't think has been mentioned so far. And that concern, strangely enough, is this very forum. Because people are now freer than ever to question the rankings openly and in public, and to point out how the rankings may be deviating from the stated criteria, it puts the rankers on the spot to be able to justify their rankings. If the members of the ranking committee feel that they are going to be called on the carpet for rankings that deviate from the official criteria, they are more likely to exercise less flexibility in deviating from the criteria. In other words, there is a potential for some loss of the very freedom that has helped to make the rankings so good over the years.
So my suggestion, as a barely-audible voice in the crowd, is to be a little loose about pointing out rankings that apparently deviate from the stated criteria. Such deviations have been a strength of the rankings, not a weakness. I don't want to see the members of the ranking committee feeling pressured to be more rigid about the criteria to avoid criticism, at the expense of the quality of the rankings.