T&FN Rankings Methodology Out-moded?


Main message board: for the discussion of topical track & field items only.

T&FN Rankings Methodology Out-moded?

Postby Marlow » Thu Jul 11, 2013 10:15 am

Sacrilege?!

After reading the “Post-American trials performance in Europe” thread, it dawns on me that the tried-and-true methodology of the T&FN Annual Rankings may no longer be viable. First of all, the T&FN Rankings are the de facto Gold Standard. All attempts to quantify rankings through some BCSian algorithm are always doomed to failure, because they can’t take into account the vicissitudes of an athlete’s season. They rarely pass the common sense filter. T&FN takes the normally abhorred ‘selection committee’ approach and it works! But . . . the methodology, over 60 years old, may no longer reflect the realities of 21st century pro T&F.

The sacrosanct criteria are, in order of importance:
1. honors won
2. head-to-head record
3. sequence of marks

So let’s examine each.
1. Honors won – obviously still the best way to separate the best of the best. It starts with the OG/WC results, but also looks at ‘lesser’ meets and considers order-of-finish there. We need to come back to this category in a bit.

2. Head-to-head – Back in the day, this category made a lot of sense, because the Peaking Phenomenon was not as pronounced, and athletes tended to try all-out at every major meet they went to. But now, athletes tend to ‘train through’ meets, even Diamond League meets, with an eye on the bigger prize, the OG or WC. The Diamond League is an excellent concept, but wins and losses there are not as important, over the course of an entire season, as being ready when your training is designed to prepare you to be (‘rest’ being the key component there).
Plus, not all meets are created equal. Athlete A may have a losing, 4-7, head-to-head record, but in the key matchups that the athlete was pointing towards, s/he may be 3-0. An early-season or late-season meet can’t compare with a mid-season one.

3. Sequence of marks. While we Marks Snobs love this category, think about Ricky Bruch sitting in Malmo, Sweden, throwing 225’ every other day and compiling an unbeatable string of marks. Obviously big marks are only important when they are made at big (int’l) meets, not the National Championships in your backyard.

(to be cont'd)
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Thu Jul 11, 2013 10:15 am

(cont'd)

So where does that leave us? With a need to rank ‘events-in-meets’ and look at records there. By this I mean that not all events are created equal at all meets. The Zurich GL has been generally regarded as the best one-off meet every year, but not every event is as ‘significant’ as the next. It depends of who shows up and how good the marks are. If Bolt, Powell, Gatlin, Gay, etc. show up and run in the 9.6-9.8 range, THAT is a ranking data-point worth noting. If people are ducking other, and it’s won in 9.9 (in a +wind), not so much.

What this means is that the Ranking Criteria need to reflect the realities of each event in each meet and that meet’s significance in the season – in a much more nuanced way than just the a, b, c way as nominally given. I’m going to guess that they already do much of what I’m saying here, but one casualty of a new-think needs to be dropping the head-to-head category altogether, at least in the most literal sense.

Too complicated? Perhaps . . . but we’re worth it!!
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby aaronk » Thu Jul 11, 2013 10:43 am

Agree....across the board.

1. I'm a "marks snob" too, so for me, records, whether WR, CR, HSR, MR.....or even a significant PR....is worth more to me than whatever medal someone wins.
A medal in, say, the men's 5K, won in 13:35....is worth MUCH less than a medal won in 12:57!!
(Weather and other factors taken into account.)

2. Agree totally about head-to-heads. Except in the high hurdles, especially the women's, WHERE are the head-to-heads?? Bolt and Blake in the same meet.....but in different races! Actually, recently the women's HJ has been good in this area, with Vlasic, Barrett, and Chicherova meeting often.

3. Honors won. Agree!! Don't look at the meet, look at the field IN the meet!! An OG or WC event won against a "weak" field should be worth less than, say, Stanford or Pre or Oxy won against a solid field!!

VERY interesting thread, Marlow!!
Can't wait for GH to come in on it!! :shock:
aaronk
 
Posts: 3354
Joined: Sun May 06, 2012 9:39 am

Re: T&FN Rankings Methodology Out-moded?

Postby batonless relay » Thu Jul 11, 2013 11:01 am

Though I want to say that the methodology is outmoded and should be changed, what exactly would it be changed to? I'm a marks snob, too; and I believe the the European meets get a pass for excellence that they shouldn't but we also have to remember that May/June in Europe is like March/April in the US. You can't count their races when they're slow and then NOT count the US races - especially when there is such a big climate difference between the primary track countries of North America and Europe. If Americans are in Europe running slow than it MUST be counted. And, if NACAC or EAA athletes are in AUS or RSA in February than the marks MUST be counted. You can't have it both ways.

Also, head to head matters. There is just too much differentiation between the meets that marks alone would even bother a marks snob like me. Example? Stadium races versus field races. Euro DL races versus Clermont, Rieti, Port of Spain, Eugene, Des Moines, Indianapolis, Anything run in Texas, 6000 ft of altitude, loughborough, Hawai'i discus...the list goes on.

Honors won? Please tell me you're exaggerating that anything run at Stanford or Oyx or Pre would count for anything. It only tells us how fast someone runs in a non-race setting. Unless and until pacing/rabbits are outlawed no one should ever base a distance race purely on time.
batonless relay
 
Posts: 700
Joined: Tue Feb 19, 2013 9:40 am

Re: T&FN Rankings Methodology Out-moded?

Postby Daisy » Thu Jul 11, 2013 11:26 am

Marlow wrote:but one casualty of a new-think needs to be dropping the head-to-head category altogether

I think you're correct that many of the nuances you describe already come into play in that smoke filled room. To the extent that I wonder if the head to head is really a problem. Is it always the case that the athlete with the better head to head is above the athlete they have beaten? I have not poured over the rankings in as much detail as you, but I wonder if the 'lesser' athlete is ranked higher more often than we think?
Last edited by Daisy on Thu Jul 11, 2013 11:52 am, edited 1 time in total.
Daisy
 
Posts: 13153
Joined: Sat Oct 08, 2005 4:32 am

Re: T&FN Rankings Methodology Out-moded?

Postby bushop » Thu Jul 11, 2013 11:35 am

Marlow wrote:(cont'd) What this means is that the Ranking Criteria need to reflect the realities of each event in each meet and that meet’s significance in the season – in a much more nuanced way than just the a, b, c way as nominally given. I’m going to guess that they already do much of what I’m saying here, but one casualty of a new-think needs to be dropping the head-to-head category altogether, at least in the most literal sense.

Too complicated? Perhaps . . . but we’re worth it!!

I cannot glean what you mean here...
Either they think the value of the meet through green
... or they don't red
... or am I missing a connection?
bushop
 
Posts: 1868
Joined: Sat Oct 08, 2005 4:32 am
Location: near the toys and tape measures

Re: T&FN Rankings Methodology Out-moded?

Postby aaronk » Thu Jul 11, 2013 11:42 am

Are we talking about World rankings......event-by-event......or USA rankings, event-by-event?
Or is this about AOY, either World or USA??

Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!!
This contradicts the "honors won" criteria, as well as head-to-head!!
1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!

I'm sure there are other, better, examples of anomalies in the rankings!!
aaronk
 
Posts: 3354
Joined: Sun May 06, 2012 9:39 am

Re: T&FN Rankings Methodology Out-moded?

Postby mcgato » Thu Jul 11, 2013 11:47 am

Things must be a little slow where Marlow is at these days. A lot of words about the subject, but nothing much of substance. I believe just about everything that he mentioned is wrong or needs to change in the rankings is already accounted for by the rankers.

So I challenge Marlow to take a look at last year's world rankings, and tell me where they went terribly wrong. Be as specific as possible. But not too wordy.

Many of the events reflect the results of the biggest meet of the year, with some changes here and there. Most notable are the events where one country had many more than 3 of the top 10 in the world. A few events don't follow the Olympic results very well, but those are like the men's 1500, where the medalists had few other top results. There is some switching of positions from the Olympic results, but usually because one person dominated the season, but had a narrow loss in the Olympics.
mcgato
 
Posts: 1605
Joined: Sat Oct 08, 2005 4:32 am
Location: Hoboken

Re: T&FN Rankings Methodology Out-moded?

Postby 26mi235 » Thu Jul 11, 2013 11:54 am

The adage is: If it ain't broke, don't fix it. Thus, you think it is broken. Let us start by getting one thing straight. Track and Field is a sport, and the essence of sport is competition. To think that a fast mark coming in fifth or wherever in an event run for time is superior to winning a race of significance is fundamentally flawed, and so those that do the ranking will not be moved by the LetsRun metric of performance. Time is a very good indicator for the run-of-the-mill people, but is inadequate for determining those that are the best.
26mi235
 
Posts: 16323
Joined: Sat Oct 08, 2005 4:32 am
Location: Madison, WI

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Thu Jul 11, 2013 11:57 am

mcgato wrote:So I challenge Marlow to take a look at last year's world rankings, and tell me where they went terribly wrong. Be as specific as possible. But not too wordy.

Done already (sorta). I have being doing the 'Marks Snob Rankings' in some events for the past couple of years and my outcomes have differed from T&FN's. I do not pretend to be 'better' (!), but just a different way to look at things, more aligned with what I've said (so verbosely) here.

I did a search and here's one of them (there's others):

viewtopic.php?f=1&t=44562&p=730094&hilit=+marks+snob+rankings+#p730094
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby Pego » Thu Jul 11, 2013 12:17 pm

Marlow, you made me look and it does not seem that your and T&FN results are that much different. Here is Marlow's example, a 2011 wPV.

Marlow

1. Suhr
2. Spieglbg
3. Murer
4. Feo
5. Isi
6. Strutz

T&FN

1. Jenn Suhr (US)
2. Fabiana Murer (Brazil)
3. Silke Spiegelburg (Germany)
4. Yelena Isinbaeva (Russia)
5. Svetlana Feofanova (Russia)
6. Martina Strutz (Germany)
Pego
 
Posts: 10199
Joined: Sat Oct 08, 2005 4:32 am
Location: beyond help

Re: T&FN Rankings Methodology Out-moded?

Postby huntinwr » Thu Jul 11, 2013 12:22 pm

aaronk said: "Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!!
This contradicts the "honors won" criteria, as well as head-to-head!!
1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!"

I'm sure there are other, better, examples of anomalies in the rankings!![/quote]

Actually, aaronk, your memory is slightly faulty. For 1974, Dick Taylor of New Zealand was ranked #1 by virtue of winning the Commonwealth Games. Pre actually only ranked 10th(1!) probably because his run was basically a solo effort with no real competition. And it was actually in Eugene, although the previous year, he ran an AR 6mile in Bakersfield (27:09.4) :)
Rankings courtesy of T&FN! :)
huntinwr
 
Posts: 282
Joined: Wed Mar 11, 2009 12:49 pm
Location: Walla Walla, Washington

Re: T&FN Rankings Methodology Out-moded?

Postby batonless relay » Thu Jul 11, 2013 12:28 pm

There is some legitimacy to the argument, though. Is a win in the 100m at European Championships in 2012 greater than a win in the 100m at SEC's or NCAA's? Is a 3rd place at Finnish championships for Javelin better than a win at NCAA's? Is a win at the Melbourne Classic or Moscow Challenge or Golden Spike Ostrava better than the w100m at USATF? Is a World Challenge distance race in Kingston better than an Area Permit race in Padova?
batonless relay
 
Posts: 700
Joined: Tue Feb 19, 2013 9:40 am

Re: T&FN Rankings Methodology Out-moded?

Postby aaronk » Thu Jul 11, 2013 12:40 pm

huntinwr wrote:aaronk said: "Can't remember all the details----don't own the issue anymore.....but in the 1975 Rankings issue....for the year 1974.....I recall Pre's ONE 10K, albeit in AR time (27:43), got him ranked # 1 in the world!!
This contradicts the "honors won" criteria, as well as head-to-head!!
1974 wasn't a very important year (no OG, with only the Euro and Commonwealth Champs to compete), so maybe a fast time in a "local" meet (actually, I think he ran it in Bakersfield!) counted more than it would have in an Olympic year!!"

I'm sure there are other, better, examples of anomalies in the rankings!!


Actually, aaronk, your memory is slightly faulty. For 1974, Dick Taylor of New Zealand was ranked #1 by virtue of winning the Commonwealth Games. Pre actually only ranked 10th(1!) probably because his run was basically a solo effort with no real competition. And it was actually in Eugene, although the previous year, he ran an AR 6mile in Bakersfield (27:09.4) :)
Rankings courtesy of T&FN! :)[/quote]

Thanks for the correction!
I think I remember why Pre's race stood out for me.
I remember being totally PISSED OFF at T&FN for ranking him 10th!!
Being a "marks snob", I've always believed marks should be rated # 1.
(Especially back then, when pacers/rabbits weren't common. I don't think Pre was paced in that run!)

Soooooo......how did Pre rank off that 6 mile in 1973?? :D
aaronk
 
Posts: 3354
Joined: Sun May 06, 2012 9:39 am

Re: T&FN Rankings Methodology Out-moded?

Postby booond » Thu Jul 11, 2013 12:42 pm

I'm much more an anti-marks guy and am more concerned with who beat who when. It's about winning races against great competition.
booond
 
Posts: 572
Joined: Fri Jun 22, 2012 11:25 am

Re: T&FN Rankings Methodology Out-moded?

Postby br » Thu Jul 11, 2013 12:43 pm

2 American records in one race. 26:58.8 6 mile, 27:43.6 10,000m.

http://www.flickriver.com/photos/thehap ... 752520320/

10,000m finishing times:
1. Pre, OTC, 27:43.6 (bests Frank Shorter's (Florida TC) 1972 27:51.4 time
Also bests Hayward Field record of 28:35.6 set by Shorter in 1972)
2. Terry Williams,regon, 29:00.8
3. Pat Tyson, OTC, 29:03.0
4. Mike Long, OTC, 29:05.4
5. Jon Anderson, OTC, 29:11.8
6. Eric Sigmont, OTC 29:42.6
7. Damien Koch, OTC, no time
br
 
Posts: 478
Joined: Sat Oct 08, 2005 4:32 am

Re: T&FN Rankings Methodology Out-moded?

Postby huntinwr » Thu Jul 11, 2013 12:50 pm

Soooooo......how did Pre rank off that 6 mile in 1973?? :D[/quote]

Unranked. 1974 was the only year he ranked at all at 10,000.
huntinwr
 
Posts: 282
Joined: Wed Mar 11, 2009 12:49 pm
Location: Walla Walla, Washington

Re: T&FN Rankings Methodology Out-moded?

Postby huntinwr » Thu Jul 11, 2013 12:52 pm

[quote="br"]2 American records in one race. 26:58.8 6 mile, 27:43.6 10,000m.

6 mile time was actually 26:51.8, but appreciate the pictures! 376 yards in 44.8 would have been quite a sprint for anyone, especially Pre :)
huntinwr
 
Posts: 282
Joined: Wed Mar 11, 2009 12:49 pm
Location: Walla Walla, Washington

Re: T&FN Rankings Methodology Out-moded?

Postby aaronk » Thu Jul 11, 2013 1:14 pm

Here's another one that I thought at the time went against T&FN own criteria for rankings.
The rankings for the women's 1500 for 2011 (in Feb 2012 issue).

1. Uceny
2. Jamal
3. Simpson
4. Mishchenko
5. England
6. Selsouli

Here's (verbatim) T&FN's write-up for the event:

Please disregard that WC final. As least as a predictor of how the Rankings would shake-out. The medalists ended up ranking 3-5-7, while the first two in our compilation finished 10-12 in Daegu. What can happen when a "tactical" race is compounded by significant contact......

Heretofore known as a good racer, Uceny actually put up the fastest time of the year in winning at the Brussels finale. That combined with an early pair of DL wins (plus a 2nd and a 3rd) more than made up for her Korean tumble....Selsouli came on as a big factor in September, but it was too late.

Does everybody agree??
What happened to "honors won"?
And if marks are criteria # 3, then why make such prominent mention of Uceny having run the year's fastest!!
Last edited by aaronk on Thu Jul 11, 2013 1:17 pm, edited 1 time in total.
aaronk
 
Posts: 3354
Joined: Sun May 06, 2012 9:39 am

Re: T&FN Rankings Methodology Out-moded?

Postby mcgato » Thu Jul 11, 2013 1:17 pm

aaronk wrote:Can't remember all the details----don't own the issue anymore.....
Could be thinking of Bruce Bickford who ranked #1 in 1985 for 10,000. I think that it was noted that he only ran one that year. But the 10 is hard to rank without a championship meet, as there are only a few top quality races over the year.

batonless relay wrote:Is a win in the 100m at European Championships in 2012 greater than a win in the 100m at SEC's or NCAA's?
That is why the IAAF scrapped their point based ranking system. They treated the EC as superior to the NCAA in all events, which is stupid. TFN also thought that was a stupid system and wrote disparaging things about it. They held their rankings as superior, because they knew the difference between the quality of meets by event.

So far people keep bringing up hypothetical issues that TFN understands and factors into their rankings. Nobody has pointed out a single ranking from TFN that is out of line. Thus I have to conclude that they do as good of a job as can be done.
mcgato
 
Posts: 1605
Joined: Sat Oct 08, 2005 4:32 am
Location: Hoboken

Re: T&FN Rankings Methodology Out-moded?

Postby sjm1368 » Thu Jul 11, 2013 1:33 pm

My only issue with the rankings is that they matter so much. Athlete contracts are tied to TFN rankings.

It's hard to realize that a committee looking at results can decide whether an athlete makes $40k the next year or $25k. Or something similar.

That just sucks that a committee decides that
sjm1368
 
Posts: 59
Joined: Sat Oct 08, 2005 4:32 am
Location: Tx

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Thu Jul 11, 2013 1:45 pm

sjm1368 wrote:That just sucks that a committee decides that

The committee most certainly does NOT decide that. THE SHOE COMPANIES decide what criteria they want to recognize and THEY pick T&FN (and wisely so).
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby gh » Thu Jul 11, 2013 1:49 pm

sjm1368 wrote:My only issue with the rankings is that they matter so much. Athlete contracts are tied to TFN rankings.

It's hard to realize that a committee looking at results can decide whether an athlete makes $40k the next year or $25k. Or something similar.

That just sucks that a committee decides that


T&FN's Rankings existed for decades before there were such things as shoe contracts.

The shoe companies came late to the party and apparently were looking for an impartial method of assessing value, so they (without asking us, I might add) (and at no compensation to us, I might also add) hopped a ride on our bandwagon.

While we're flattered that the shoe people have elected to use our musings (which were created solely for the enjoyment of our subscribers, nothing else) as gospel, it has certainly done much to poison our relationship with some athletes and their agents through the years.

The only thing we can say is that not everybody can be No. 1, and if we did a ranking the reverse way to please somebody, there's always somebody on the other side who feels equally aggrieved. A true lose-lose for us in many ways.
gh
 
Posts: 46327
Joined: Sat Oct 08, 2005 4:31 am
Location: firmly at Arya's side!

Re: T&FN Rankings Methodology Out-moded?

Postby DrJay » Thu Jul 11, 2013 2:40 pm

Well, someone needs to say that the Annual Edition is one of the favorite issues for all of us each and every year. Always has been, always will be. Especially coming during one of those cold, dark, dreary winter months when very little real track is happening. Thanks to all those involved and don't stop doing it.
DrJay
 
Posts: 5485
Joined: Sat Oct 08, 2005 4:32 am
Location: Woodland Park, CO

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Thu Jul 11, 2013 2:53 pm

gh wrote:a. which were created solely for the enjoyment of our subscribers, nothing else
b. a true lose-lose for us in many ways.

a. the Rankings are certainly 'enjoyable', but they have become much more than that. They are, arguably, the most important service that T&FN provides to the world-wide athletics community, among many, many other things.
b. That is most unfortunate, but human nature being what it is [echo], it's understandable.

We would all be much the poorer without them, so make no mistake, this thread is not meant as denigration.
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby aaronk » Thu Jul 11, 2013 3:20 pm

T&FN World Rankings vs the Oscars.

1. For the winners (and even the nominees), winning this award is the epitome of your career. It means you've arrived.

2. The winners (and the finalists) are always heavily criticized.....and seen as not worthy, sometimes even of being nominated in the 1st place.

3. The Oscars are thought to be political. T&FN Rankings......not!!

4. Lesser known movies (independents) are rarely, if ever given top prize. (Although that seems to be changing somewhat.....a little bit more each year.) With T&FN, winning the Rankings is based on achievement....period!! Doesn't matter if you have a famous coach, are tied to a famous corporation, etc. Perform well, win the Biggies, and you Rank!!

5. Both are endlessly open for energetic, strident, lively, and fun debate!

6. Both have survived for many decades.....despite all the changes in their respective fields (Oscars....from silents to sound to color to whatever! T&FN Rankings from AAU to TAC to USATF.....from amateur to pro......from Harrison Dillard to Aries Merritt!!)

Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!! :P
aaronk
 
Posts: 3354
Joined: Sun May 06, 2012 9:39 am

Re: T&FN Rankings Methodology Out-moded?

Postby mcgato » Thu Jul 11, 2013 3:36 pm

aaronk wrote:Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!! :P
Ummm, Chariots of Fire????
mcgato
 
Posts: 1605
Joined: Sat Oct 08, 2005 4:32 am
Location: Hoboken

Re: T&FN Rankings Methodology Out-moded?

Postby Daisy » Thu Jul 11, 2013 4:56 pm

mcgato wrote:
aaronk wrote:Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!! :P
Ummm, Chariots of Fire????

Wasn't Forrest Gump a runner!
Daisy
 
Posts: 13153
Joined: Sat Oct 08, 2005 4:32 am

Re: T&FN Rankings Methodology Out-moded?

Postby Bob Duncan » Thu Jul 11, 2013 7:26 pm

Daisy wrote:
mcgato wrote:
aaronk wrote:Now...if only a movie about TRACK AND FIELD..........would win an OSCAR!!! :P
Ummm, Chariots of Fire????

Wasn't Forrest Gump a runner!

Ouch! Nothing like going for a run and having someone yell at you "Run Forrest Run!"
And that damned Vangelis song...Certainly not the kind of music to make one get pumped up, although I place more blame on that slow motion beach scene than the music. Back in the day, someone would be playing that song ad nauseum at every friggin' race. I remember doing a marathon where it seemed like the song was playing the entire time. Then we passed someone's house who was playing Black Sabbath. That certainly got the juices flowing!

Sorry to get off topic, but I couldn't resist.
Bob Duncan
 
Posts: 845
Joined: Sat Oct 08, 2005 4:32 am

Re: T&FN Rankings Methodology Out-moded?

Postby oldvaulter » Thu Jul 11, 2013 9:06 pm

Great topic, and one that I have contemplated initiating from time to time. Over the years I have felt that the stated rankings criteria were deficient in various ways, but that the acutal rankings tended to be pristine. I might quibble with a place here and there, but I always come away feeling that the ranking committee did a fantastic job.

I think one of the reasons that the rankings are so good in spite of deficiencies in the ranking criteria is that the committee members are free to deviate from the criteria as they see fit. There are so many special circumstances that can and do arise that a lot of flexibility in the application of the criteria is needed to get the rankings right. The members of the ranking committee have, in my opinion, exercised this flexibility admirably to come up with outstanding results in the final published rankings.

But I do have one concern about the rankings future that I don't think has been mentioned so far. And that concern, strangely enough, is this very forum. Because people are now freer than ever to question the rankings openly and in public, and to point out how the rankings may be deviating from the stated criteria, it puts the rankers on the spot to be able to justify their rankings. If the members of the ranking committee feel that they are going to be called on the carpet for rankings that deviate from the official criteria, they are more likely to exercise less flexibility in deviating from the criteria. In other words, there is a potential for some loss of the very freedom that has helped to make the rankings so good over the years.

So my suggestion, as a barely-audible voice in the crowd, is to be a little loose about pointing out rankings that apparently deviate from the stated criteria. Such deviations have been a strength of the rankings, not a weakness. I don't want to see the members of the ranking committee feeling pressured to be more rigid about the criteria to avoid criticism, at the expense of the quality of the rankings.
oldvaulter
 
Posts: 970
Joined: Sat Oct 08, 2005 4:31 am
Location: rural Northern California

Re: T&FN Rankings Methodology Out-moded?

Postby bushop » Thu Jul 11, 2013 9:20 pm

oldvaulter wrote:... it puts the rankers on the spot to be able to justify their rankings.

Two assumptions here:
1.) They read this forum.
2.) They give a hoot what we think.
bushop
 
Posts: 1868
Joined: Sat Oct 08, 2005 4:32 am
Location: near the toys and tape measures

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Fri Jul 12, 2013 4:23 am

bushop wrote:
oldvaulter wrote:... it puts the rankers on the spot to be able to justify their rankings.

Two assumptions here:
1.) They read this forum.
2.) They give a hoot what we think.

Yeah, I very much doubt they worry about being second-guessed; they probably worry more about their peer rankers' opinions. I imagine some smoke-filled back room at the Ramada with fifteen dossiers spread out on the table for the Hammer. One by one a dossier is retired from the table till there are 10. Then one by one the weakest is brought forward - #10, #9, #8, till there are but two left. Sometimes #1 is obvious, sometimes it's a struggle. Tentatively one is pushed forward for the top spot. A quick glance around the room reveals scowls. It is withdrawn. The other is pushed out . . . nods. White smoke puffs out from the room . . . we have a new #1! :P
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill

Re: T&FN Rankings Methodology Out-moded?

Postby MJR » Fri Jul 12, 2013 5:43 am

Rarely are the individual event rankings more than slightly controversial. T&FN crew does a good job of weighing the relative performances of each athlete against one another. The one area that is a bit shady is the overall AOY. The major gripe w/ the AOY is the long standing bias toward the glamor events over others. The performances in other events are not given equal consideration. It seems that favorite performer status has as much pull, if not more in some cases, as quality of the year depending upon the event the athlete does.
MJR
 
Posts: 1814
Joined: Sat Oct 08, 2005 4:31 am
Location: on walkabout....

Re: T&FN Rankings Methodology Out-moded?

Postby wamego relays champ » Fri Jul 12, 2013 5:51 am

Out-moded or non-viable for what purpose?

Yes, an athlete's training and peaking objectives may make placings at certain meets unimportant to them and their coaches, and so they would certainly weight the head-to-head criterion less than the World Ranking would.

But the World Rankings are not intended to state which athletes came the closest to meeting their own personal objectives over the course of the season. They simply try to answer the questions "Who was better than whom?" and "Who was the best?" given the body of work for the entire season. Such a study demands that significant weight be given to head-to-head match-ups.

Yes, these purposes may conflict which means that the value of the Rankings as a metric for sponsorship compensation or to judge an athletes (or their coach's) own view of success in a given year is limited. But as a measure of competitive excellence I think the present criteria work just fine. And there is just enough human subjectivity in the process that the rankers can consider some meet-specific conditions that might render one result as less important than another.

.
wamego relays champ
 
Posts: 478
Joined: Sat Oct 08, 2005 4:32 am

Re: T&FN Rankings Methodology Out-moded?

Postby gh » Fri Jul 12, 2013 5:54 am

MJR wrote:Rarely are the individual event rankings more than slightly controversial. T&FN crew does a good job of weighing the relative performances of each athlete against one another. The one area that is a bit shady is the overall AOY. The major gripe w/ the AOY is the long standing bias toward the glamor events over others. The performances in other events are not given equal consideration. It seems that favorite performer status has as much pull, if not more in some cases, as quality of the year depending upon the event the athlete does.


And if the Rankings crew did the AOY voting the results would be vastly different. That's the problem with democracies!
gh
 
Posts: 46327
Joined: Sat Oct 08, 2005 4:31 am
Location: firmly at Arya's side!

Re: T&FN Rankings Methodology Out-moded?

Postby Marlow » Fri Jul 12, 2013 6:08 am

wamego relays champ wrote:Out-moded or non-viable for what purpose? . . .
the World Rankings . . . try to answer the questions "Who was better than whom?" and "Who was the best?" given the body of work for the entire season. Such a study demands that significant weight be given to head-to-head match-ups.

Aye, there's the rub. If the desire is simply identify who had the best season, day in and day out, then yes, head-to-head is the way to go. But I think the real goal is (should be?) is to identify who had the best season overall (OK, that's a pretty nebulous word). In that regard, minor meets shouldn't count as much as the biggest ones, and that's where the "head-to-head record" breaks down.
Marlow
 
Posts: 21088
Joined: Thu Jan 24, 2008 5:00 pm
Location: Somewhere over the . . . hill


Who is online

Users browsing this forum: Bing [Bot], user4 and 9 guests