KDD Cup 2020 AutoGraph Challenge Forum

Go back to competition Back to thread list Post in this thread

> rank average seems unfair

If I have a great model which get me a pretty high score, I will let some of my friends submit lots of model that get a bit lower score than me.

For example:
My score : 80,80,80,80,80
others:79,80,80,80,80
80,79,80,80,80
80,80,79,79,80
...

So if some has a model which perform better than me at every datasets except one (90,90,90,78,90).

My rank: 2,2,2,1,2
His rank: 1,1,1,n,1 (n = the num of my friends)

It's simple that I can beat EVERYONE except you have a model get five higher scores than me.

Posted by: qqerret @ April 2, 2020, 6:09 a.m.

Hi,

Thanks for the question!

1. The current leaderboard is the feedback leaderboard based on performances on feedback datasets. The final ranking will be determined by another 5 final datasets from which participants can't have feedback on performance. So even if some tricks are played in feedback leaderboard, it will not guarantee any advantage against other participants in the final leaderboard

2. We will ask participants to send real personal identification information before revealing the final results

3. Any cheating or attempt to cheat will have the risk of invalidating the participant(s)

Best regards,
KDD Cup 2020 AutoGraph Organizing Team

Posted by: automlai @ April 2, 2020, 12:53 p.m.

Same here,
why not take the average accuracy over all datasets?

Posted by: axmat @ May 5, 2020, 3:47 p.m.

Hi,

Different datasets have different accuracy performance with different variances. It is thus inappropriate to average them as a ranking reference.

Best regards,
KDD Cup 2020 AutoGraph Organizing Team

Posted by: automlai @ May 6, 2020, 10:43 a.m.
Post in this thread