
Principle #1: Evaluation Drives Progress
Objective, quantitative evaluation drives progress:
● The choice of evaluation metric determines the direction of progress
● Arguably the most important single decision in the course of a project
Leaderboard-driven research:
● Be sure the evaluation metric corresponds closely to the end goal
● Avoid subjective evaluation (e.g. human inspection)
Hypothesis-driven research:
● Formulate a hypothesis:
○ “Double-Q learning outperforms Q-learning because it reduces upward bias”
● Verify hypothesis under a broad range of conditions
● Compare like-for-like not against existing state-of-the-art
● Seek understanding rather than leaderboard performance
评论