Caltech Challenge 2007Instructions | Development Kit | Caltech-256 Dataset
This challenge explores visual recognition algorithm performance over broad statistics of image categories. You will train on Caltech-256 images, and test on both Caltech-256 and a fresh set of images. We hope to identify 'difficult' categories and determine where different algorithm's strengths lie.
Final Results (Updated: 10/14/2007)View the final results here!
Update (Updated: 10/2/2007)We are no longer requiring contestants to re-run their results on a new set of test image - the existing challenge seems to already be using up everybody's available CPU time! If you are running short of time, just submit as many confusion matrices as you can. Your results will still be posted.
Any and all submissions accepted - but please try to submit within 24 hours of the workshop, so that I have time to update the web page.
Registration (Updated: 9/6/2007)Please register by sending mail to: Greg Griffin. Apologies for delayed replies: a response should come within 24 hours.
Schedule (Updated: 10/1/2007)
Challenge DescriptionYou will perform a 1-out-of-256 classification task.
The Challenge scores performance based on three standards:
|1. Standard||Correctly classified fraction of images (i.e. the mean of the confusion matrix diagonal).|
|2. Originality||Rewards algorithms that are good on categories that other algorithms found difficult.|
|3. Similarity||Confusing a dog with a goat is less egregious than confusing a tree with a kitchen appliance.|
How to submitYou will send us 256x256 confusion matrices for 5, 10, 20, and 50 training examples. Instructions.
Development KitThe development kit includes Matlab scripts to show images and generate training sets.