Instructions

Each team will perform experiments using the 4 displayed values of Ntrain. The example is a correlator which samples images onto an 8x8 grid and then classifies these feature vectors with SVM. The documentation ends with an example script that implements this classifier and generates formatted results.

Experiment Ntrain Ntest Ncat Trials Training
Files		 Test
Files:		 Your Results
(example of what
you'll email us)		 Performance:
1 5 25 256 1...10 1/train 1/test conf5.dat 4.58 ± .26%
2 10 25 256 1...10 2/train 2/test conf10.dat 5.06 ± .26%
3 20 25 256 1...10 3/train 3/test conf20.dat 6.60 ± .26%
4 50 25 256 1...10 4/train 4/test conf50.dat 8.85 ± .26%

You can create the above file lists using the Caltech 256 development kit with these training set instructions. We use fixed random number seeds to ensure that teams use the same files for each trial, thus no team can "get lucky". If you are not using MATLAB, files are listed in the table above.

Generating Your Contest Results

For each experiment (4) and trial (10 per experiment)
  1. Load Ntrain training images from each category
  2. Extract features and train your algorithm
  3. Load Ntest test images from each category
  4. Classify images and generate a confusion matrix
Your experiments result in 40, 256x256 confusion matrices. Average over the 10 trials in each experiment to produce 4 final confusion matrices, and email these to us. Lable them conf5.dat, conf10.dat, conf20.dat, and conf50.dat.

Greg Griffin
Last modified: Tue Sep 11 15:32:28 PDT 2007