Line 38: | Line 38: | ||
-q ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.tlist.xml \ | -q ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.tlist.xml \ | ||
-r ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.rttm \ | -r ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.rttm \ | ||
− | -o | + | -o dev_scores_4fusion.txt \ |
− | -t | + | -t num_queries_in_data.ref \ |
-g -z qnorm -m 1 -n 1 \ | -g -z qnorm -m 1 -n 1 \ | ||
./scores/dev/akws_br-devterms.stdlist.xml \ | ./scores/dev/akws_br-devterms.stdlist.xml \ | ||
Line 48: | Line 48: | ||
:* This process will generate 2 output files: | :* This process will generate 2 output files: | ||
− | ::'''1.''' | + | ::'''1.''' dev_scores_4fusion.txt - It contains the scores ready to be used for the following fusion stage in the following general format (one row per detection candidate): |
<query_id> <file_id> <start_time> <duration> <sc1> <sc2> ... <scN> <label> | <query_id> <file_id> <start_time> <duration> <sc1> <sc2> ... <scN> <label> | ||
Line 55: | Line 55: | ||
::* If the -g option is selected (like in this case) the <label> column contains 0s and 1s for the false and true trials respectively (derived from de rttm file). If the -g option is not selected, the last column simply contains a column with 1s. | ::* If the -g option is selected (like in this case) the <label> column contains 0s and 1s for the false and true trials respectively (derived from de rttm file). If the -g option is not selected, the last column simply contains a column with 1s. | ||
− | ::'''2.''' | + | ::'''2.'''num_queries_in_data.ref - This second (optional) output file contains the number of times each query appears in the collection data and it is used later in the fusion stage. |
*You can have a look to the general usage of this script calling it without arguments: | *You can have a look to the general usage of this script calling it without arguments: | ||
Line 83: | Line 83: | ||
-q ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.tlist.xml \ | -q ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.tlist.xml \ | ||
-r ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.rttm \ | -r ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.rttm \ | ||
− | -o | + | -o eval_scores_4fusion.txt \ |
-z qnorm -m 1 -n 1 \ | -z qnorm -m 1 -n 1 \ | ||
./scores/eval/akws_br-evalterms.stdlist.xml \ | ./scores/eval/akws_br-evalterms.stdlist.xml \ | ||
Line 96: | Line 96: | ||
* Type the following command: | * Type the following command: | ||
− | ./bin/fusion.sh | + | ./bin/fusion.sh dev_scores_4fusion.txt eval_scores_4fusion.txt queries_in_data.ref |
:* '''Warning''' Before running the script, you will have to change the MATLAB_BIN variable to be the path where your Matlab binary is actually installed. | :* '''Warning''' Before running the script, you will have to change the MATLAB_BIN variable to be the path where your Matlab binary is actually installed. | ||
Line 109: | Line 109: | ||
:* '''Warning''' The call to the fusion script can take several minutes. | :* '''Warning''' The call to the fusion script can take several minutes. | ||
− | :* This call permits learning from the | + | :* This call permits learning from the dev_scores_4fusion.txt (with groundtruh information) the calibration and fusion parameters that are applied both to the development and to the evaluation scores. It produces 3 output files: |
::'''1.''' dev_fusion.scores - file containing the development well-calibrated fusion scores | ::'''1.''' dev_fusion.scores - file containing the development well-calibrated fusion scores |
A. Abad, L. J. Rodriguez Fuentes, M. Penagarikano, A. Varona, M. Diez, and G. Bordel. On the Calibration and Fusion of Heterogeneous Spoken Term Detection Systems. In Interspeech 2013, August 25-29 2013
README.txt - Contains pretty much the same information of this wiki site ./bin/ - Contains the different scripts necessary for calibration and fusion ./bin/PrepareForFusion.sh - Main script than normalizes, aligns, hypotesizes missing scores and creates the groundtruh for the input systems ./bin/align_score_files.pl - Used by ./bin/PrepareForFusion.sh ./bin/create_groundtruth_centerdistance.pl - Used by ./bin/PrepareForFusion.sh ./bin/heuristicScoring.pl - Used by ./bin/PrepareForFusion.sh ./bin/fusion.sh - This script takes the output of ./bin/PrepareForFusion.sh to learn the fusion parameters and apply them to the evaluation scores ./bin/raw2stdslist.sh - Converts scores from a raw (internal) format to the stdlist xml format of the SWS evaluation ./scoring_atwv_sws2013/ - Contains the Mediaeval 2013 atwv scoring package ./scores/ - Contains sample dev (./scores/dev/akws_br-devterms.stdlist.xml, ./scores/dev/dtw_br-devterms.stdlist.xml) and eval scores (./scores/eval/akws_br-evalterms.stdlist.xml, ./scores/eval/dtw_br-evalterms.stdlist.xml) ./sample_results/ - Contains the sample (intermediate) results that are obtained if you run the instructions bellow
STEP1 Prepare the development scores for learning the calibration/fusion
./bin/PrepareForFusion.sh \ -q ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.tlist.xml \ -r ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.rttm \ -o dev_scores_4fusion.txt \ -t num_queries_in_data.ref \ -g -z qnorm -m 1 -n 1 \ ./scores/dev/akws_br-devterms.stdlist.xml \ ./scores/dev/dtw_br-devterms.stdlist.xml
<query_id> <file_id> <start_time> <duration> <sc1> <sc2> ... <scN> <label>
Usage: PrepareForFusion.sh -q <tlistxml> -r <rttm> -o <outputfile> [opts] <stdlistfile1> [stdlistfile2] [stdlistfile3] ... [stdlistfileN]
<stdlistfile*> input score stdlist file in the SWS2012 format (*.stdlist.xml) | - Required argument (at least 1) -q <tlistxml> termlist file in the SWS2012 format (*.tlist.xml) | - Required parameter -r <rttm> rttm file in the SWS2012 format (*.rttm) | - Required parameter -o <outputfile> output file name | - Required parameter -z <value> score z-norm type (none, qnorm, fnorm, qfnorm, fqnorm) | - Default: none -g add ground-truth information to the outputfile -t <filename> saves the number of true terms in the reference per query (implies -g) -m <value> apply majority voting fusion with <value> minimum number of votes | - Default: 1 -n <value> method for creating default scores (0: average of the other detections (MV approach); 1; min per query, 2: global min, 3: histogram based) | - Default: 0 -d debug mode, done remove auxiliar files stored in /tmp/tmpdir.$$ -h help
NOTE: Requires Matlab (or octave) and perl. It also depends on the perl scripts align_score_files.pl, heuristicScoring.pl and create_groundtruth_centerdistance.pl that should be located in the same folder of the main script
STEP2 Prepare the evaluations scores for fusion
./bin/PrepareForFusion.sh \ -q ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.tlist.xml \ -r ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.rttm \ -o eval_scores_4fusion.txt \ -z qnorm -m 1 -n 1 \ ./scores/eval/akws_br-evalterms.stdlist.xml \ ./scores/eval/dtw_br-evalterms.stdlist.xml
STEP3 Train fusion parameters using thedevelopment scores and apply it to the evaluation scores
./bin/fusion.sh dev_scores_4fusion.txt eval_scores_4fusion.txt queries_in_data.ref
<query_id> <file_id> <start_time> <duration> <score> <decision>
./bin/raw2stdslist.sh dev_fusion.scores \ ./scoring_atwv_sws2013/sws2013_dev/sws2013_dev.tlist.xml > fusion-devterms.stdlist.xml
./bin/raw2stdslist.sh eval_fusion.scores \ ./scoring_atwv_sws2013/sws2013_eval/sws2013_eval.tlist.xml > fusion-evalterms.stdlist.xml
Reference results
dev (mtwv/atwv) | eval (mtwv/atwv) | |
---|---|---|
akws-br | 0.1571 / 0.1408 | 0.1441 / 0.1271 |
dtw-br | 0.2066 / 0.2012 | 0.1654 / 0.1581 |
fusion | 0.2731 / 0.2713 |