TextVQA Challenge is available at this link.
In MMF, we provide the starter code for various baseline models for this challenge. TextVQA dataset will also be automatically downloaded during first training.
In this tutorial, we provide steps for running training and evaluation with M4C model on TextVQA dataset and generating submission file for the challenge. The same steps can be used for other models.
Follow the prerequisites for installation of mmf here.
Training and Evaluation
For running training on
train set, run the following command:
This will train the
m4c model on the dataset and generate the checkpoints and best trained model (
m4c_final.pth) will be stored in an experiment folder under the
./save directory by default (unless
env.save_dir is overriden).
Next run evaluation on the validation
This will give you the performance of your model on the validation set. The metric will be TextVQA Accuracy.
Predictions for Challenge
After we trained the model and evaluated on the validation set, we will generate the predictions on the
test set which can be submitted to the Test Standard phase. The prediction file should contain the following:
- Question ID,
With MMF you can directly generate the predictions in the required submission format with the following command:
This command will output where the generated predictions JSON file is stored.
Submission for Challenge
Next you can upload the generated json file to EvalAI page for TextVQA here. Follow these steps:
To check your results, you can go in 'My submissions' section and select 'Validation Phase' and click on 'Result file'.
Now, you can either edit the M4C model to create your own model on top of it or create your own model inside MMF to beat M4C in challenge.