Handling Multiple Benchmark Datasets with Different Licenses in MOT

[stellawang196]

In some cases, models are evaluated on multiple public benchmark datasets, each potentially released under a different license. What is the recommended approach in MOT for labeling the license in such cases?

[lehors]

This is indeed a current limitation of the MOF & MOT. We have no provision for multiple licenses applying to the same component. Such as the case you bring up where the dataset is made of a collection of datasets with different licenses.
Thank you for raising this issue. We ought to think about a way to address it in the future.

In the meantime I think we should record the most restrictive license of the sets since what really matters is whether the licenses are permissive open licenses or not. If the most restrictive license is a permissive open license then the datasets will be evaluated as included and open. If the most restrictive license is not a permissive open license then the datasets will be evaluated as included but not open. Which gives us the right result in both cases. If the various licenses are equivalent any one of them can be entered without affecting the results.

[lehors]

We should add this recommendation to the documentation.