Data and Tools
Updates for the Evaluation Period
1- We've updated the action KB file[1] and the action-command training mapping file[2] to correct some small inconsistencies. Please, update your algorithm using the new files.
2- The evaluation period has started. The input file is available in our website[3] and the codalab competition[4] is ready to receive the submissions.
The Test Collection
Actions are composed of a name, a description, parameters, attributes and additional information. Except for the name, none of the fields is mandatory. The same happens for the additional information, which also varies according to the action provider. The high variety for the test collection both for the API specification and for the supporting data is planned and desired as we aim to reproduce the lack of standardisation existing on real-world API documentation.
Attributes
Attributes are the content returned after the action execution. When not described, we can admit that the action returns a single non-named string output.
If and Foreach constructors
In addition to the actions present in the Action Knowledge Base, the mappings also use If and Foreach constructors, having the same semantics commonly expressed in programming languages.
Special Tags
Mappings also make use of two tags that are expressed in between brackets (like in html).
<returnX>
The return tag represents the content returned by the action X, where X is the sequential identifier starting by 1.
<item>
The item tag is used only in the context of Foreach constructors. It represents the iterated item.
Both tags can be expanded with an attribute name. Examples of valid tags are:
<return1> - meaning the data returned by the first action in the scenario.
<item>.url - represent the attribute url of the item.
For any question or clarification, please send us a message.
Training Data
The training data is composed of three json files: Actions KB, User KB and the Command-Action mapping.