Task Description
The Test Collection
The test collection consists of a set of natural language commands. Each natural language command is mapped to one or more actions within the APIs Knowledge Base (KB). The interpretation of the natural language command and its mapping to the API is also dependent on the user personal context (User KB), which contains the user personal data (contacts, addresses, emails, phones, ...).
The test collection consists of:
Natural Language Command: A natural language command that can be mapped to one or more API frames.
Action KB: The set of available API frames along with their respective documentation. The information describing the API elements does not follow a strict pattern. While some documentation have rich natural language descriptions or show usage examples, others are succint and just contain the frame and parameter names. The same occurs concerning data format, data type. This test collection reflects the variability and heterogeneity that we find in real world APIs.
Command-Action Mapping: The set of mappings from natural language commands to the actions from the Action KB.
User KB: A personal user information dataset.
Example
An example of a natural language command is shown below:
Command: Translate “I love you” to German.
Output: Executable Command {
ACTION ID: 2
LIB: "newmashape"
NAME: "Translate Text"
PARAMS:
languageConverted: “en”
languageToConvert: “de”
text: “I love you”
}
45-mashape is the unique identifier of the action in the KB.
The User KB is used to support co-referential mentions, making the interaction in the programming task more natural.
Consider the following example:
Command: Send an SMS to John Smith saying I will be late tonight.
The effective parameter of the action "send sms" is the phone number of John Smith, which should be retrieved from the user KB, according to the excerpt example bellow.
{
"agenda":[
{
"name":"John Smith",
"address":"7 North Avenue, New York, USA",
"mail":"john@smith.com",
"phone":"111 555 777",
"tags":null
}
]
}
Diagram
Trial Data
The trial data is composed of three json files: Actions KB, User KB and the Command-Action mapping. A short excerpt of the set of commands is provided below.
Evaluation
The final dataset contains commands and their mappings to the Actions. Given a command in natural language, it is expected that the participating systems provide:
- The correct action(s) be identified;
- The correct mapping of text chunks within the natural commands to parameters;
The participating systems will be evaluated with regard to precision, recall, and f1-score. Participating teams are allowed to use resources such as language resources. Baselines based on simple string matching approaches and random selection will be made available.