Task Description

The Test Collection


The test collection consists of a set of natural language commands. Each natural language command is mapped to one or more actions within the APIs Knowledge Base (KB). The interpretation of the natural language command and its mapping to the API is also dependent on the user personal context (User KB), which contains the user personal data (contacts, addresses, emails, phones, ...).

The test collection consists of:

Natural Language Command: A natural language command that can be mapped to one or more API frames.


Action KB: The set of available API frames along with their respective documentation. The information describing the API elements does not follow a strict pattern. While some documentation have rich natural language descriptions or show usage examples, others are succint and just contain the frame and parameter names. The same occurs concerning data format, data type. This test collection reflects the variability and heterogeneity that we find in real world APIs.


Command-Action Mapping: The set of mappings from natural language commands to the actions from the Action KB.


User KB: A personal user information dataset.





An example of a natural language command is shown below:


Command: Translate “I love you” to German.


Output: Executable Command {

    ACTION ID: 2
    LIB: "newmashape"
    NAME: "
Translate Text"

        languageConverted: “en”
        languageToConvert: “de”
        text: “I love you”


45-mashape is the unique identifier of the action in the KB.


The User KB is used to support co-referential mentions, making the interaction in the programming task more natural.


Consider the following example:


Command: Send an SMS to John Smith saying I will be late tonight.


The effective parameter of the action "send sms" is the phone number of John Smith, which should be retrieved from the user KB, according to the excerpt example bellow.


         "name":"John Smith",
         "address":"7 North Avenue, New York, USA",
         "phone":"111 555 777",










Trial Data



The trial data is composed of three json files: Actions KB, User KB and the Command-Action mapping. A short excerpt of the set of commands is provided below.

Action KB file

User KB file

Command-Action mapping file





The final dataset contains commands and their mappings to the Actions. Given a command in natural language, it is expected that the participating systems provide:

  1. The correct action(s) be identified;
  2. The correct mapping of text chunks within the natural commands to parameters;

The participating systems will be evaluated with regard to precision, recall, and f1-score. Participating teams are allowed to use resources such as language resources. Baselines based on simple string matching approaches and random selection will be made available.

Contact Info

Andre Freitas - University of Passau andre.freitas@uni-passau.de
Juliano Sales - University of Passau juliano-sales@uni-passau.de
Siegfried Handschuh - University of Passau siegfried.handschuh@uni-passau.de

Other Info


  • Trial data released
  • Traning data released
  • Codalab competition released
  • New version of the traning data released
  • Evaluation period has started