Task 3: Community Question Answering
Building on the success of SemEval 2015 Task 3 “Answer Selection in Community Question Answering” (see the task description paper), we propose an extension, which covers a full task on Community Question Answering (CQA) and which is, therefore, closer to a real application (see, e.g., Qatar Living forum).
CQA systems are gaining popularity online. Such systems are seldom moderated, quite open, and thus they have little restrictions, if any, on who can post and who can answer a question. On the positive side, this means that one can freely ask any question and expect some good, honest answers. On the negative side, it takes effort to go through all possible answers and to make sense of them. For example, it is not unusual for a question to have hundreds of answers, which makes it very time-consuming for the user to inspect and to winnow through them all. The present task could help to automate the process of finding good answers to new questions in a community-created discussion forum (e.g., by retrieving similar questions in the forum and by identifying the posts in the answer threads of those similar questions that answer the original question well).
In essence, the main CQA task can be defined as follows:
“given (i) a new question and (ii) a large collection of question-comment threads created by a user community, rank the comments that are most useful for answering the new question"
Since automatic systems are typically imperfect, from the user perspective, it is more convenient to rank the comments according to their probability of relevance, therefore we focus on ranking tasks.
The test question is new with respect to the collection, but is expected to be related to one or several questions in the collection. The best answers can come from different question-comment threads. In the collection, the threads are independent of each other and the lists of comments are chronologically sorted and contain some meta information (e.g., date, user, topic, etc.). The comments in a particular thread are intended to answer the question initiating that thread, but since this is a resource created by a community of casual users, there is a lot of noise and irrelevant material, apart from informal language usage and lots of typos and grammatical mistakes. Interestingly, the questions in the collection can be semantically related to each other, although not explicitly.
Our intention is not to propose just another regular Question Answering task. Similarly to last year, we have three objectives: (1) to focus on semantic-based solutions beyond simple “bag-of-words” representations and “word matching” techniques; (2) to study the new NLP phenomena connected with the community question answering scenario, e.g., relation between comments in a thread, relations between different threads and question-to-question similarity; and (3) to facilitate the participation of non IR/QA experts to our challenge. The third point is achieved by explicitly providing the set of potential answers (the search engine step is already carried out by us) to be (re)ranked and by defining two optional sub-tasks apart from the main CQA task, namely:
- A. Question-Comment Similarity: given a question from a question-comment thread, rank the comments according to their relevance (similarity) with respect to the question (this corresponds to the last year's task: SemEval-2015 Task 3).
- B. Question-Question Similarity: given the new question, rerank all similar questions retrieved by a search engine (assuming that the answers to the similar questions should be answering the new question as well).
Note that the two previous subtasks should give participants enough tools to create a CQA system to solve the main task. Nonetheless, one can approach CQA without necessarily solving the two task above. Participants will be free to use whatever approach they want. Note also that the participation in the main task and/or the two subtasks will be optional. Participants can go for any possible combination. There will be separate evaluation and analysis for each subtask.
Keeping with the multilinguality feature from 2015, we provide data for two languages: English and Arabic. For English, we use real data from the community-created Qatar Living Forums. The Arabic data has been collected from medical forums (webteb, altibbi, and consult islamweb) with a slightly different procedure. Due to the different nature of the sources, the dataset for Arabic is different and we are only proposing the main ranking CQA task on this data, i.e., finding good answers for a given new question. Finally, we provide training data for all languages and subtasks with human supervision. All examples have been manually labeled by a community of annotators in a crowdsourcing platform.
For a precise definition of all subtasks and the evaluation see the Task Description page. The corpora and the tools can be downloaded from the Data and Tools page.
(Simplified) English Example
Let us consider the following question, which is NOT in Qatar Living (QL):
Q: Can I drive with an Australian driver’s license in Qatar?
Retrieved Questions:
Q1: How long can i drive in Qatar with my international driver's permit before I'm forced to change my Australian license to a Qatari one? When I do change over to a Qatar license do I actually lose my Australian license? I'd prefer to keep it if possible...
---> question similar to Q1 (Task B)
Comment to Q1:
depends on the insurer, Qatar Insurance Company said this in email to me
“Thank you for your email! With regards to your query below, a foreigner is valid to drive in Doha with the following conditions: Foreign driver with his country valid driving license allowed driving only for one week from entry date Foreign driver with international valid driving license allowed driving for 6 months from entry date Foreign driver with GCC driving license allowed driving for 3 months from entry”
As an Aussie your driving licence should be transferable to a Qatar one with only the eyetest (temporary, then permanent once RP sorted).
---> good answer to Q1 (Task A):
---> good answer to Q (Task C, main task)
Q2: Hi there :D dose anyone knows how much would it cost to get a driving license !! although i have had it before in my country so practically i know how to drive. any HELP !?
---> not similar to Q, i.e., a negative example for Task B
Comment to Q2: Why no-more short course? let me know plz. if someone knows driving very well then he/she can get license after shore course ?
---> negative example for Q2 (Task A)
---> negative example for Q (Task C)
Note that instead of just classifying or finding a limited set of similar questions and correct answers, we will ask the participants to provide relevance rankings.
(Simplified) Arabic Example
New question
Q: الى ماذا يمكن ان يدل الالم في الاسنان الذي يظهر فقط عند شرب او اكل شيئ ساخن او بارد وفي اوقات اخرى لا يوجد الم. ما هو العلاج
(What does the pain of teeth mean when it only appears when drinking something cold or eating something hot or cold, while it doesn't exist otherwise?)
Retrieved questions:
Q1:عند شرب الماء البارد اشعر ببعض الالم في مجموعه من الاسنان فما هو السبب </QAquestion>
(While drinking cold water I feel pain in some of my teeth, what is the reason?)
A1: اسباب حساسية الاسنان يعود الى تاكل الطبقة الخارجية الحامية للاسنان و التي تسمى المينا .... و انكشف طبقة العاج الغنية بالمستقبلات الحسية .... و هناك عدة مسببات منها: التسوسات و النخور - انحسار اللثة بسبب الالتهابات اللثوية و اربطة السن- التفريش الخاطئ للاسنان - الاكثار من استخدام المنتجات المبيضة للاسنان - كثرة تناول الاطعمة و المشروبات الغازية الحمضية - انكسار الاسنان لا بد من الاسراع بمراجعة الطبيب حتى لا تتفاقم المشكلة
(The reason of teeth sensivity is the decay of the outer layer the protets the teeth ...)
Good Answer (label: Direct)
Q2: منذ شهر تقريبا خلعت طاحونة العقل واصبح هناك الم بالضرس المجاور لها واخبرتوني ان الجذر انكشف ولكني لا زلت اعاني عند شرب الماء البارد فالي متي سيبقي الجذر مكشوفا وهل هناك علاج مفيد لتلك الحالة </QAquestion>
(1 month ago, I removed my wisdome tooth, and since then I get pain in the next tooth. You answered before that the nerve is exposed and it should take a while. I still have pain when I drink cold water, How long this would take? Is there a cure to this pain?)
A2:ان قلع ضرس العقل يترك خلفه حفره بجا نب الضرس المجاور وتظهر هذه الاعراض وخاصة عند شرب البارد والساخن واكل السكاكر والحامض لا يوجد وقت محدد ولكن عندما تلتئم الحفره تماما يزول الالم ولكن بامكانك استخدام معاحين اسنان للحساسيه والمضمضه بمضمضات تحتوى على الفلورايد مثل la calut sensetive mouth wash وفى اعتقادى ان المساله مسالة وقت
(Removing a tooth makes the next one exposed and these are the symptons that you have. There is no exact time for cure, but you can use "la calut sensetive mouth wash" that can help)
Relevant Answer: (label: Relevant)
Q3: اعاني من شق في احد اسناني الامامية وكذلك لاحضت مؤخرا تكدس الطعام بين الاسنان عند الاكل بمعني يلزمني اني افرشي اسناني بعد اي اكل يدخل فمي فما السبب والعلاج
(I have a crack in one of my teeth and food stuck there and I have to clean every time I eat. What can be the reason of this to happen?)
A3:ومن اسباب وجود الشقوق على طبقة المينا شرب السوائل البارده والساخنه في نفس الوقت.وكثرة تناول الحمضيات تضر في طبقة المينا.</QAanswer>
(Usually drinking cold and hot water at the same time could be the reason. Also eating a lot of acidic food causes this)
Irrelevant Answer: (label: Irrelevant)
The above rank would be optimal although in the evaluation we will not take into account the position of Relevant with respect to Irrelevant questions. Thus the important action would be to put the Direct answer on top.