arduous-logo

ARDUOUS: 1st International Workshop on
Annotation of useR Data for UbiquitOUs Systems 


  • Affiliated to IEEE PerCom 2017
  • March 17, 2017 (full day)
  • Kona (Big Island), Hawaii, USA

Call for papers

Datasets and annotation tools & Live annotation session
Submission guidelines
Registration
Important dates
Accepted papers
Keynote talk
Workshop program
Program committee

With ubiquitous systems relying more and more on large datasets for designing and testing models of users' activities, the process of data labelling is becoming a major concern for the community. Labelling user data is a central part of the design and evaluation of pervasive systems that aim to support the user through situation-aware reasoning. It is essential both in designing and training the system, either through the definition of a suitable situation model in knowledge-driven applications, or though the preparation of training data for learning tasks in data-driven models. Hence, the quality of annotations can have a significant impact on the performance of the derived systems. Labelling is also vital for validating and quantifying the performance of applications. In particular, comparative evaluations require the production of benchmark datasets based on high-quality and consistent annotations.

This workshop focuses on:

  • the role and impact of annotations in designing ubiquitous applications,
  • the process of labelling, and the requirements to produce high quality annotations, and
  • tools and automated methods for annotating user data.

The goal of the workshop is to bring these three topics to the attention of researchers from interdisciplinary backgrounds, and to initiate a reflection on possible resolutions of the related problems.

Call for papers

We invite you to submit papers with a maximum of 6 pages that offer new empirical or theoretical insights on the challenges and innovative solutions associated with labelling of user data, as well as on the impact that labelling choices have on the user and the developed system.

The topics of interest include, but are not limited to:

  • methods and intelligent tools for annotating user data for pervasive systems;
  • processes of and best practices in annotating user data;
  • methods towards an automation of the annotation;
  • improving and evaluating the annotation quality;
  • ethical issues concerning the annotation of user data;
  • beyond the labels: ontologies for semantic annotation of user data;
  • high-quality and re-usable annotation for publicly available datasets;
  • impact of annotation on a ubiquitous and intelligent system's performance;
  • building classifier models that are capable of dealing with multiple (noisy) annotations and/or making use of taxonomies/ontologies;
  • the potential value of incorporating modelling of the annotators into predictive models.

Furthermore, we encourage the discussion on the challenges and requirements for annotating textual resources so that they can be automatically interpreted and utilised by ubiquitous applications.
Examples of such resources are textual instructions such as recipes and manuals, natural language conversations, and social media posts.
Example datasets are listed below:
http://www.wikihow.com (textual instructions)
http://www.bbc.co.uk/food/recipes (cooking recipes)
http://www.cs.cmu.edu/~enron/ (emails NLP dataset)
http://cs.ubc.ca/labs/lci/bc3.html (blog and email datasets)
http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22/ (twitter sentiment analysis training corpus)
https://developers.google.com/search/docs/data-types/recipes (annotated recipes for search in Google)


Datasets and annotation tools & Live annotation session
We encourage workshop participants to experiment with a common set of annotation tools and datasets, possibly in addition to their own. The results of these experiments and challenges encountered may be exploited and discussed in the participants' own submissions.

In addition, they will also be discussed on the workshop day during a live annotation session followed by a discussion session. The live annotation session will allow participants to experiment with both an online and an offline annotation tools on a same project, and to exchange and reflect on the process. If the discussion obtains enough participation, its results will be reported in a common article after the workshop.

Instructions

A description of the experiment and instructions for downloading, installing, and using the annotation tools and datasets may be found here.


Datasets
Annotation tools
  • SPHERE annotation tool: android application download
  • Rostock Univeristy annotation tool: web-based, download
For any trouble with downloading or using the datasets and annotation tool, please email Kristina Yordanova at kristina.yordanova@uni-rostock.de or Adeline Paiement at csatmp@bristol.ac.uk.

Submission guidelines

Format
Maximum of 6 pages including references, formatted in accordance with the IEEE Computer Society author guidelines. The IEEE LaTeX and Microsoft Word templates, as well as related information, can be found at the IEEE Computer Society website (http://www.computer.org/web/cs-cps/authors and https://www.ieee.org/conferences_events/conferences/publishing/templates.html).

All papers included in the PerCom 2017 Workshops will appear in the conference proceedings published by IEEE.

Submission

Through the EDAS submission system here.

Review process

The review process will be double blind.

Registration

There will be no separate workshop registration, but one single registration will cover both conference and workshop participation. It is mandatory that at least one author will register and will participate to present the paper during the technical sessions of workshops.

Important dates

Submission deadline: November 21, 2016 (extended)
Notification: December 23, 2016
Camera ready version: January 13, 2017 (FIRM)
Author registration due by: January 13, 2017 (FIRM)
Workshop: March 17, 2017 (full day)


Accepted papers

With the help of the members of the Technical Program Committee, we selected 8 papers to be presented in the workshop technical program:
  • Alexander Diete, Timo Sztyler, Heiner Stuckenschmidt. A Smart Data Annotation Tool for Multi-Sensor Activity Recognition
  • Fausto Giunchiglia, Enrico Bignotti, Mattia Zeni. Personal context modelling and annotation
  • Przemyslaw Woznowski, Emma Tonkin, Pawel Laskowski, Niall Twomey, Kristina Yordanova, Alison Burrows. Talk, text or tag? The development of a self-annotation app for activity recognition in smart environments
  • Frank Krüger, Christina Heine, Sebastian Bader, Albert Hein, Stefan Teipel, Thomas Kirste. On the Applicability of Clinical Observation Tools for Human Activity Annotation
  • Przemyslaw Woznowski, Alison Burrows, Pawel Laskowski, Emma Tonkin, Ian Craddock. Evaluating the use of voice-enabled technologies for ground-truthing activity dat
  • Michael Edwards, Jingjing Deng, Xianghua Xie. Labeling Subtle Conversational Interactions
  • Joseph Rafferty, Jonathan Synnott, Chris Nugent, Gareth Morrison, Elena Tamburini. NFC based dataset annotation within a behavioral alerting platform
  • Carl M Olsson. Engagement Issues in Self-Tracking: Lessons Learned from User Feedback of Three Major Self-Tracking Services
Keynote talk

Speaker:
Prof. Takuya Maekawa, Osaka University
 
Title:
Good pervasive computing studies require laborious data labeling efforts: Our experience in activity recognition and indoor positioning studies
 
Abstract:
Preparing and labeling sensing data are necessary when we develop state-of-the-art sensing devices or methods in our studies. Since developing and proposing new sensing devices or modalities are important in the pervasive computing and ubicomp research communities, we need to prepare high quality labeled data by making use of our limited time whenever we develop a new sensing device. In this keynote talk, we first introduce our recent studies on activity recognition and indoor positioning based on machine learning and then introduce important things for preparing labeled data learned from our experience in our research activities. 

Workshop program
 
9:00 Keynote Talk
Prof. Takuya Maekawa - Good pervasive computing studies require laborious data labeling efforts: Our experience in activity recognition and indoor positioning studies
10:00 Break
10:30 Short presentation of the accepted papers
  A Smart Data Annotation Tool for Multi-Sensor Activity Recognition
Alexander Diete, Timo Sztyler, Heiner Stuckenschmidt
  Personal context modelling and annotation
Fausto Giunchiglia, Enrico Bignotti, Mattia Zeni  
  Talk, text or tag? The development of a self-annotation app for activity recognition in smart environments
Przemyslaw Woznowski, Emma Tonkin, Pawel Laskowski, Niall Twomey, Kristina Yordanova, Alison Burrows
  On the Applicability of Clinical Observation Tools for Human Activity Annotation
Frank Krüger, Christina Heine, Sebastian Bader, Albert Hein, Stefan Teipel, Thomas Kirste  
  Evaluating the use of voice-enabled technologies for ground-truthing activity data
Przemyslaw Woznowski, Alison Burrows, Pawel Laskowski, Emma Tonkin, Ian Craddock  
  Labeling Subtle Conversational Interactions
Michael Edwards, Jingjing Deng, Xianghua Xie  
  NFC based dataset annotation within a behavioral alerting platform
Joseph Rafferty, Jonathan Synnott, Chris Nugent, Gareth Morrison, Elena Tamburini  
  Engagement Issues in Self-Tracking: Lessons Learned from User Feedback of Three Major Self-Tracking Services
Carl M Olsson  
12:00 Lunch
13:15 Poster session - all accepted papers
14:00 Live annotation session
14:45 Break
15:15  Panel Discussion
17:00 Wrapping-up

Program Committee

Organisers
Kristina Yordanova, University of Rostock, Germany
Adeline Paiement, Universities of Bristol and Swansea, UK
Jesse Hoey, University of Waterloo, Canada

Live Annotation Session Chair

Max Schröder, University of Rostock, Germany

Committee Members
Alison Burrows, University of Bristol, UK
Pete Woznowski, University of Bristol, UK
Niall Twomey, University of Bristol, UK
Tom Diethe, University of Bristol, UK
Rachel King, University of Reading, UK
Thomas Kirste, University of Rostock, Germany
Peter Flach, University of Bristol, UK
Ian Craddock, University of Bristol and Toshiba Research, UK
Gustaf Hendeby, Linköping University, Sweden
Pietro Carnelli, University of Bristol, UK
Thierry Declerck, German Research Centre for Artificial Intelligence, Germany
Peter Haddawy, Mahidol University, Thailand
Philipp Koldrack, German Center for Neurodegenerative Diseases, Germany
Aftab Khan, Toshiba Research, UK
Galia Angelova, Bulgarian Academy of Sciences, Bulgaria
Frank Krüger, University of Rostock, Germany
Sebastian Bader, University of Rostock, Germany
Cecily Morrison, Microsoft Research Cambridge, UK
Leandro Pecchia, University of Warwick, UK
Aristodemos Pnevmatikakis, Athens Information Technology, Grece
Antonio Krüger, Gearman Research Centre for Artificial Intelligence, Germany
Thomas Ploetz, Newcastle University, UK
Edith Law, University of Waterloo, Canada
Alex Mihailidis, University of Toronto, Canada
Diane Cook, Washington State University, USA
Chris Nugent, Ulster University, UK
Ian Cleland, Ulster University, UK
Stephen Intille, Northeastern University, USA
Alexandros Zenonos, University of Southampton, Toshiba Telecommunications Research Laboratory, UK

For further information regarding the workshop and paper submission, please contact Kristina Yordanova at kristina.yordanova@uni-rostock.de or Adeline Paiement at csatmp@bristol.ac.uk.